A Theoretical Understanding of Gradient Bias in Meta-Reinforcement Learning
Authors: Bo Liu
Xidong Feng
Jie Ren
Luo Mai
Rui Zhu
Haifeng Zhang
Jun Wang
Yaodong Yang
Venue: NeurIPS 2022 Accept (2022)
Keywords: Meta Reinforcement Learning
Gradient Bias

Abstract: Gradient-based Meta-RL (GMRL) refers to methods that maintain two-level optimisation procedures wherein the outer-loop meta-learner guides the inner-loop gradient-based reinforcement learner to achieve fast adaptations. In this paper, we develop a unified framework that describes variations of GMRL algorithms and points out that existing stochastic meta-gradient estimators adopted by GMRL are actually \textbf{biased}. Such meta-gradient bias comes from two sources: 1) the compositional bias incurred by the two-level problem structure, which has an upper bound of $\mathcal{O}\big(K\alpha^{K}\hat{\sigma}_{\text{In}}|\tau|^{-0.5}\big)$ \emph{w.r.t.} inner-loop update step $K$, learning rate $\alpha$, estimate variance $\hat{\sigma}^{2}_{\text{In}}$ and sample size $|\tau|$, and 2) the multi-step Hessian estimation bias $\hat{\Delta}_{H}$ due to the use of autodiff, which has a polynomial impact $\mathcal{O}\big((K-1)(\hat{\Delta}_{H})^{K-1}\big)$ on the meta-gradient bias. We study tabular MDPs empirically and offer quantitative evidence that testifies our theoretical findings on existing stochastic meta-gradient estimators. Furthermore, we conduct experiments on Iterated Prisoner's Dilemma and Atari games to show how other methods such as off-policy learning and low-bias estimator can help fix the gradient bias for GMRL algorithms in general.

A Theoretical Understanding of Gradient Bias in Meta-Reinforcement Learning
Bo Liu" Institute of Automation, Chinese Academy of Sciences benj iu jenijaiiiiu.e.ecemmalll....
Xidong Feng" University College London xidong. feng 200uc1 ac uk
Jie Ren
Luo Mai University of Edinburgh luo mai @ed. ac. uk
Rui Zhu DeepMind ruizhu@google.com com
University of Edinburgh j jieren98060gmail com
Haifeng Zhang Institute of Automation, CAS Nanjing Artificial Intelligence Research of IA haifeng. zhang@ia.an ac. cn
Jun Wang
University College London
jun. wangdes ucl. ac uk
Yaodong Yang*
Institute for AI, Peking University Beijing Institute for General Al yaodong. eeoddong-yanggppe yang@pku. edu. en
Abstract
Gradient-based Meta-RL (GMRL) refers to methods that maintain two-level op- timisation procedures wherein the outer-loop meta-learner guides the inner-loop gradient-basee reinforcement learner achieve fast adaptations. In this paper, develop a unified framework that describes variations GMRL algorithms and points out that existing stochastic meta- gradient estimators adopted by GMRL are actually biased. Such meta-gradient bias comes from two sources: 1) the composi- tional bias incurred by the two-level problem structure, which has upper bound of O Ka 0 f 0.5) w.r.f. inner-loop update step K, learning rate a, estimate variance &2 and sample size [T], and 2) the multi-step Hessian estimation bias Åμ due the of autodiff, which has polynomial impact O(K 1) (Az))=))) on the meta- geat-ggradient bias. We study tabular MDPs empirically and offer quantitative evidence that testifies our theoretical findings on existing stochastic meta-gradient estimators. Furthermore, we conduct experiments on Iterated Prisoner's Dilemma and Atari games to show how other methods such as off- off--olliy learning and low-bias estimator can help fix the gradient bias for GMRL algorithms in general.
Introduction
Meta Learning, also known as learning to learn, is proposed to equip intelligent agents with meta knowledge for fast adaptations [30]. Meta- Reinforcement Learning (RL) algorithms aim to train RL
"Equal contribution, the order is determined by flipping coin. See Appendix for more details.
t Cooresponinng author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

agents that can adapt to new tasks with only few examples [5, 10, 16]. For example, MAML-RL [10], a typical Meta-RL algorithm. learns the initial parameters of an agent's policy so that the agent can rapidly adapt to new environments with limited number of policy-gradient updates. Recently, Meta-RI. algorithms have been further developed beyond the scope of learning fast adaptations. An important direction is to conduct online meta- gett--graieet learning for adaptively tuning algorithmic hyper-parameters [39, 41] or designing intrinsic reward [42] during training in one single task. Besides, there are Meta-RL developments that manage to discover new RL algorithmss by learning algorithmic components or other fundamental concepts in RL, such as the policy gradient objective [26] or the TD-target [40], which can generalise to solve distribution of different tasks.
In general, gradient-basee based Meta-RL (GMRL) tasks can be formulated by two-level optimisation procedure. This procedure optimises the parameters of an outer-loop meta-leamer, whose objective dependent on step policy update process (e.g., stochastic gradient descent) conducted by an inner-loop learner. Formally, this procedure can be written as:
max (Φ) (Φ) == (0.
(I)
s.t. gi+l 00 av ( (0. K- -I,,
where O are inner-loop policy parameters, & are meta parameters, is the learning rate, Jin and JOut are value functions for the inner and the outer-loop learner.
In solving Eq. (1), estimating the meta- veeta-gradeet of JK (Φ) from a two-level optimisation process is non-trivial, how to conduct accurate estimation on V v (Φ) has been a critical yet challenging research problem [23, 29, 33].
In this paper, we point out that the meta-gradient estimators adopted by many recent GMRL. methods [26, 40] are in fact biased. We conclude that such bias comes from two sources: (1) the compositional bias and (2) the multi-step Hessian bias, The compositional bias origins from the discrepancy between the sampled policy gradient Jl and expected policy gradient VaJin, and the multi- step Hessian estimation bias occurs due to the biased Hessian estimation v2, resulting from the employment of automatic differentiation in modern GMRL implementations.
There are very few prior work that investigates the bias in GMRL, most of them limited to the Hessian estimation bias in the MAML-RL setting [29, 33]. Our paper investigates the above two biases in broader setting and applies on generic meta-gradient estimators. For the compositional bias term, offer the first theoretical analysis its quantity, based on which then investigate current mitigation solutions. For the multi-step Hessian bias, we provide rigorous analysis in GMRL settings particularly for those GMRL tasks where complex inner-loop optimisations are involved.
For the rest of the paper, we first introduce a unified Meta-RL framework that can describe variations of existing GMRL algorithms. Building on this framework, we offer two theoretical results that 1) the compositional bias has an upper bound of O 0m||T|-055) with respect to the inner-loop update step K, the learning rate a, the estimate variance o and the sample size [r]. and 2) the multi-step Hessian bias Åς has polynomial impact of ((K- - - To validate our theoretical insights on these two biases, we conduct a comprehensive list ablation studies. Experiments over tabular MDP with MAML-RL [10] and LIRPG [42] demonstrate how quantitatively these two biases influence the
k-1).
estimation accuracy, which consolidates our thcorics.
Furthermore, our theoretical results help understand to what extent existing methods can mitigate theses two bias empirically. For the compositional bias, we show that off-policy learning methods can reduce the inner-loop policy gradient variance and the resulting compositional bias. For the multi-step Hessian bias, we study how the low-variance curvature technique based on Rothfuss et al. [29] can help correct the Hessian bias for general GMRL problems. We test these solutions on environments including iterated Prisoner's Dilemma with off-policy corrected LOLA-DiCE [13], and eight Atari games based on MGRL [39] with the multi-step Hessian correction technique. Experimental results confirm that those bias-correction methods can substantially decrease the meta-gradients bias and improve the overall performance on rewards.
2

---

2 Related Work
A pioneering work that studies meta-gradient estimation is Al-Shedivat et al. [1] who discussed the biased estimation problem of MAML and proposed the E-MAML sample based formulation to fix the meta-gradient bias. Following work includes Rothfuss et al. [29]. Liu et al. [23], Tang et al. [33] that tried to fix the meta-gradion estimation error by reducing the estimation variance so as to improve the performance. Morcover, Foerster et al. [12]. Farquhar et al. [8], Mao et al. [24] discussed the higher-order gradient estimation in RL. Recently, Bonnet et al. [4], Vuorio et al. [36] proposed algorithms to balance the bias variance trade-off for meta-grat veta-gradient estimates. Within theoretical context of GMRL, most theoretical analysis focuses on convergence to stationary points. Fallah et al. [6] established convergence guarantees of basdde meta- meta-learning algorithms for supervised learning with one-step inner-loop update. Ji et al. [18] extended the analysis to multi-step inner loop updates. Fallah et al. [7] established convergence for the E-MAML formulation. Our work is different from all the above prior work from three folds: (1) we study the additional bias term (the compositional bias) (2) we consider different formulation (expected update while sample-based update in [36]. refer to Appendix B for more discussions) (3) we study a broader scope of applications that include different GMRL algorithm instantiations and settings.
In the following part, we review existing GMRL algorithms and categorise them into four research topics. We offer one typical example for each topic in Table and further discussed their relationship and how they can be unified into one framework in Sec. 3. We also provide a more self-contained explanation for each topic in Appendix A for readers that are not familiar with GMRL.
Few-shot RL. The idea of few-shot RL is to enable RL agent with fast learning ability. Specifically, the RL agent is only allowed to interact with the environment for few trajectory to conduct task- specific adaptation. Conducting few-shot RL has two approaches: gradient based and context based. Context based Meta-RL involves works [5, 15, 27, 37], which uses neural network to embed the information from sew-shot interactions so as to obtain task-relevant context. Gradient based few-shot RL [1, 23, 29] focus on meta-learning the model's initial parameters through meta-gradient descent.
Opponent Shaping. Foerster et al. [13], Letcher et al. [22], Kim et al. [19] explicitly models the learning process of opponents in multi-agent learning problems, which can be thought of as GMRL problems since meta-gradis gradient estimation these works involve differentiation over opponent's 's policy updates. By modelling opponents' learning process, the multi-agent learning process can reach better social welfare [13, 22] or the ego agent can adapt to a new peer agent [19].
Single-lifetime Meta-gradie
RL. This line of research focuses on learning online adaptation over algorithmic hyper-parameters to enhance the performance of an RL agent in one single task, such as discount factor [39, 41], intrinsic reward generator [42], auxiliary loss [34], reward shaping mechanism [17], and value correction [44]. The main feature of online meta-gradient RL is that they are under single- single-lifetime setting [40], meaning that the algorithm only iterates through the whole RL learning procedure in one task rather distribution of tasks.
Multi-lifetime Meta-gradient RL. Compared to the previous single-lifetime settings. "multi- lifetime" refers settings where agents learn to adapt on a distribution of tasks environments. The meta-learning target includes policy gradient or TD learning objectives [3, 20, 26). intrinsic reward [43], target value function [40], options in hicrarchical RL [35], and recently the design of curriculum in multi-agent learning [9].
3 A Unified Framework for Meta-gradient Estimation
In this section, we derive general formulation for meta- meta-gradient estimation in GMRL. This formula- tion enables us to conduct general analysis about meta meta-gradiem estimation and we will show how algorithms in the four research topies mentioned in Sec. 2 can be described through it.
We propose that a general GMRL objective with K-step inner-loop policy gradient update can be written as the following objective
max ( = (*).0..(.c 0K 0 + a $
V S
(2)
1=0
3

---

Table 1: Four typical gradient-based Meta-RL (GMRL) algorithms.
Category
Algorithms
Meta parameter
Inner parameter 0
Few-shot RL
MAML [10]
Initial Parameter
Initial Parameter
Opponent Shaping
LOLA [13]
Ego-agent Policy
Other-agent Policy
Single-lifetime MGRL
MGRL [39]
Discount Factor
RL Agent Policy
Multi-lifetime MGRL
LPG [26]
LSTM Network
RL Agent Policy
We denote the meta-parameter as 4. and the pre- and post-adapt inner parameters as O and 0* respectively. The meta objective as fOut, the inner loop objective as رار 'he general objective for GMRL is to maximise the meta objective (Ø, e²), where the inner-loop post-adapt parameters are obtained by taking K policy gradient steps. Note that all inner-loop updates refer to an expected policy gradient (EPG) and all bias term we discuss hereafter is the bias w.r.f the exact meta-gradient in this K-step EPG inner-loop setting. We discuss the truncated K-step EPG setting in Appendix B. The form of exact meta- teeta-grddiett voJ (Ø) is given by the following proposition via the chain rule.
Proposition 3.1 (K (K-step Meta-Gradient) The exact meta- trat--gradiett to the objective in Eq. (2) can be written as:
FaJK
(Ø)
= JoJo (k) vva vxx
Jou 66.8%..
K-1
K-1 IT J=++
(3)
V
1=0
The derivation of the above proposition is in Appendix E.I.
For each topic mentioned in Sec. 2, we pick one GMRL algorithm example to illustrate how they can be fit into this framework. To describe gradd-t estimation in RL, we start with basic notations. Consider discrete-time finite horizon Markov Decision Process (MDP) defined by (S, p.r, y. H). At each time step r, the RL agent observes state 8, E S, takes an action a, E A based on the policy (o(a,ls) (s,))) parametrised with A E Rd, transits to the next state S,+1 E S according to the transition func- tion (s,,+1ls,, a,) and receives the reward r(s,, a,) We define the return R(T) Σ/= y' r(s,, a,) as the discounted sum of rewards along a trajectory by agent policy. The objective for the RL agent is to maximise the expected discounted sum of rewards V(0)- F--p(r:0) [R()). Then the RL agent updates parameter using policy gradient given by VoV(O) = Er-p(r:0) [Vo log no ((()) where V, log (T( = >/O Vo log πg(a, (s,). For two-agent RL problems, we can extend the MDP to two-agent MDP (or, Stochastic games [31]) P. r1,
:= (5o, ao..... H=1..SH) sampled
defined by
H), the learning objective for agent is to maximise its value function of (0) (Ex--((T:)) [R](T:)].
MAML. Finn et al. [10] optimized over meta initial parameters to maximize the return of one-step adapted policy: 01 0 + a VV)). In MAML-RI, jOun Φ 0¹) degenerates to V(01) and & and represent the same initial parameters. The meta- getaigrrdient can be derived in the form of Eq. (3): Va0 V V(O"), where vo0! = 1 a (B).
LOLA. Foerster et al. [13] proposed a new learning objective by including an additional term that accounts for the impact of the ego policy to the anticipated opponent's ' gradient update. For LOLA- agent with parameters 6 it will optimise its return over nne-step-lookaeee opponent parameters 0!. The meta-gradient can be shown as: V&Vi (Φ, 0¹ 0')+ V V (i 0¹), where Va0' = a.V. 0)
MGRL. Xu et al. [39] proposed to tune the discount factor y and bootstrapping parameter à in an online manner. The main feature of MGRI. is to conduct inner-loop RI. policy 0 update and outer-loop meta parameters & = (y.d) d) alternately. In MGRL, jour Φb. 0¹) degenerates V(0*). The meta-gradient takes the form: v V V(el), V = a aV (g.
LPG. Oh et al. [26] aimed to learn a neural network based RL algorithm, by which RL agent can be properly trained. In LPG, 0 represents the RL agent policy parameters and & is the meta- parameter of neural LSTM RL algorithm, (' 0) denotes f(6. 0), which is the output of meta- network & for conducting inner-loop neural policy gradients. The meta-gradient can be shown as:
N90' VO ((0)), V,0' VVVvVol (b.
Remark. The analytical form of exact meta-gradient given in Eq. (3) involves computation of first-order gradient In yot out Jacobian V, J and Hessian v3.1mm In practice,

---

these four quantities can be estimated by random samples from inner-loop update step 0 to K - 1, which denoted by o -0:K-1 derived as:
.22 TO:K- T3. As result, the estimated gradient v (Ø) can be . -l:K-1
(0) = V jout jK
jou ( 00 T3) + a VV Vgn
joul (8.0" .z)).
gr
K-1
(4)
Vo 0 D I
(1 0: 0')))
f=1+1
The post-adapt inner parameter estimate takes the form 0² = + a ΣK- V ft 0' :Q))
4 Theoretical Analysis of Meta-gradient Estimators
In this section, we systematically discuss and theoretically analyse the bias and variance terms for dettaggradeent estimations the current GMRL literature. We highlight two important sources of biases in veetaggadient estimations: the compositional bias and the multi-step Hessian bias. Our analysis builds on the following three assumptions:
Assumption 4.1 (Lipschitz continuity). The outer-loop objective function JOur satisfies that jour (-,0) and ouur (Φ..) are Lipschitz continuous with constants my and me respectively, m maxg ma. m2 max mæ. VaJOul (-.0) and VgJOut (Φ.-) are Lipschitz continuous with constants ue and ng respectively, ful maxe uaa, 142 max jgg. The inner-loop objective function JIn satisties that V. O)and Vo ($. :) are Lipschitz continuous with constants cæ and cæ respectively, Ci maxg ca. C2 max că. V? (o. Lipschitz continuous with constants Pé. P2 max ma.
Assumption 4.2 (Bias of estimators). Outer-loop stochastic gradient estimator VgJOut (4, 0, r) and Jot ($, 0, r) are unbiased estimator of V (Φ.0 0) and VgJOui ($.0). Inner-loop stochastic gradient estimator (g 0, r) is unbiased estimator of VoJ (. 0).
Assumption 4.3 (Variance of estimators). The outer-loop stochastic gradient estimator
V Jot (Φ,0, 0. T) and JoJOut (Φ. T) has bounded variance, i.c. V Joul (6. (T.
V Jut Out (6.-) s (o))?, and E, [lll (Φ,,,T) - - (Φ. (0.)112= s (o2)².
The above three assumptions are all common ones adopted by existing work [6, 7, 18). We futher discussed the limitation of assumptions, which are presented in Appendix
4.1 The Compositional Bias
Recall the K-step inner-loop meta- gradient estimate in Eq. (4), existing GMRL methods usually get unbiased outer-loop gradient estimator V y ($. 0* TT) and V Vax jou (t. 0* .T3),
.T)), then the algorithm can get unbiased meta-gradient estimation by plugging in unbiased inner-loop gradient estimation 0x where F|00| g However, this is not true because of the compositional optimisation structure.
Consider non-linear compositional scalar objective f((o)), ), the gradient estimation bias comes from the fact that
f(0*) = f(E[0* # ( ).
If one substitutes the non-linear function f(ek) ) with Vox (s, 0*) and Vgg (6. 0K), then typical meta praadegr estimation in GMRL introduces compositional bias:
E|V gx jout Φ. 0 Ta)] BV[ 8x (
(b. 0x) # Jon (0. )²)
(5)
B Juut ($.0 0x TT)] V Jut
Out ($. 0* # J ($. 0*)
which leads meea-gradient estimation bias. The following lemma characterises compositional bias.
Lemma 4.4 (Compositional Bias). Suppose that Assumption 4.1 and 4.2 hold, let å ( f(o*) be the compositional bias and Co the Lipschitz constant of f(), [T] denote number of trajectories used to estimate inner-loop gradient in each inner-loop update step, the learning rate, then we have,
[1] s Co (o((1 (1+ -1) dc s C - -0
T
(6)
c2VIT]
where 8m = max; v VIV jun(H. 0' T.)l iE (0... K 1).
5

---

Proof. See Appendix F.1 for a detailed proof.
Lemma 4.4 indicates that the compositional bias comes from the inner-loop policy gradient estimate, concerning the learning rate a, the sample size |T] and the variance of policy gradient estimator O'n- This is fundamental issue in many existing GMRL algorithms [13, 39] since applying stochastic policy gradient update can introduce estimation errors, possibly due to large sampling variance, therefore 0² gk also implies that the bias issue becomes more serious under the multi-step formulation since each policy gradient step introduces estimation error, resulting in composite biases.
4.2 The Multi-step Hessian Bias
Recall the analytical form of the exact meta-gradient in Eq. (3), estimating Vg0K involves computing Hessian V? Jh ($.01) 01). In Eq. (4), the Hessian term is estimated by "" TT)) where
or {I...., 1}. Hessian estimation is a non-trivial problem in GMRL, biased Hessian estimation issue has been brought up in various MAML-RL papers[23, 29, 33J. offer a brief summary in Appendix C due to page limit. Beyond MAML-RII, many recent GMRL work suffers from the same bias due applying direct automatic differentiation. For example, existing work such Eq. (3) in Oh et al. [26]. Eq. (4) in Zheng et al. [43], Eq. (12), (13), (14) in Xu et al. [39] and Eq. (5), (6). (7) in Xu et al. [40] suffers from this issue. Interestingly, most of them are coincidentally unbiased if they only conduct only one-step policy gradient update in the inner-loop. For K-step GMRL when K 1, VaOl e in meta-gradient writes as:
FOO' = Vo Jla 0'.0').
We can see from Eq. (7) that it would not involve Hessian v computation if & * 0. To further illustrate, in one-step MGRL, we can show that the estimation of VaO are unbiased because it takes derivatives w.z.f meta-parameters $ = (y. which don't 't have gradient dependency on the trajectory distribution. However, when it takes more than one-step inner-loop policy gradient updates, the meta- eeta-ggrdieet estimation will get the hessian estimation bias. As result, for the reason that when K > I in K K-step GMRL, the va0K in meta meta- gradient takes the form:
K-1
V PØOK
Σ
VoV
($. e') I (I+ (
(8)
1=0
f=t+]
This is the reason why we name by multi-step Hessian bias.
4.3 Theoretical Varsane Analysis
Based on Lemma 4.4 and the discussion in Sec. 4.2, we can derive the upper bound on the bias and variance of the meta-gradient with K-step inner-loop updates.
Theorem 4.5 (Upper bound for the bias and the variance). Suppose that Assumption 4.1 and 4.2 and 4.3 hold. Let Jg.0 denote V on Ha. denote V2 == =IE[[V (Φ) V (.) V be the meta-gra geeaa-gradeet estimation bias, set = + ac2, Then the bound of bias hold:
* s s0 (B An) k-1 (BII - - + (K (K-1)) AI
(9)
Let (0K)²= V (V be the melu-gradies estimation variance, set V1 = (1 + accz),, V2 20² 3073) the estimation variance is given by
(0*)² s o (V1 + 82,) k-1 [E[[10* 8* =x (K (K-1)
(10)
+ V/2+ + (Vi + A2, + 0 k-1 (Vi (Z (A3 + +
+(22+
k-1
where å, = max [[]i.o]] Joall. Âh = maxe | Ho.oll (2,) (oH)²=
maxpxo V[Jaol]
maxe v[Aae]
[7]

---

Proof. See Appendix G.I for a detailed proof.
Theorem 4.5 shows that the upper bound of bias and variance consists of two parts: the first term indicates compositional bias E[ &x in Lemma 4.4, while the second term refers to the second-order estimation bias (å,, ÂH) and the variance (ōt,, Õh)). Several observations can be made based on Theorem 4.5: 1) the compositional bias exists in both upper bound of bias and variance; 2) the multi-step Hessian bias has polynomial impact on the upper bound of bias and variance; 3) most importantly, many existing GMRL algorithms suffer from the compositional bias; moreover, the Hessian bias can significantly increase meta gradient bias in the multi-step inner-loop setting.
4.4 Understanding Existing Mitigations for Meta-gradient Biases
Based on the above theoretical analysis, we now try to explore and understand how existing methods handle these two estimation biases.
Off-policy Learning. From the Lemma 4.4 and the discussion in Sec. 4.1, we know that the compositional bias is caused by the estimation error between 0 and 6f In this case, simple idea is that one can leverage off off-policy learning technique [32] to handle the compositional bias problem by reusing samples T0.-1.0 together with 7/,0 approximate 0f, we want 0 to stay close to of The intuition behind this correction is to enlarge the sample size |r| so that the compositional bias be minimised according to Lemma 4.4. Specifically, we can apply importance sampling technique to
correct the compositional bias, i.e., PT-p(r:0) EH11 no(a, (s.) (T) In practice, we also need to manually control the level of off- off-policyness to prevent the variance introduced by the importance sampling process from increasing the estimation variance conversely.
pla,(s,)
Multi-step Hessian Estimator. From the theoretical analysis in Sec. 4.3, we can know that the Hessian estimation bias can significantly increase the meta-gradient estimation bias in the multi-step inner-loop setting. Many low-bias hessian estimator have been proposed in the scope of MAML-RL. However, the effect of them have never been verified in the general GMRL context. Here we have a second look at the Low Variance Curvature (LVC) method [29], one can replace the original log-likelihood with the LVC operator in the policy gradient step. As such, the Hessian estimator
v ( 0) takes the form: VOET=p((:9) Σ Naxzia; (R where 1 is the stop-gradient IRg(a,
operation which detaches the gradient dependency from the computation graph. As shown in [29], the LVC operator will ensure an unbiased first-order policy gradient and low-biased low-variance second-org -oderr policy gradient. Essentially, this operator only corrects the meta- reta-gradient update and leaves the inner-loop gradient estimation formula untouched.
5 Experiments
In this section conduct empirical evaluation of our proposed bias analysis and the proposed methods to mitigate the biases, and our experiments cover all GMRL fields listed in Table 1. In particular, conduct a tabular MDP experiment to show the existence of two biases discussed in Sec. using MAML [10] and LIRPG [42]. In order to show how the proposed methods can mitigate the compositional bias, consider Iterated Prisoner Dilemma (IPD) problem and LOLA [13] with off policy corrections in Sec. 4.4; Similarly, we conduct evaluations on Atari games using MGRL [39] with LVC corrections in Sec. 4.4 mitigate the Hessian estimation bias. We open source our code at https:/ /gggttt...b... com com/Benjiiccceeeee amij eecs wwweeeeeeeeeeeeeeeeeeee whw.oeeiial--eeeeeee
5.1 Investigate the correlation and bias of meta-gradient estimators
Firstly, we conduct experiments study different meta-gradics estimators using tabular example using MAML and LIRPG. To align with existing works in the literature, we adopt the settings of random MDPs in [33] with the focus on meta-gradient estimation. We refer readers to Appendix I.I.1 for more experimental details.
MAML-RL. Recall that the meta veta-ggaaieet in MAML is V (kt(())) To control the effect of gradient
estimations, we use the three estimators to estimate following three terms: (I) inner-loop policy gradient Vn (II) Jacobian/Hessian (0);; (III) outer-loop policy gradient V JOu (0k Refer to Appendix 1.1.2 for the implementation of decomposing meta- deta-ggrdieet estimation with different estimators.
7

---

(a) MAML-LVC
(b) MAML-AD
(c) LIRPG-LVC
(di LIRPG-AD
1.5
1.
115
640
1.50
SEC
SEC
16
E5c
E5c
64
54
55c
125
126
Samples (Log Scale)
46
Samples (Log Scale)
40 10
12
Step
Step
Figure 1: (a, b) Ablation study on sample size and estimators in MAML-RL. "S" is for stochastic estimation while "E" is for exact solution. AD refers to automatic differentiation. (c.d) Ablation study on sample size, steps and estimators in LIRPG.
To show how the estimation is biased, we use a stochastic estimator denoted as S, and exact analytic calculator denoted as for all three derivative terms. Thus, we can have 7 valid permutations in the experiment to validate the estimation, where the rest EEE estimator is the exact gradient.
Firstly, conduct our ablation studies by comparing the correlation between of meta-gradient with the exact one. The correlation metric, which determined by bias and variance, can show how the final estimation quality is influenced by these two bias terms. As the quality of stochastic estimators vary from many factors, we conduct this ablation study under extensive combinations of estimation algorithms (including DiCE [12], Loaded-DiCE [8], LVC [29], and pure automatic differentiation in original MAML [10]), learning rates, sample sizes, .etc. Due to the page limit, the results illustrated in Fig. ((abb) only include ablation study on sample size using LVC/automatic differentiation and estimation using exact gradients for (III), the outer-loop policy gradient. For the rest ablation study and more evaluation metrics (variance of estimation), refer Appendix 1.1.3.
We start our evaluation by increasing the sample size. For simplicity, we only conduct one inner step here. In Fig. (a), we leverage the LVC [29] estimator. We can that correct inner-loop policy estimation and/or a Hessian estimation can significantly improve the estimation quality (as SEE ESE SSE in low sample size case). By increasing the sample size, we between them shrinking, which verifies the finding in Lemma 4.4. In this case the compositional bias correction shares the same importance with the Hessian bais correction (SEE ESE). We also compare it with the original yet biased gradient estimator of MAML in Fig. (b)), in which SEE ESE in all sample size settings. In fact, since the gradient estimation biased, only SEE achieves near 1.0 correlation provided sufficient samples, again confirms the importance of Hessian bias corrections in Sec. 4.2.
the gap
Comp-bias: Step
(b) Comp- blas; Learning
Comp-bias: Sample Size
(D) Hessian
con Laa scato)
Bias (Log Scale)
Blas (Log
In
Step size
LLaaming Rate
Sample size (Log scale)
Hessian Bias Coef
Figure 2: (a, b, c) Ablation study of meta- getaigga bias due to the compositional bias in different estimators, step sizes, learning rates. Loaded-DiCE, LVC and AD achieve exactly the same composi- tional bias because they have the same first-order gradient, (d) Ablation study of meta-gradient bias due to the Hessian bias in different learning rates and Hessian bias coefficients.
Beside the correlation result above, we also add additional experimental results in Fig. 2 over the pure meta-gradient bias term introduced by compositional bias and Hessian bias. It can also be regarded as an empirical verification of our Lemma 4.4 and Theorem 4.5. In Fig. 2 (a, b. c), we mainly study How (a) the inner-loop step size, (b) learning rate and (c) sample size influence final meta-gradient bias. successfully validates our Lemma 4.4 (O(Kx Hna|r| about the exponential impact from the inner-loop step K (approximately linear relationship between log-scale bias and step size (a)), the polynomial impact from the learning rate a (approximately Concave downward relationship between log-scale bias and learning rate in (b)) and the polynomial impact
8

---

(a) LOLA-DICE
(b) Ablation: Compositional Bias
(c) Off-polliyy LOLA-DiCE
-14
-32
Return
Return
Return
-04
Inner, 128, Outor 128
policyyyyyy off peoomppppp. pon, hessian on_comp_off hesslan
iener 1024 128
hessian 128
ineer_ peart_ Duter_128
-14
comp_1024 hessian 128
inner orer_exact_ton wwt.ccatt_ttterccmm Outer_eract
comp exact_hessian.... 128
Iterations
Iterations
Iterations
B legend means we use A samples to estimate inner-loop gradient while B samples estimate outer-loop gradient. The 'exact' means we use analytical solution of policy gradient instead of estimation.
Figure 3:
Experiment result of LOLA-DiCE over 10 seeds. The
from the sample size a (approximately negative linear relationship between log-scale bias and log- scale sample size in (c)). In the second MAMLL-essian experiment, conduct experiments to verify the polynomial impact O(K 1)(A)) on the meta-gradient bias introduced by the multi-step Hessian estimation bias &Ø (The Concave downward relationship in (d)). In our implementation, we manually add the Hessian bias error into the estimation and control the quantity of it by multiplying different coefficients.
LIRPG. In this setting, we follow the algorithm of intrinsic reward generator presented in [42]. In tabular MDP, have additional meta intrinsic reward matrix d. Starting from 0º, the inner-loop process takes policy gradient based on the new reward matrix Rnew R &: gitl 0' *'VVVi 1 ""00'', ..). {0, 1...K 1}. The meta-gradient estimation of the intrinsic reward matrix VgJOu((0 is needed in this case. Note that in the outer loss R so the outer loss is (Ou()*) rather than jou (, 0*). Compared with MAML-RL, the object of meta-update (intrinsic matrix) and the object of inner-update (policy parameters) are different, which help us identify the problem mentioned in Sec. 4.2.
the original reward matrix
In this case, we choose the LVC and AD estimator. We conduct ablation study on inner-step and sample size shown in Fig. I(c,d). With more sample size and less step size, the correlation increases for both estimator. Two important features are: (1) With 1-step inner-loop setting, both estimator performs similarly in the correlation. (2) With multi-step inner-loop setting, LVC based estimator still reach relatively high correlation while MAML-biased estimator directly reaches low correlation after 5- -tspp inner-loop. The phenomenon shown here corresponds exactly to the Hessian estimation issue we discuss in Sec. 4.2 and the bias issue will be more severe with multi-step inner-loop setting.
5.2 Compositional bias/off-policy learning in LOLA
In this subsection, we conduct three experiments Iterated Prisoner Dilemma (IPD) with the LOLA algorithm to show: (1) The effect brought by different inner/outer estimators. (2) The effect brought compisitional bias (Sec. 4.1) (3) How off-policy correction (Sec. 4.4) can help the LOLA algorithm. Please refer to Appendix 1.2.1, 1.2.2 for more experimental setting and results.
Ablation on LOLA-DiCE inner/outer estimation. We report the result of conducting ablation study for different inner/outer-loop estimation of LOLA-DiCE in the Fig. 3(a). Here the inner-loop esti- mation refers to "
( Φ 0) while outer-loop estimation refers to /Ot 0!) and vg. ($, 0¹).
The return shown in Fig. 3(a) reveals us two findings: 1) The inner-loop gradient estimation plays an important role for making LOLA work-the default batch size 128 fails while the batch size 1024 succeeds. 2) The looe gradient estimation also crucial to the performance of LOLA-Exact. Furthermore, we continue conducting ablation studies on the inner-loop gradient update.
Ablation on compositional bias. Since the unbiased DiCE estimator is used LOLA-DiCE algorithm, there is no Hessian estimation bias in the LOLA algorithm. Thus, we mainly discuss the problem of compositional bias brought in Fig. 3(b). We also apply the implementation decompose meta-gradient estimation with different estimators. Fig. 3(b) show us the ablation study over compositional bias, which reveals that: compositional bias may decrease the performance and by adding more samples or using analytical solution, the performance can start to improve.
Sec. 5.1 to

---

BeamRRide
Assault
VORL MictL- 2.0nC
MONL MON--3 LMC
NORL MONL-S-DCC
MORLS PIGTL-S-00 UPC
MORL PI 2-11C
cigy Aeigy
20
Updatesik)
Updatee00
Updatesti)
Updatesik)
Updatesskt
Figure 4: Experimental results on Atari game over random seeds.
Off-policy DiCE and ablation study. We use the off off-policy learning to conduct inner-loop update and keep the outer-loop gradient same as before. By combing DiCE and off poolic learning, we
Ty Talat, mo (s?)
have off-policy DICE JOFF-DICE, E XH11 I[
R₁ where $, 0 refer to the in (a)
current policy, μl, H22 refer the behaviour policy for agent and agent 2, respectively. H is the trajectory length and R refers to the reward for agent. Note that the off off-policy DiCE here can not only lower the compositional bias by lowering the first-order policy gradient error, but also helps lower the Hessian variance theoretically. By the decomposition trick, we conduct experiments by traversing over all learning settings for Comparisons between different settings verify that off off-policy DiCE can increase performance by either lowering the compositional bias, the hessian variance, both.
which shown Fig. 3()).
5.3 Multi-step Hessian correction on MGRL
Finally, conduct experiment over MGRL [39]. See Appendix 1.3.1 and I.3.2 for experimental settings. When applying the LVC estimator in MGRL, we get the new inner-loop update equation:
E7-p(r0) Σ Vazsla, is.) (g)(T) vo(ss)) where o (y, refer to meta-paramters and refers the RL policy, (T) denotes d-return, (o(5;) denotes value prediction. We conduct experiment on eight environments of Atari games. We follow previous work [4] to use the "discard" strategy in which conduct multiple virtual inner-loop updates for meta gradient estimation. This strategy designed to keep the inner learning update unchanged.
Lng
In Fig. (4), we show five environments comparing three variants of algorithm: 1) Baseline Advantage Actor-critic(A2C) algorithm [25]; 2) 3-step MGRL + A2C; 4) 3-step MGRL + A2C + LVC correction. The "3-step" means we take 3 inner-loop RL virtual updates for calculating meta-gradient. Refer to Appendix I.3.3 for experimental results on all eight environments. Compared with 3-step MGRL, the MGRL with LVC correction can substantially improves the performance, which validates the effectiveness of the multi-step Hessian correction in Sec. 4.4 for handling meta-gradient estimation bias and bring in better hyperparameter- tuning in RL. Note that the fact that A2C algorithms can achieve better results compared with 3-step MGRL is consistent with the results [39].
6 Conclusion
In this paper, we introduce a unified framework for studying generic meta-gradient estimations in gradient-basee Meta-RL. Based on this framework, we offer two theoretical insights that 1) the compositional bias has upper bound of OKKa o ( with respect to the inner-loop update step K, the learning rate a, the estimate variance &f and the sample size [T], and 2) the multi-step Hessian bias ÅS has polynomial impact of O((K- 1)(âr))-I)) To validate our theoretical discoveries, we conduct a comprehensive list of ablation studies. Empirical results over tabular MDP, LOLA-DiCE and MGRL validate our theories and the effectiveness of correction methods. We believe our work can inspire more future work on unbiased meta peeti-gradeent estimations in GMRL.
Acknowledgements
We would like to thank Yunhao Tang for insightful discussion and help for tabular experiments.
10

---

References
[1] Maruan Al-Shedivat, Trapit Bansal, Yuri Burda, Ilya Sutskever, Igor Mordatch, and Pieter Abbeel. Continuous adaptation via meta-learning in nonstationary and competitive environments. arXiv preprint arXiv: /7 10.03644, 2017.
[2] Atilim Gunes Baydin, Robert Cornish, David Martinez Rubio, Mark Schmidt, and Frank Wood. Online learning rate adaptation with hypergradient descent. arXiv preprint arXiv:1703.044 040722 2017.
[3] Sarah Bechtle, Artem Molchanov, Yevgen Chebotar, Edward Grefenstette, Ludovic Righetti, Gaurav Sukhatme, and Franziska Meier. Meta learning via learned loss. In 2020 25th Interna- tional Conference on Pattern Recognition (ICPR), pages 4161-4 1661-168. IEEE, 2021.
[4] Clément Bonnet, Paul Caron, Thomas Barrett, Ian Davies, and Alexandre Laterre. One step time: Pros and cons of multi-step meta-gradient reinforcement learning. arXiv preprint arXiv:2111.00200 2021.
[5] Yan Duan, John Schulman, Xi Chen, Peter Bartlett, Ilya Sutskever, and Pieter Abbeel. RI: Fast reinforcement learning via slow reinforcement learning, arXiv preprint arXiv: (611.026170777 2016.
[6] Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. On the convergence theory of gradient- based model-agnostic meta-learning algorithms. In International Conference on Artificial Intelligence and Statistics, pages 1082-1092. PMLR, 2020.
[7] Alireza Fallah, Kristian Georgiev, Aryan Mokhtari, and Asuman Ozdaglar. On the conver- gence theory of debiased model-agnostic meta-reinforcement learning. Advances in Neural Information Processing Systems, 34:3096-3107, 2021.
[8] Gregory Farquhar, Shimon Whiteson, and Jakob Foerster. Loaded dice: trading off bias and variance in anyer score function estimators for reinforcement learning. arXiv preprint arXiv:1909.105: 10549, 2019.
[9] Xidong Feng, Oliver Slumbers, Ziyu Wan, Bo Liu, Stephen McAleer, Ying Wen, Jun Wang, and Yaodong Yang. Neural auto-curricula two-player zero-sum games. Advances Neural Information Processing Systems, 34, 2021.
[10] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adapta- tion of deep networks. In International Conference on Machine Learning, pages 1126-1135. PMLR, 2017.
[11] Jakob Foerster, Richard Y Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch. Learning with opponent-learning awareness. Proceedings of the 17th International Conference Autonomous Agents and MultiAgent Systems, pages 122-130. 2018.
[12] Jakob Foerster, Gregory Farquhar, Maruan Al-Shedivat, Tim Rocktäschel, Eric Xing, and Shimon Whiteson. Dice: The infinitely differentiable monte carlo estimator. In International Conference Machine Learning, pages 1529-1538. PMLR, 2018.
[13] Jakob N Foerster, Richard Y Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch. Learning with opponent-learning awareness, arXiv preprint arXiv: 1709.04326 2017.
[14] Luca Franceschi, Michele Donini. Paolo Frasconi, and Massimiliano Pontil. Forward and reverse gradient-bart based hyperparameter optimization. In International Conference on Machine Learning, pages 1165-1173. PMLR, 2017.
[15] Haotian Fu, Hongyao Tang. Jianye Hao, Chen Chen, Xidong Feng, Dong Li, and Wulong Liu. Towards effective context for metaareeinforeement learning: an approach based on contrastive learning, arXiv preprint arXiv:2009. 2020.

---

[16] Abhishek Gupta, Russell Mendonca, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Meta- reinforcement learning of structured exploration strategies. Advances in Neural Information Processing Systems, 31:5302-5311 2018.
[17] Yujing Hu, Weixun Wang, Hangtian Jia, Yixiang Wang. Yingfeng Chen, Jianye Hao, Feng Wu, and Changjie Fan. Learning to utilize shaping rewards: A new approach of reward shaping. Advances in Neural Information Processing Systems, 33, 2020.
[18] Kaiyi Ji, Junjie Yang, and Yingbin Liang. Multi-step model-agnostic meta-learning: Conver- gence and improved algorithms. CoRR, abs/2002.07836, 2020.
[19) Dong-Ki Kim, Miao Liu, Matthew Riemer, Chuangchuang Sun, Marwa Abdulhai, Golnaz Habibi, Sehastian Lopez-Cot, Gerald Tesauro, and Jonathan P How. A policy gradient algorithm for learning to learn in multiagent reinforcement learning. arXiv preprint 1rXiv:2011.00385 2020.
[20] Louis Kirsch, Sjoerd van Steenkiste, and Juergen Schmidhuber. Improving generalization in meta reinforcement learning using learned objectives. In International Conference on Learning Representations, 2019.
[21] Ilya Kostrikov. Pytorch implementations of reinforcement learning algorithms. https://
gi gitbbb.oo wwwc wwto///kkssrrooo//yooeoooo
ali- aacc-po--ccken www-t-ckttraaa gail, 2018.
[22] Alistair Letcher, Jakob Foerster, David Balduzzi, Tim Rocktäschel, and Shimon Whiteson. Stable opponent shaping in differentiable games. International Conference on Learning Representations, 2018.
[23] Hao Liu, Richard Socher, and Caiming Xiong. Taming maml: Efficient unbiased meta- reinforcement learning. In International Conference on Machine Learning, pages 4061-4071. PMLR, 2019.
[24] Jingkai Mao, Jakob Foerster, Tim Rocktäschel, Maruan Al-Shedivat, Gregory Farquhar, and Shimon Whiteson. A baseline for any order gradient estimation in stochastic computation graphs. In International Conference on Machine Learning, pages 4343-4351. PMLR, 2019.
[25] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforce- ment learning. In International conference on machine learning, pages 1928-1937. PMLR. 2016.
[26] Junhyuk Oh, Matteo Hessel, Wojciech M Czarnecki, Zhongwen Xu, Hado van Hasselt, Satinder Singh, and David Silver. Discovering reinforcement learning algorithms. arXiv preprint arXi::2007.0874. 2020.
[27] Kate Rakelly, Aurick Zhou, Chelsea Fino, Sergey Levine, and Deirdre Quillen. Efficient off-policy metu-reinforrement learning via probabilistic context variables. In International conference machine learning, pages 5331-5340. PMLR, 2019.
[28] Jie Ren, Xidong Feng, Bo Liu, Xuchai Pan, Yao Fu, Luo Mai, and Yaodong Yang. Torchopt: An efficient library for differentiable optimization. CoRR, abs/2211.000 106934, 2022, doi: 10.48550/ arXiv:221 1. .69944 URL https: //doi org/100 48550 arxiv. .2w.i.2211............1211111111 069344.
[29] Jonas Rothfuss, Dennis Lee, Ignasi Clavera, Tamim Asfour, and Pieter Abbeel. Promp: Proximal meta-policy search. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[30] Jürgen Schmidhuber. Evolutionary principles in self-referential learning, or on learning how to learn: the meta-meta-.. hook. PhD thesis, Technische Universität München, 1987.
[31] Lloyd S Shapley. Stochastic games. Proceedings of the national academy of sciences, 39(10): 1095-1100 1100, 1953.
[32] Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
12

---

[33] Yunhao Tang, Tadashi Kozuno, Mark Rowland, Remi Munos, and Michal Valko. Unifying gradient estimators for metu-reinforrement learning via off- off-policy evaluation. Advances in Neural Information Processing Systems, 34, 2021.
[34] Vivek Veeriah, Matteo Hessel, Zhongwen Xu, Richard Lewis, Janarthanan Rajendran, Junhyuk Oh, Hado van Hasselt, David Silver, and Satinder Singh. Discovery of useful questions as auxiliary tasks. arXiv preprint mexivv/9909 0e0e.9999700,, 2019.
[35] Vivek Veeriah, Tom Zahavy. Matteo Hessel, Zhongwen Xu, Junhyuk Oh, lurii Kemaev, Hado van Hasselt, David Silver, and Satinder Singh. Discovery of options via meta-learned subgoals. arXiv preprint arXiv:2102.066 102.04441, 2021.
[36] Risto Vuorio, Jacob Austin Beck, Gregory Farquhar, Jakob Nicolaus Foerster, and Shimon Whiteson. No DICE: An investigation of the Dirs-variance tradeoff in meta- eeta-graddents. In Deep RL Workshop NeurIPS 2021, 2021.
[37] Jane X Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. arXiv preprint arXiv: 1.016676110,,,, 2016.
[38] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Machine learning, 8(3-4):229-22 226, 1992.
[39] Zhongwen Xu, Hado van Hasselt, and David Silver. Meta-gradient reinforcement learning. arXiv preprint arXiv: 1805. 09800, 2018
[40] Zhongwen Xu, Hado van Hasselt, Matteo Hessel, Junhyuk Oh, Satinder Singh, and David Silver. Meta-gradient reinforcement learning with an objective discovered online. arXiv preprint arXiv:2007.08433 2020.
[41] Tom Zahavy, Zhongwen Xu, Vivek Veeriah, Matteo Hessel, Junhyuk Oh, Hado van Has- selt. David Silver, and Satinder Singh. self-tuning actor-critic algorithm. arXiv preprint MeXi::0002112988 2020.
[42] Zeyu Zheng, Junhyuk Oh, and Satinder Singh. On learning intrinsic rewards for policy gradient methods. Advances Neural Information Processing Systems, 31:4644 4654, 2018.
[43] Zeyu Zheng, Junhyuk Oh, Matteo Hessel, Zhongwen Xu, Manuel Kroiss, Hado Van Hasselt, David Silver, and Satinder Singh. What can learned intrinsic rewards capture? In International Conference on Machine Learning, pages 11436-1 1446. PMLR, 2020.
[44] Wei Zhou, Yiying Li, Yongxin Yang, Huaimin Wang, and Timothy Hospedales. Online meta- critic learning for off-policy actor-critic methods. Advances Neural Information Processing Systems, 33, 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Appendix B.
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Sec. 4
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix F and Appendix G
3. If you experiments...
13

---

(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]. See Appendix
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes], See Fig. 1, 4.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix I
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you't 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "aRdu14kdp5q", "reviewer": "Reviewer_4HrY", "summary": "This paper studies the bias of gradient-based meta-RL algorithms. The authors provide a bias bound that is made up of a compositional bias and the multi-step Hessian estimation bias, which also suggests ways of understanding current algorithms. ", "questions": "Could the authors run the experiments for more iterations? I would also like to see if both sources of bias can be combined together or will they suggest new algorithmic designs. Besides, more related work on the theory side is encouraged. ", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "pros: 1. The paper is well written and easy to follow. Both the background and the analysis are thorough. \n\n2. The bias theorem may also guide future algorithmic designs of meta-RL algorithms, considering where the bias comes from.\n\n3. The experiments also provide evidence supporting the theorem.\n\ncons: 1. In experiments Fig. 2 and Fig. 3, it seems that the curves have not converged.\n\n2. I'm not an expert in meta-RL. Are there other existing works that give bias bounds? I am only aware of the algorithmic designs mentioned in the paper, but I didn't see the related work in the theory aspect.\n\n3. It seems that the authors only evaluate existing modifications to support the theorem instead of proposing new algorithms. Are there ways to combine some of them together to reach the best theoretical and experimental results? ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Zobs-aLt9cd", "reviewer": "Reviewer_UkbE", "summary": "Gradient-based meta-learning (GMRL) has been a topic of interest in the NeurIPS community recently. The authors derive formal bounds on the bias and variance of the estimated meta-gradient in commonly-used algorithms for gradient-based meta-reinforcement learning, identifying two sources of meta-gradient bias in particular: \"compositional bias\" (accumulated across inner loop steps estimated with small batch sizes) and \"multi-step Hessian bias\", caused by using approximate adaptation in the inner loop. Using these observations, the authors applying existing techniques for variance reduction to new GMRL settings, showing that they can reduce bias and improve GMRL performance in some settings.", "questions": "How specific are the conclusions in this paper to gradient-based meta-RL? To what extent are they useful for gradient-based supervised meta-learning?\n\nIdeally, the authors might evaluate horizons that are deeper than only 5 gradient steps. Does LVC continue to provide effective mitigation at greater depth?\n\nMost importantly, the experimental evaluations are relatively limited. It's personally difficult for me to judge the extent of the impact of the theoretical results alone, and I therefore would have found it quite interesting to consider other methods than just LVC. Could the authors justify why LVC is a particularly representative method to use for most of the evaluations?", "limitations": "More discussion of the limitations of the theoretical results would be welcome (e.g., how realistic are assumptions 1-3?).", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Analysis of a relevant topic of interest to the NeurIPS community (gradient-based meta-RL)\n- Intuitive theoretical result motivating application of existing techniques to new meta-RL settings\n- Overall clear presentation\n\nWeakness:\n- Unclear what the extent of the impact of the theoretical results are\n- Little methodological novelty (which could be okay)\n- Experimental evaluations are okay, but the choice of methods to evaluate seems arbitrary and perhaps limited (see final question below)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "WmlKCbuRv1v", "reviewer": "Reviewer_H2bz", "summary": "The paper presents a theoretical analysis of two sources of bias in the gradient estimations of meta-reinforcement learning methods that perform a two level optimization of an outer meta-learner and an inner gradient-based RL learner. The authors derive an upper bound to the first source of bias, named compositional bias as well as an analysis of the bias of the estimated Hessians coming from the use of auto-diff systems. Finally, the paper presents an empirical evaluation in tabular MDPs that supports the theoretical findings.", "questions": "Given the previous doubts on the compositional, I would ask the authors to:\n1) Prove that the compositional bias is actually strictly greater than 0.\n2) Comment on my second concern, related to the fact that this in not actually a bias, but a finite-sample error that goes to zero with more samples\n3) How does the correlation plotted in Figure 1 relate to the compositional bias identifies. It seems to me again, that the plotted measure is related to MSE instead of the bias, reinforcing my doubts on the analysis performed.", "limitations": "The authors have discussed their limitations", "rating": 6, "confidence": 3, "soundness": 1, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "=========Update after Rebuttle====\nMy main concern on the validity of the 2 sources of bias has now been addressed and I agree that the bias terms exists and are strictly higher than 0. For this I will gladly raise my score from 3 to 6.\n\nThe paper proposes a general study on gradient estimation meta-reinforcement learning.\nWhile the setting itself is interesting to the community I have some major concerns regarding the theoretical results which are listed as follow.\nA) On the gradient expression. (Proposition 3.1)\nI have some concerns regarding the correctness of the proof.\nIn particular, I would ask the authors to derive step by step  Equation 18 as according to my computation, the total derivative has been simplified with the gradient, which, in general is not correct.\nB) On the compositional bias. (Section 4.1)\nTo the best of my understanding, the authors did not show that this term actually exists, instead they provide an upper bound on it (i.e., one might take a general unbiased estimator and upper bounds its bias with any positive number).\nThe second concern regarding the compositional bias is the fact that, as Lemma 4.4 shows, the compositional bias is upper bounded by the empirical variance divided by the number of trajectories taken to estimate the inner loop gradients.\nThis term tends to zero as the number of trajectories tends to infinity. Similar results holds (in high probability) for standard policy gradient setting (see for instance \"Adaptive Batch Size for Safe Policy Gradients\" (Papini 2017) or \"Smoothing Policies and Safe Policy Gradients\" (Papini 2019)): the error (not bias) in the gradient comes from a term that depends on the variance and the number of samples used (this can also be seen as a consequence of the Chebyshev inequality). This seems very similar to what the authors claim. In this sense, I am a bit confused on the value of Lemma 4.4 (which has a direct impact on the main theoretical result). More specifically, it seems to me that rather than a bias we are just considering an error in the gradient estimation (that, of course, occurs with finite samples).\n\nC) On the multi-step hessian bias. (Section 4.2)\nIt is not clear to me where this bias comes from and seems simply to be a bug of differentation software. The discussion of Section 4.2 is  unclear to me.\n\nD) On the trade-off (Section 4.3)\nGenerally speaking, Theorem 4.5 seems to be auto-referential.\nThe contribution of multi-step hessian bias is direct: it just states that this error exists and it is accounted in the computation.\nThe same holds for the compositional bias.\nRegarding the variance, instead, the results seems incredibly loose since it does not tend to 0 (V1 * (K-1)) (more than a worst case solution).\nMoreover, generally speaking, Theorem 4.5 shows very little insights and its contribution seems to be minor w.r.t. Lemma 4.4.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "K5Sml_K8vqs", "reviewer": "Reviewer_RjbJ", "summary": "The paper comprehensively investigates the gradient bias in Meta-Reinforcement Learning. The authors point out two source of bias: the compositional bias and the multi-step Hessian bias. The paper then gives a generic bound on the bias and variance of the final meta-gradient. The paper mitigates the two biases using off-policy training and LVC estimator respectively. A debased algorithm is tested on 8 Atari games.", "questions": "Line 52: not clear what does the impact mean here \nLine 204 typo: an extra \"the\"\nLine 216: It seems there are multiple bias terms in (9). Why is $\\hat \\Delta_J$ less important than $\\Delta_C$ and $\\Delta_H$? It would be better to use $\\Delta_C$ instead of $E[\\|\\hat \\theta^K - \\theta^K\\|]$ here as it has already been defined.\nLine 218: What does the variance term imply here? Is there any algorithm in the literature that can balance bias and variance? How well do they work in the decomposition in (9)?\nLine 219. The use of punctuation is bad here. What set is the maximization taken over?\nLine 318. What is a DiCE estimator?\n", "limitations": "The authors did not address any potential negative societal impact. As a theoretical paper, I don't think it is necessary.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nI believe the paper is significant in the following ways\n1. This is one of the first paper to give a generic analysis on gradient bias, which applies to most of the algorithms.\n2. The GMRL framework proposed in the paper applies to a number of important meta-RL frameworks.\n3. The analysis that decomposes the bias terms is novel and interesting.\n\nWeakness:\n1. I am concerned about the actual contributions of the paper. The paper provide theoretical analysis that decompose the gradient bias into several terms. However,  the experiments are only weakly dependent on the theories.\n\nTo be specific, the paper adapts off-policy training to mitigate the compositional bias. Enlarging the number of data to mitigate bias is a trivial solution opposed to actual bias correction. As the paper mentioned, off-policy training suffers the shifts in the data collection policies and importance sampling process has to be used. LVC is a method existing in the literature. Thus, the paper is not able to propose a novel algorithm that actually balances the bias and variance based on Theorem 4.5 to achieve a better performance. \n\nThe experiments in 5.1 simply show that the performance is higher when replacing sample-based estimator with exact solution, which is quite obvious in my opinion.\n\nThe paper does not clearly state the main contribution over the literature. There are papers [33, 6] discussing bias in meta-RL. I believe [6] also considers the compositional bias. I don't understand the main contribution here except for proposing a more general framework that applies to different cases.\n\n2. The paper is not well-organized and it can be hard to follow. Different sections are not strongly connected.\n\n\n\n \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept