Distributional Meta-Gradient Reinforcement Learning
Authors: Haiyan Yin
Shuicheng YAN
Zhongwen Xu
Venue: ICLR 2023 poster (2023)
Keywords: Reinforcement Learning
Meta Learning

Abstract: Meta-gradient reinforcement learning (RL) algorithms have substantially boosted the performance of RL agents by learning an adaptive return. All the existing algorithms adhere to the same reward learning principle, where the adaptive return is simply formulated in the form of expected cumulative rewards, upon which the policy and critic update rules are specified under well-adopted distance metrics. In this paper, we present a novel algorithm that builds on the success of meta-gradient RL algorithms and effectively improves such algorithms by following a simple recipe, i.e., going beyond the expected return to formulate and learn the return in a more expressive form, value distributions. To this end, we first formulate a distributional return that could effectively capture bootstrapping and discounting behaviors over distributions, to form an informative distributional return target in value update. Then we derive an efficient meta update rule to learn the adaptive distributional return with meta-gradients. For empirical evaluation, we first present an illustrative example on a toy two-color grid-world domain, which validates the benefit of learning distributional return over expectation; then we conduct extensive comparisons on a large-scale RL benchmark Atari 2600, where we confirm that our proposed method with distributional return works seamlessly well with the actor-critic framework and leads to state-of-the-art median human normalized score among meta-gradient RL literature.

DISTRIBUTIONAL LEARNING
REINFORCEMENT
Haiyan Yin, Shuicheng Yan & Zhongwen Xu" Sea AT Lab
ABSTRACT
Meta-gradient reinforcement learning (RL) algorithms have substantially boosted the performance of RL agents by learning an adaptive return. All the existing al- gorithms adhere to the same reward learning principle, where the adaptive return is simply formulated in the form of expected cumulative rewards, upon which the policy and critic update rules are specified under well-adopted distance metries. In this paper, we present a novel algorithm that builds on the success of meta- gradient RI. algorithms and effectively improves such algorithms by following a simple recipe, i.c., going beyond the expected return to formulate and learn the return in a more expressive form, value distributions. To this end, we first for- mulate distributional return that could effectively capture bootstrapping and dis- counting behaviors over distributions, to form an informative distributional return target value update. Then we derive an efficient meta update rule to learn the adaptive distributional return with meta-gradients. For empirical evaluation, we first present illustrative example on a toy two-color grid-world domain, which validates the benefit of learning distributional return over expectation; then we conduct extensive comparisons on large-scale RL benchmark Atari 2600, where we confirm that our proposed method with distributional return works seamlessly well with the actor-critic framework and leads to state-of-the- median human normalized score among meta-gradient RL literature.
INTRODUCTION
Meta-gradient reinforcement learning (MGRL) (Xu et al. 2018;; Sutton 2022) is a family of al- gorithms that leverage the gradient of the gradient descent update learn better objectives for reinforcement learning (RL). The MGRL paradigm has achieved substantial performance break- throughs and dominated the state-of-the-art model-free RL algorithms in various domains, such as
Atari (Xu et al., 20188; Zahavy et al. 2020 Flennerhag et al., 2022), DMLab (Zahavy et al. 2020), and DeepMind Control (Zahavy et al.
2020).
The major algorithmic research on MGRL has been developing towards more and more ambitious targets. For example, it starts from learning two fundamental hyperparameters (i.e.. discount factor and bootstrapping factor A) (Xu et al., 2018b). and then goes to self-tuning 20+ hyperparameters high-performing RI. agent (Zahavy et al.) 2020). As powerful tool for discovering RL seman- tics from the data, MGRL has been utilized to discover intrinsic rewards (Zheng et al.) 2018 (020), auxiliary tasks (Veeriah et al. 2019), options (Veeriah et al., 2021). etc. Recently, has even been used to tackle the more flexible form of parameterizing the objective, i.e. using black- neural networks to learn the objectives from the environment interactions and learning context (Xu et al.) 2020 Oh et al., 2020). However, MGRL research has been limited to scalar form, where either meta-parameters are scalar ones (Xu et al.] (2018b Zahavy et al. (202)), or outputs of meta networks scalars (Xu al. 2020). This is in line with the traditional definition of returns (Sutton & Barto 1998), as an expected sum of discounted rewards. However, note that MGRL is not limited to pro- viding RL objectives, while we could also meta-learn a broader concept, e.g., learning rate (Sutton 1981 Baik et al. 2020 Pinion et al. 2021 and exploration (Gupta et al. 2018 Xu et al. 2018a).
In this work, we investigate an essential step of extending the algorithmic scope of MGRL distri- butional methods, where RL algorithms take away the expectations in Bellman updates and consider
"Corresponding author: Zhongwen Xu

---

richer formulation of value distributions (Bellemare al. (017). This is essential for RL updates and even more critical for MGRL which enjoys multiple satisfying properties. Sparse signals are hard to learn in RL, and the bi-level optimization nature of the MGRL makes it extremely diffi- cult to propagate helpful signals to learn meta-parameters. By modeling value distributions, the RL algorithm can naturally produce rich set of predictions, which provides dense and informative signals for meta-optimization. Moreover, some distributional RL methods come with new opti- mization properties in the loss form (not available in conventional non-distributional RL methods), e.g., with KL divergence between discrete distributions (Bellemare et al. 2017) or quantile regres- sion (Dabney et al. (018b). Despite the great accomplishment of distributional RL for non- non-meta policy learning, the attempt to learn a distributional meta objective has not yet been made by any prior MGRL methods.
This paper aims to develop a novel distributional meta-gradient RI algorithm that can discover meaningful value distributions online. Furthermore, we aim to both methodologically establish a novel distributional meta-gradient framework and find out the intriguing effect of applying distribu- tional meta-gradien techniques on real-world problems, for which we conduct extensive evaluations with motivating toy domain well as large-scale end end-to-end policy training problems. The eval- uation results well demonstrate the effectiveness of our method.
Overall, the contributions of our paper
follows:
(i) We present novel meta-gradient variant of the RI. method, which considers approximating the distributional return in value update.
(ii) Our proposed distributional meta- graa-gradient algorithm is general and can be compatible with almost all the existing meta-gradient RL. approaches.
(iii We conduct an extensive empirical study on toy control domain and the large-scak scale bench- mark Atari 2600. Our results have demonstrated substantial improvements in our method over strong baseline methods on the major evaluation domains.
2 RELATED WORK
In recent decades, meta-learning, or learning-to-leamm. has been explored extensively within the machine learning community (Hospedales et al.] [002]] Sutton 2022), e.g., learning neural net- work update rule (Bengio et al. [1990), adapting the learning rate (Sutton 1992, Schraudolph & Giannakopoulos 1999). and transferring domain-invariant knowledge (Thrun & Mitchell [995). Notably, it has also driven many advances in the RL problems. One direction for such attempts is to learn meta-optimizers (Andrychowicz et al. [016)] Wichrowska et al.) 2017) or meta- policies (Duan) al. 2016 Wang et al. (017)) parameterized by recurrent neural networks, which capture the high- level time-dependent information meta knowledge. Another direction is gradient-based meta- learning, introduced in Model-Agnostic Meta-Learning (MAML) (Finn et al., (017a), as well as its followup works (Finn & Levine 2018; Finn et al. 2017b; Grant et al. [2018] Al-Shedivat et al.) 2018). Those works focus on learning good initialization of the model for one or few-shot multi- task learning with meta-gradients.
In this paper, we consider solving meta-gradient RL problem from a fresh new angle, i.e., optimiz- ing the adaptive return distributions. Policy learning with distributional return has been sidered in existing RL literature for off-policy learning only. All the existing methods fall into the strand of DQN variants which replace the n-step return in DQN with distributions during Q-learning In (Bellemare et al. 2017). the return is formulated as a categorical distribution with fixed values for the base, where the Bellman operator shifts the base values of the distribution, and the target distribution is contracted and projected upon the fixed base. In (Dabney et al.) 2018b), the return is formulated as quantile distributions, where the algorithm tries to learn the values corresponding to each quantile, alleviating the need for distribution projection or value range approximation. In (Dab-) ney et al. 2018a) the quantiles are approximated by mapping from values drawn from the random distribution (proposals) into quantiles, resulting
unlimited number of implicit distributions. In Waan et al.) 2019). the proposals in implicit quantiles are replaced by the output of a learned proposal network. In contrast to the aforementioned off ool-cy distributional RL methods, we tackle a novel problem of learning adaptive distributional return in on-policy actor-critic algorithms with meta-gradient.
2

---

Compared to existing meta-gradien RL methods with scalar return method applies a distribu- tional return, which is advantageous as (1) it can offer a more informative target to approximate, (2) it naturally provides a richer set of hyperparameters to be tuned with white-box meta- leattl,, which offers more feedback in the meta-gradient, and (3) approximating such a distributional return naturally results in meaningful auxiliary tasks to facilitate more efficient policy and value training.
PRELIMINARY
Quantile Distributional Value Update In this work, we consider learning the return quantile distribution. One reason for us to choose quantile distribution rather than any other type of distri- bution is, the Bellman operator for quantile distribution does not need perform projection (lose precision) or manually define the return boundaries, as C51 (Bellemare et al. 2017)) does. Also, our proposed method employs deterministic prediction function for quantiles which could be conve- niently trained, unlike the stochastic mappings employed by IQN (Dabney et al. 2018a), where the stochasticity would bring the noise the meta gradient and confuse the training of adaptive return. We denote the distributional return function over each state as Zo(s). where 0 is its parameter set. With quantile, the distributional return can be approximated by a uniform mixture of N Dirac func- tions, i... (s)s) = N Σq= 50, (s), where 0g is the parameter for the q-th Dirac function which can be trained by quantile value update. For a given quantile distribution Z and quantile т, the value of the quantile function is denoted as F7 (T). The quantile distribution can be trained by quan- tile regression loss LT (.) which penalizes overestimation errors with weight T and underestimation errors with weight т, where [0, 1|:
CTT Æz (2 - -]],
where p=(n)= (r {(uco));
Vu E R.
To smooth the quantile loss at zero, we could apply Huber loss over it, i.e.
< chuber (u) = 2 u
if if S k: otherwise.
(2)
k (|ul k),
Thus the Quantile Huber Loss can be derived as follows,
Tr. T - S(weo)]
Siceoyl--huubee chuber (u).
(3)
White-box Meta-Gradient RL White-box meta-gradient RL methods use bi-level optimization scheme to update important high-level parameters to characterize the return and the composed loss function by meta gradient. We denote the high-level parameter set as 17. In the prior work from (Xu eet al., 2018b). 7 {y, A}; later, loss weights and importance sampling threshold have been added 7J. We denote the RI loss function [T, 0,)). During bi-level optimization, the meta-gradient methods first update the model parameter with the gradient J(() w.r.t 0:
[TT,0, = (J(r.
Of(r, A. m) = 8 J(T,0,n)
(4)
00
On
30 On
where Ç is the learning rate and the first-order gradient function f(-) is differentiable w.r.t 17 Then, an alternative batch of transitions 7' is sampled to optimize the model in the second update stage, where outer objective J' computed from the updated model parameters 0' and a reference hyperparameters set r/ is optimized (e.g., default values for ry') w.r.t vy:
aJ'(T', 0''0'''))) n1) aJ' (''(''0''')) 00' On
(5)
00
On
Since 0' is differentiable w.r.t 7, the meta gradient is computed as the gradient of the outer objective J'(r', 0', vy') w.r.t 1). Thus meta-gradient RL enables the effective high-level parameters in learning- to-learn to compute the adaptive return and/or the importance of loss function components.
4 DISTRIBUTIONAL META- GRADIETT RL ALGORITHM
Actor-Critic with Distributional X-Return The principle of RL is to optimize the agent's 's cumu- lative rewards, also known the return (denoted G), from the sequence of experiences collected
3

---

from the environment, where the experience time t id denoted as tuple 7, (S A,, R4, S(+1) with St. A, and R2 denoting the state, action, and reward for the state-action pair at step t, respec- tively. In practice, the return can be formulated in various forms. One well-adopted form of return via bootstrapping. Let T denote the terminal step and ?y the discount factor. An example n-step return (Sutton & Barto 1998) is defined as:
G (Te) = Ro+1 + RRe2 + 72 Re+3 + .... +y (((+n))
(6)
(T)) = Ro+1 + 7 - () + (T+L)
(7)
where the return is the sum of n step rewards and the bootstrapped value for St+n- To better leverage the return G2 derived over multiple steps, there is also X-return or TD(A) return, which measures the return as the weighted TD error over all time steps (Sutton 1988). where there is another high- level parameter À introduced to characterize the X-return function. When À O, it terminates the bootstrapping from further steps, and when À = 1, it promotes pure bootstrapping. White-box meta- gradient RL methods attempt meta-learn the crucial hyperparameters characterizing the X-return with a bi-level optimization regime.
We consider the training of actor-critic agents with the critic approximating quantile distributional return predicted by value function Zo(S), where S is state, 6 is the parameters of the value function, and ZS) follows the distributional Bellman optimality operator (Bellemare al. 2017).
TZ(S) 2
+ A (ISss((()))))
(8)
where U _ V denotes equality of probability laws, i.e., the random variable U is distributed ac- cording to the same law as V. Our proposed distributional adaptive return can naturally result in larger set of self-tuning hyperparameters. The critic is updated towards a canonical distributional A-return denoted as I following a distributional TD(A) algorithm,
r = Re41 + 1142(1 (441) Z(S++1) 7411441TH
20
P z(S)) + Σ A=
[II 1=1
mtsdets)
(Retk-1 + 70- (8 7(S++z))
where I is essentially the sum of the predicted return distribution Z(S) and the backpropagated TD error over all the future steps & where the later could be computed iteratively and weighted by the product of discounting factor and trace in their exponential forms. When training the actor- critic with I under a distributed regime like IMPALA (Espeholt et al. 2018) for efficient learning, the training data would become off- off-policy and we need to perform off-policy correction to get the unbiased gradient. Hence distributional X-return with off-policy correction in the form of v-trace could be defined iteratively follows,
r 2 Z(S.) +
Σ Ca+ (II f=1
00
(IToroett)) bctks where C.. Am (min(
T(Aml ((AAml)m)) (10) 4(Am S.))
where cm is the truncated importance sampling (IS) ratio with ē being the clipping threshold, and T(), sa()) are the present and previous behavior policies, respectively. To update the distributional actor-critic with off-policy correction, we optimize the following objective function:
Lz = Thr
(7,)) (sg < (S)),
L* ((TT(-))))
1 N
T(A[S)) p M(A[S)
Az =
(S) - ZT)( (min( N Σ sg (
C. = -z((T)) log TO (AIS),
Lo = Wr Lx WH Lx + wz Lz,
(11)
where Az denotes the advantage computed as the sum over quantile error of the distributional return, I'
(4) (S) denotes the g-th quantile in the distributional A-return, LzIC,/Lx and wz/w,/wH are functions and weights for the value/policy/entropy losses respectively, sg(.) the stop- gradiena function and the first term in C.r corresponds to the advantage.
4

---

Distributional Meta-Gradient Update
Our proposed distributional Meta-Gradis Gradient RL algorithm adopts a white-box bi-level optimization scheme to learn adaptive distributional return T?. We denote the meta parameters characterizing the adaptive return I and important weights in the actor- critic loss function Ly as a set 7, where = {), 7, a; WH; Wi, wz) ). where a is the weight for the leaky-vtrace in STACX (Zahavy al., (2020) control the off- off-policy correction ratio. Formally, the adaptive return can be formulated as follows,
D : R4t1 + + - ((S+1) + r2 Z(S++1) +
> (II 3=1
17
= Ret1 + (Z((4)) A> L CII md
(I 6
In other words, distributions and vectorize the À and in the set 1). Furthermore, we can also specify a vectorial quantile loss weight wz. Thus, method results in much larger hyperparameter set compared Yыk,, and 2, where nz and
consider quantile-specifi discounting and bootstrapping operator over the
with STACX, i.e., /y for DrMG, STACX and MG are 3 x nz + 3, 7 Tock represents the dimension of quantiles and the number of STACX's multi-head blocks.
At the inner loop, the inner loss ci 0 n) is first computed to update 0 towards using the gradient aci (T, 0, 1))009 where is batch of data sampled to update the inner loss and rj is drawn from the meta parameters.
[(T; 0; - (T,0,n) -(wz
- ack.
0Lz
OC, 0LH (wN
(wa
(12)
00
00
00
00
where the gradient of the distributional value loss is derived as follows,
0Lz(T,0,1)
wz (I*(S)-Ze)) (Z)(S) 00
if (T(() Zo(5)) S
00
Wz [T,(F)-2(SS]]
(,,(F)-2o(S)) (Z.
otherwise.
(13)
At the outer loop, we evaluate the high-level parameters by sampling an alternative batch of tran- sitions 7' and computing the outer loss ( 0', ry'), where '/' corresponds to some reference
hyperparameters for 7/, e.g., the default values predefined like those other algorithms. The meta gradient to update the meta parameter 77 is defined as follows.
('')0',m') - acout (T', 0', m/) = =
aCout 0',0',n)
00'
82 cin (r,
f(F':0(:n)
On
00'
On
00 dn
(14)
where ç' the learning rate for the outer loss. The meta gradient evaluated at the outer loss interacts
(7,0.1) with meta parameters through 8e' which could be further expanded a0 ""c'n"(r,0.0))
80 Bn based on the
an chain rule. With the definition of the distributional return, meta-learning variants become learning
the bootstrapping or termination condition over the distributions, rather than scalars, which would bring informative information for return approximation with a greater degree of adaptability. Our proposed Distributional return Meta-Gradient algorithm is denoted as DrMG.
5 EXPERIMENTS
K
5.1 Two COLORS
Settings We consider a x 7 gord-world, where the green-colored agent is assigned to collect the key to open the door at the yellow- colored goal. There are two types of keys; col- lecting one of them would get a reward of 0.7 and the other 0.3. The agent receives reward
K
Two colors grid world environment and the reward curve for DrMG, MG and a vanilla distribu- tional method QR without meta-gradient.
Figure I:
5

---

of +1 upon successful navigation. After episode terminates, randomly generate locations for both keys and the goal. We toggle the reward at every le5 steps to investigate the agent's adaptation ability. Thus a key-agnostic optimal agent could collect +1. ppn *T expected return, where Tpen and 7 are the step-wise penalty and episode horizon. We assume that the agent approximating the distributional return should be able encode the dynamics in its return that it can adapt better compared to the baseline agent approximating the scalar return.
Results We compare our proposed DrMG agent with a meta gradient baseline algorithm MG as well as vanilla distributional algorithm QR. where QR an IMPALA variant which updates a non-adaptive distributional return D The results are shown in Figure [ We configure DrMG, MG, and QR agents by almost identical hyperparameter sets. Since the RL agents are unaware of the reward toggling during policy training. their adaptation ability is characterized by their efficiency in modeling/approximatig the return during the training process. We notice that after the reward toggle, DrMG achieves peaks with substantially higher return compared to that for MG and QR. Furthermore, QR slightly outperforms MG, which shows that approximating distributional return benefits the task adaptation.
5.2
ATARI 2600 200M
Settings We testify our method on Atari 2600 Benchmark (Bellemare al. 2013) under the 200M setting, where the setting aligns with prior works (Xu et al. [018b] Zahavy et al. 2020). Our agent developed upon IMPALA (Espeholt et al., 2018), an efficiett distributed actor-critic framework. The actor-critic module employs a deep ResNet (He et al. forward model without LSTM. The network takes grayscale images with size 84 x 84 as input. The backbone is connected to policy-value head where the value head outputs distributional quantiles with a dimension of N and the policy head outputs policy logits with a dimension of [.A]. The meta module is joint set of scalars and vectors, with each vector entry representing distributional hyperparameter in 7/-
2016) as the backbone, which is a feed-
For training, we strictly follow the 200M regime without extra data experience reuse. During training, we sample two batches with identical batch sizes and use them to form two combinations of inner data and outer data to compute the meta-gradient update loop twice. Each task runs with identical hyperparameters and device configurations for hardware and software. We highlight that DrMG brings very little computational overhead; namely, DrMG only brings IN 1| 256 more parameters compared to MG and it is more computationally efficient than STACX since STACX introduces multiple policy-value heads as its auxiliary task components. All the games employ the same wrapper, for which the action repeat is 4, stick action probability is O, episodic life is false, and the maximum episode length is 108,000. All the tasks adopt the 'no-ops starts' protocol, where at the start of each episode, the agent randomly samples period to take a dummy action '0' up to 30 steps. We present the environment wrapper setting Sec. A.1.2 and the detailed model/experiment configurations in Sec. A.2 For comparison, we evaluate the performance
each game in terms of
Hours Rours-"rand. where Rours/ Rrand/ F human
human normalized score (HNS) defined as HNS
human " "rand"
are the episode rewards for our/random/human expert policies.
Results For evaluation over the 57-task bench- mark, we compute the median HNS for each method. We consider the following five most re- lated baselines: C51, QR-DQN, Implicit Quan- tile Network (QQN), Self-Tuning Actor-Critic with auXiliary tasks (STACX) and MG. The first three are off-policy value-based methods which are DQN- variants, while STACX and MG are white-box meta- gradient RL baselines that are IMPALA-variante the same DrMG. There is another recently published meta-gradient RL work Bootstrapped Meta-Gradient (BMG), which is excluded here for comparison be- cause the work employs a different Atari 2600 bench- mark with altered task and model settings, motivated
40
DrMG
CS1
QR-DON
IQN
MG
2.5
15.
1.0
0.0
Figure 2: The learning curve in terms of me- dian HNS evaluated on DrMG and its baselines.

---

1060
318
1000
3404
Breakout (fast)
DemonAttach (fast)
(c) Gopher (medium)
1254
560
1500
30-
1009
33
(d) Kangaroo (medium)
(e) Robotank (medium)
(f) Spacelnvaders (medium)
DrMS
180
45%
335
450
BeamRider (slow)
(h) Centipede (slow)
(i) Zaxxon (slow)
Figure 3: Reward learning curves for our method and meta-gradient RL baselines MG and STACX evaluated on a representative set of Atari games. Overall, DrMG results in stable and efficient progress in reward learning across all the fast / medium / slow groups. Its median HNS score on Atari 2600 benchmark is 379%.
solve orthogonal problems than our work. The detailed explanation presented in Ap- pendix A.1.2 We show the learning curves for median HNS in Figure 2 We do not present the curve for STACX in Figure 2, because the performance of our carefully reimplemented STACX drops quite lot compared to the score in its original paper (even the reimplementation adopts an identical framework, hyperparameters, and model architectures, due to the notorious reproduction issue for Atari 2600 200M series of methods with missing official releases of the baselines experi- mental code). Nevertheless, the learning curves for STACX on representative games can be found in Figure
From the median HNS curves shown in Figure 2 we can first observe that DrMG performs sub- stantially better than the conventional distributional RL methods C51, QR-DQN and IQN, which demonstrates updating the distributional return adaptively with meta-gradient and the actor-critic framework more effective than updating the distributional returns by off-policy DQN variants. We also find that DrMG brings noticeable performance improvement over the strong white-bon -ox meta-gradien RL method MG. The median HNS for DrMG 379% and that reported MG is 287%. These results show that optimizing the distributional return in our proposed X-distributional form more efficient than learning the sealar return.
We present the learning curves of our method, MG and STACX on set of representative Atari games in 3 For computational efficiency, the training adopts an early stopping policy, where the training terminates after the agent surpasses fixed HNS standard (450% in our evaluation) for 22 steps. We classify the games into three strands [fast, medium, slow), based on the behav- ior of the RL agents to hit the designated HNS standard, where fast=(Breakout DemonAnack), Robotank, Gopher, Kangaroo} and slow [BeamRider, Zaxxon, Cen- t. Games in fast, medium and slow groups consume < 10%, 30% 60% and ~200 frames
medium= medium=((pacelmadeee (Saacelwvaerss

---

6.00
140
(tuuti) Ond
HIG5415
(a) Breakout (fast)
(b) Krull (medium)
(c) Spacelnvaders (medium)
Figure 4: Ablation study results comparing DrMG with non-adaptive distributional actor- critic baseline QR. Baselines learning with -on-adaptive distributional return performs substantially
worse than DrMG.
to hit the HNS, respectively. We examine the fast group to show that learning the distributional return DrMG would not slow down the learning process compared its counterparts learning scalar return. On the classic Atari game Breakout, DrMG converges noticeably faster than MG and STACX, and on DemonAltack, DrMG learns as fast as MG. is also desirable evaluate learning efficiency for the games from the medium group since different methods behave very differently in making progress in those tasks. From the results, we can see DrMG progresses significantly faster than the other two methods in Robotank and Kangaroo. In Kangaroo, method saves almost 1/3 and 1/2 frames compared to MG and STACX. In Space the progress DrMG is compa- rable to the others. In the slow group, the RL algorithms strive progress with slow learning paces. All three games are challenging ones for our method as well as the other baselines to achieve high HNS. So. their performance
them is crucial to derive the median HNS score, i.e., pulling back anyone could potentially lower the median HNS. We can observe that DrMG leads to fairly stable performance among those in slow group. In BeamRider, Centipede and Zaxxon, DrMG achieve better asymptotic rewards than the baselines. It is also shown in Figure 3 that the performance of STACX is not ideal, e.g., irregular performance in DemonAttack and Kangaroo.
5.3
ABLATION STUDY
We perform ablation studies to investigate the effects of the following designs in method: (i) the adaptive distributional return vs. non-adaptive distributional return; (ii) X-distributional return n--tep distributional return; (iii) vector hyperparameter 1) scalar vy (iv) adopting BMG's bootstrapped inner steps on top of our method, where the results of ablation study (iv) are presented appendix A.3 for detailed comparison//iicuusion.
Adaptive vs. non-adaptive distributional return We investigate the effects of adaptive/non- adaptive distributional return by comparing DrMG with two ablated baselines. The meta param- eter set is split into groups 1 {no. m where no {), a, and 7 [WH, Wa wz }. The first baseline only meta-update 7h, denoted as DrMG(m) (complete version is denoted as
DYNG StOp
388
SWHG- DYNG H8S5-d.5
27
(a) Breakout (fast)
(b) Robotank (medium)
(c) Kangaroo (medium)
Figure 5: Ablation study results comparing the X-return used in DrMG with ablated baseline DrMG+n-step which replaces the X-distributional return with an n-step distributional return, where
consider two cases = 3 and 5.
8

---

6.00
4.00
540
130
20
(a) Breakout (fast)
(b) SpaceInvaders (medium)
(c) Centipede (slow)
Figure 6: Ablation study results for comparing our proposed vector hyperparameter set (DrMG(n)) with scalar set hyperparameters n (DrMG()). DrMG(f) performs comparable to DrMG(1) the fast and medium games and underperforms DrMG(1) in the slow game.
DrMG( 7))) The second baseline is QR, non-meta method equivalent DrMG(()). We com- pare them in Breakout, Krull, and Spacelnvaders. The results are shown in Figure 4 We first notice that both versions of non-adaptive distributional X-refurn work considerably well with the actor- critic IMPALA framework (outperform their corresponding DQN variants by large margins). Using QR can learn as smoothly as DrMG in Krull but for the other two domains, it underperforms DrMG, which shows the effectiveness of using meta- graddgg to optimize distributional return. DrMG()) outperforms QR in 2/3 games, but underperforms DrMG(10, m, which demonstrates the superi- ority of meta-learning adaptive return in distributional meta graddggg algorithms.
A A-distributional return vs. n-step distributional return We investigate the effects of A-/n- step distributional return by comparing DrMG with a meta geta-gradieet RL baseline that replaces the X-distributional return by n-step return computed over the return distributions, denoted as DrMG+n-step. We consider two values for n in evaluation, where n = and n = i.e. resulting in baselines 3rMe+ and DrMG+5-step. We adopt three domains, DemonAnack; Spaceln- vaders, and Centipede. The results are shown in Figure We can observe that replacing X-return with n-step return substantially pulls down the rewards ther confirms that backpropagating the distributional TD errors from A-return is highly efficient to formulate the adaptive distributional return target.
in the fast game Breakout. This fur-
Vector hyperparameters vs. scalar hyperparameters We then investigate the effects of vec- tor hyperparameters 17 vs. scalar hyperparameters by training DrMG with two kinds of adaptive distributional return parameterizations. When using scalar hyperparameters îj. the return for each quantile defined in A shares the same discounting factor trace X, well as cost for the quantile regression loss. When using vector form of 7). each quantile head enjoys quantile-specific pa- rameterization for the adaptive return. We evaluate DrMG(1)) and DrMG(r) in Breakout, Spaceln- vaders and Centipede, where the baselines are most distinguishable with DrMG( The results presented in Figure 6 We observe that DrMG(f)) performs fairly well in Breakout well as Spacelnvaders, where other ablated baselines easily fall in the latter case. However, for the slow game Centipede, DrMG( noticeably underperforms its vector version DrMG(
CONCLUSION
We present a meta-gradient algorithm with distributional return named DrMG, which employs a critic that learns adaptive distributional return with parameters be trained by meta-gradient derived from a white-box bi-level optimization scheme. We demonstrate that the quantile distributional return can seamlessly work with distributed actor-critic algorithms and learning such a distributional adaptive distributional return can offer more meaningful signals to update the meta parameters. We show DrMG is able to outperform its most related con-meta-learninn baseline IMPALA and several decent RL baselines video game benchmark Atari 2600. Our method is also very general and the distributional return we introduce hereby can be adapted to work with alternative meta-gradient RL methods, with the quantile distributional return being able to be flexibly replaced with other forms of distributions. We leave the work of integrating more variants of actor-critic algorithms distributional returns in DrMG as our future work. It is also worth considering applying DrMG on more practical RL domains, such as pure value-hased RI, offline R, and continuous control domains.
a motivating Two-Colors Grid- World domain and the challenging

---

ETHICS AND REPRODUCIBILITY STATEMENT
Our work tackles synthetic digital game-playing applications and therefore we believe our work free from ethical problems. We present details on the distributional framework, Atari wrapper settings, and model hyperparameters in Appendix. We also include the code for the two important algorithmic components, networks, and the two-level meta-update step function in our work.
REFERENCES
Maruan Al-Shedivat, Trapit Bansal, Yura Burda, Ilya Sutskever, Igor Mordatch, and Pieter Abbeel. Continuous adaptation via meta-learning in nonstationary and competitive environments. In 6th International Conference Learning Representations (ICLR), 2018.
Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. In Advances in Neural Information Processing Systems 29: Annual Conference on Neu- ral Information Processing Systems (NIPS), pp. 3981-3989, 2016.
Sungyong Baik, Myungsub Choi, Janghoon Choi, Heewon Kim, and Kyoung Mu Lee. Meta- learning with adaptive hyperparameters. In Advances in Neural Information Processing Sys- tems 33: Annual Conference Neural Information Processing Systems (NeurIPS), 2020, virtual, 2020.
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
Mare G. Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning (ICML), volume 70, pp. 449-458, 2017.
Yoshua Bengio, Samy Bengio, and Jocelyn Cloutier. Learning a synaptic learning rule. Citeseer, 1990.
Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. In Proceedings of the 35th International Conference on Machine Learning (ICML), volume 80, pp. 1104-1113, 2018a.
Will Dabney, Mark Rowland, Mare G. Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In Proceedings the Phirty-Secondl AAAI Conference Artifi- cial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium Educational Advances in Artificial Intelligence (EAAI-18), pp- 2892-2901, 2018b.
Yan Duan, John Schulman, Xi Chen, Peter I.. Bartlett, Ilya Sutskever, and Pieter Abbeel. RI22: Fast reinforcement learning via slow reinforcement learning. CoRR, abs/161 .02779, 2016.
Lasse Espeholt, Hubert Soyer, Rémi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, Shane Legg, and Koray Kavukcuoglu. IMPALA: scalable distributed deep-rl with importance weighted actor-learner architectures. In Proceedings of the 35th International Conference on Machine Learning (ICML) volume 80, pp. 1406-1415, 2018.
Chelsea Finn and Sergey Levine. Meta-learning and universality: Deep representations and gradient descent can approximate any learning algorithm. In 6th International Conference on Learning Representations (ICLR), 2018.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning (ICML), pp. 1126-1135, 2017a.
10

---

Chelsea Finn, Tianhe Yu, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-shot visual imita- tion learning via meta-learning. 1st Anmual Conference on Robot Learning (CoRL), volume 78 of Proceedings of Machine Learning Research, pp. 357-368, 2017b.
Sebastian Flennerhag, Yannick Schroecker, Tom Zahavy, Hado van Hasselt, David Silver, and Satin- der Singh. Bootstrapped meta- In The 10th International Conference on Learning Rep- resentations (ICIR), 2022.
Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas L. Griffiths. Recasting gradient-bused meta-learning as hierarchical bayes. In 6th International Conference on Learn ing Representations (ICIR), 2018.
Abhishek Gupta, Russell Mendonca, Yuxuan Liu, Pieter Abbeel, and Sergey Levine. Meta- reinforcement learning of structured exploration strategies. In Advances in Neural Informa- fion Processing Systems 31: Annual Conference on Neural Information Processing Systems, (NeurIPS), Montréal, Canada, pp. 5307-5316, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition (CVPR), pp. 770-778, 2016.
Timothy M. Hospedales, Antreas Antoniou, Paul Micaelli, and Amos J. Storkey. Meta-learning in neural networks: A survey. IEEE Trans. Pattern Anal. Mach. Intell., 4499):514,, 2022.
Junhyuk Oh, Matteo Hessel, Wojeiech M. Czameeck,, Zhongwen Xu, Hado van Hasselt, Satinder Singh, and David Silver. Discovering reinforcement learning algorithms. In Advances in Neu- ral Information Processing Systems 33: Annual Conference on Neural Information Processing Systems (NeurIPS), volume 33, pp. 1060-1070, 2020.
Austin Derrow Pinion, Jennifer She, David Wong, Oliver Lange, Todd Hester, Luis Perez, Marc Nunkesser, Seongjae Lee, Xueying Guo, Brett Wiltshire, Peter W. Battaglia, Vishal Gupta, Ang Li, Zhongwen Xu, Alvaro Sanchez-Gonzalez, Yujia Li, and Petar Velickovie. ETA prediction with graph neural networks in google maps. CoRR, abs/2108. 1482, 2021.
Nicol N. Schraudolph and Xavier Giannakopoulos. Online independent component analysis with local learning rate adaptation. In Advances in Neural Information Processing Systems 12: Annual Conference on Neural Information Processing Systems (NIPS), pp. 789-795, 1999.
Richard Sutton. A history of meta-gradient: Gradient methods for meta-learning.
CoRR,
abs/2202.097011, 2022.
Richard S. Sutton. Adaptation of learning rate parameters. Goal Seeking Components for Adaptive Intelligence: An Initial Assessment, by AG Barto and RS Sutton. Air Force Wright Aeronauti- cal Laboratories Technical Report AFWAL-TR-81-1070, Wright-Pateersn Air Force Base, Ohio, 45433, 1981.
Richard S. Sutton. Learning to predict by the methods of temporal differences. Machine Learning, 3:9-44, 1988.
Richard S. Sutton. Adapting bias by gradient descent: An incremental version of delta-bar-delta, In Proceedings the 10th National Conference on Artificial Intelligence, pp. 171-176, 1992.
Richard S. Sutton and Andrew G. Barto. Reinforcement learning an introduction. Adaptive com- putation and machine learning. MIT Press, 1998.
Sebastian Thrun and Tom M. Mitchell. Learning one more thing. In Proceedings of the Fourteenth International Joint Conference Artificial Intelligence (IJCAI), pp. 1217-1225, 1995.
Vivek Veeriah, Matteo Hessel, Zhongwen Xu, Janarthanan Rajendran, Richard L. Lewis, Junhyuk Oh, Hado van Hasselt, David Silver, and Satinder Singh. Discovery of useful questions iliary tasks. In Advances in Neural Information Processing Systems 32: Anmual Conference on Neural Information Processing Systems (NeurIPS), pp. 9306-9317, 2019.

---

Vivek Veeriah, Tom Zahavy, Matteo Hessel, Zhongwen Xu, Junhyuk Oh, lurii Kemaev, Hado P van Hasselt, David Silver, and Satinder Singh. Discovery of options via meta-learned subgoals. Ad vances in Neural Information Processing Systems 34: Anmual Conference on Neural Information Processing Systems (NewrIPS), volume 34, pp. 29861-29873, 2021.
Jane Wang, Zeb Kurth-Nelson, Hubert Soyer, Joel Z. Leibo, Dhruva Tirumala, Rémi Munos, Charles Blundell, Dharshan Kumaran, and Matt M. Botvinick. Learning to reinforcement learn. In Pro- ceedings of the 39th Anmual Meeting of the Cognitive Science Society, CogSci, 2017.
Yufei Wang, Qiwei Ye, and Tie- Yan Liu. Beyond exponentially discounted sum: Automatic learning of return function. CoRR, 0aos//995.110 1591, 2019.
Olga Wichrowska, Niru Maheswaranathan, Matthew W. Hoffman, Sergio Gomez Colmenarejo, Misha Denil, Nando de Freitas, and Jascha Sohl-Dickstein. Learned optimizers that scale and generalize. In Proceedings of the 34th International Conference on Machine Learning (ICML), volume 70, pp. 3751-3760, 2017.
Tianbing Xu, Qiang Liu, Liang Zhao, and Jian Peng. Learning to explore via meta-policy gradi- ent. In Proceedings of the 35th International Conference on Machine Learning, (ICML), Stock- holmsmässan, volume 80, pp. 5459-5468. PMLR, 2018a.
Zhongwen Xu, Hado P van Hasselt, and David Silver. Meta- gradient reinforcement learning. Ad- vances in Neural Information Processing Systems 31: Anmual Conference on Neural Information Processing Systems (NeurIPS), volume 31, pp. 1996-2007, 2018b.
Zhongwen Xu, Hado Philip van Hasselt, Matteo Hessel, Junhyuk Oh, Satinder Singh, and David Silver. Meta-gradient reinforcement learning with an objective discovered online. In Advances in Neural Information Processing Systems 33: Anmual Conference on Neural Information Process- ing Systems (NeurIPS), volume 33, pp. 15254-15264, 2020.
Tom Zahavy, Zhongwen Xu, Vivek Veeriah, Matteo Hessel, Junhyuk Oh, Hado van Hasselt, David Silver, and Satinder Singh. A self-tuning actor-critic algorithm. In Advances in Neural Infor- mation Processing Systems 33: Anmual Conference on Neural Information Processing Systems 2020.
(NeurIPS), volume 33, pp.
Zeyu Zheng, Junhyuk Oh, and Satinder Singh. On learning intrinsic rewards for policy gradient methods. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems (NeurIPS), pp. 4649-4659, 2018
Zeyu Zheng, Junhyuk Oh, Matteo Hessel, Zhongwen Xu, Manuel Kroiss, Hado van Hasselt, David Silver, and Satinder Singh. What can learned intrinsic rewards capture? In Proceedings of the 37th International Conference on Machine Learning (ICML), volume 119, pp. 11436- 1446, 2020.
12


Reviews:
[{"review_id": "xkBfduvvi4B", "reviewer": "Reviewer_s6vn", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a Distributional return Meta-Gradient algorithm (DrMG) algorithm, which combines distributional RL (like C51) and meta-gradient RL. Specifically, the authors formulate a distributional return in a meta-gradient RL setup and derive a meta-update rule to learn the adaptive distributional return with meta-gradients. Empirically, the proposed algorithm is compared with the existing meta-gradient RL methods on 57 Atari tasks.", "summary_of_the_review": "Overall, I feel this paper proposes a good approach that combines distributional RL and meta-gradient RL. However, the empirical evaluation of their approach is not done properly. Given that the idea itself is not extremely fancy, I feel proper evaluation is very important in this case. Therefore, I cannot recommend acceptance.\n\n\n=========\nUpdate after the rebuttal period:\n\nI previously leaned toward rejection. My main concern is that the experiments are not sufficient to support their claims fully. Given that this paper's technical novelty is not super significant (combining distribution RL and meta-gradient RL), I feel proper evaluation is very important in this case.\n\nHowever, during the rebuttal period, the authors provided additional experiment results, including more random seeds for each experiment, more baselines, and more ablations. The additional results make the empirical evaluation much stronger. Therefore, my main concern is addressed, and I lean toward acceptance.", "strength_and_weaknesses": "The strength and weaknesses of this paper are both significant.\n\n## Strength\n\n- The idea of combining distributional RL and meta-gradient RL is natural and intuitive.\n\n- The proposed algorithm DrMG seems to achieve great performance on the Atari benchmark compared to existing meta-gradient RL methods.\n\n- The ablation study considers several important design choices.\n\n\n\n## Weakness\n\nThe weaknesses mainly come from the experiments.\n\n- The toy experiment \"Two Colors Grid-World\" is very confusing to me. Specifically,\n  \n  - In Sec 5.1, this paper mentions \"We toggle the reward at each 1e5 steps to investigate the agent’s adaptation ability.\" What is the exact meaning of \"toggle the reward at each 1e5 steps\"? According to the appendix, it seems the key collection reward values toggle between the two keys. However, if this is the case, then why a key-agnostic optimal agent could collect a +1.35 expected return? I think it should be 1.5.\n  \n  - In figure 1, the dashed line denoting the key-agnostic agent is neither 1.5 nor 1.35. What is the exact number of it and how do you get it?\n  \n  - Why the reward is modified in the middle of training? This is an uncommon rare practice in RL. The authors claim that \"Since the RL agents are unaware of the reward toggling during policy training, their adaptation ability is characterized by their efficiency in modeling the return during the training process.\" I do not really understand this point. Why the \"efficiency in modeling the return\" needs to be shown in such a weird setting?\n\n- The experiments on Atari are not conducted properly\n  \n  - First, it seems all the curves come without std. It is unconvincing if the experiments are only run with one random seed, especially when the curves look very unstable. I understand the computation cost of those experiments is high, but it is still necessary to try a least a few more random seeds. In STACX, they use 3 random seeds.\n  \n  - It is unclear why the experiments shown in figure 3 adopt an early stopping strategy but the experiments shown in figure 2 do not. Given that the full curves (training with 200M) should be available according to figure2, why not show the full curves in figure 3?\n\n- Minor points\n  \n  - The legends in the figures should be consistent. For example, \"MG\" in figure 2 but \"MetaGrad\" in figure 3.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "hhz6ND6KiJ", "reviewer": "Reviewer_ApMf", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In the existing meta-gradient studies, the adaptive return is simply formulated in the form of expected cumulative rewards. In this paper, the authors address the meta learning problem of the distributional actor-critic method with quantile distributions by considering the distributional $\\lambda$ return and apply the standard meta-gradient method to self-tune a set of hyperparameters. Some of the hyperparameters are quantile-specific, which offers a larger scope of hyperparameters compared to the prior works. This new algorithm outperforms the existing meta-gradient methods in Grid-world and Atari games, achieving the state-of-the-art performance.", "summary_of_the_review": "This paper proposes an effective meta gradient method to boost the performance of quantile-based distributional RL. Despite that the idea is not very novel, the experimental results on Atari are very promising, and the ablation study verifies the efficacy of the adaptive return.\n", "strength_and_weaknesses": "**Strength**\n\n- This paper provides the first distributional attempt of meta-gradient RL.\n- The empirical results are quite promising in Atari 2600 benchmarks.\n\n\n**Weakness**\n\n- The proposed algorithm appears to be a direct combination of distributional actor-critic and the standard meta-gradient method. The novelty of the algorithm is somewhat limited.\n- Comparison with baselines: This paper empirically compares DrMG mainly with the standard MG and a more recent method STACX. However, the most recent baseline Bootstrapped Meta Gradient (BMG) was excluded for comparison. The paper mentioned that “... which is excluded here for comparison because it alters some important experimental settings for 200M evaluation to boost the performance.” Despite this, it still appears feasible to adjust the setting of BMG to ensure a fair comparison. I do not fully get the difficulty here. More justification would be needed.\n\n\nAdditional issues: Some explanations of the methodology are incorrect or confusing. For example: \n- The distributional Bellman operator in Eq (8) appears incorrect (cf. Eq. (5) in [Bellemare et al., 2017]).  \n- There shall be some typos in Eq. (14), e.g., the learning rates are missing. \n- The last quality of Eq. (14) would require explanation in more detail.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Bw8-s7-s9C", "reviewer": "Reviewer_K6pA", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper is considering meta-gradient RL, algorithms that consider the gradient of the gradient descent update to learn better objectives. The authors are proposing an algorithm called DrMG which is based on two foundational choices: (a) the use of a quantile distribution for the return and (b) the use of a white-box meta-gradient RL model where the loss function depends on two sets of parameters, the low-level parameters theta and the high-level parameters eta. The algorithm used for RL is a actor-critic class algorithm, with the critic approximating a quantile distribution. ", "summary_of_the_review": "A novel contribution to the field of model free RL providing faster learning and higher performance than comparable algorithms on the standard benchmarks.", "strength_and_weaknesses": "+ Convincing justification that the estimation through a quantile distribution represented through a uniform mix of Dirac's deltas. \n+ In the evaluation on both a toy problem and on Atari games, the overall performance shows a clear improvement over the comparable algorithms.\n\n- When replacing a numerical value with a distribution, the reader would be naturally interested in how this distribution looks like. Is it Gaussian, skewed, fat tailed, multimodal distribution? The paper makes no effort to investigate this. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-Mp5S7CY0E", "reviewer": "Reviewer_LNTi", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to combine white-box meta-learning methods and\ndistributional RL. The main benefit of this combination is that the\ndistributional return can not be adaptive, and perhaps allow for more\neffective RL agents. In particular, the authors propose a meta-gradient\nthrough the parameters that define the quantile and its loss. The\ndistributional meta-update is said to be compatible will almost all\nmeta-gradient RL work. Experiments are first conducted on a small\nnon-stationary grid world problem, but the majority of the results are\non Atari 200M, where they beat distributional baselines and a\nrepresentative meta-gradient baseline in aggregated score. They note,\nhowever, that the SOTA meta-gradient baseline - STACX - is difficult to\nreproduce and do not compete directly.\n\n", "summary_of_the_review": "\n# Decision\n\nWhile there is obvious appeal in combining two currently separate but\nsuccessful RL methods (that of distributional and meta-gradient RL), I\nthink the paper as a whole does not provide enough insight into\nchallenges and benefits of such a combination. The paper clearly\ndemonstrates that the proposed method (DrMG) is able to outperform\ndistributional baselines and a meta-gradient baseline. The paper also\ndemonstrates that adaptive distributional returns are important in some\nAtari games. As a whole though, the Atari benchmark makes sound\nempirical comparison difficult due to the fact that no statistical\nevidence is presented. In addition, some important baselines are missing\n(STACX, which cannot be reproduced to the reported performance as well\nas a distributional meta-gradient baseline that does not adapt the\ndistributional hyperparameters). As the paper currently stands, I am\nleaning towards weak reject with the possibility of increasing to weak\naccept if some of the gaps in the baselines are addressed.\n", "strength_and_weaknesses": "# Strengths\n\n-   Strong empirical results, versus the baselines presented. This is,\n    however, somewhat overshadowed by STACX missing as a baseline. It is\n    also to be expected that adapting more hyperparameters allows for a\n    more flexible learner as it is more general than the baselines\n    compared against.\n\n-   The contribution is clear: both meta-gradient RL methods and\n    distributional RL are strong methods in current deep RL. Combining\n    them together is a clear and interesting research direction.\n\n# Weaknesses\n\n-   Some important baselines and experimental details are missing that\n    would help contextualize the merit of distributional meta-gradients.\n-   Overall lacking in novelty. While the contribution is clear and\n    interesting, combining meta-gradients and distributional RL is\n    rather obvious as both methods are orthogonal.\n-   The experiments are somewhat limited. Atari 200M is a large and\n    difficult problem, and success on this benchmark is commedable.\n    Being a large benchmark, however, limits the accuracy of the\n    conclusions because of the high degree of variability. Although the\n    results are averaged over 5 runs, no error bars are reported and the\n    ablation on various games is somewhat ad hoc.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster