Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a Distributional return Meta-Gradient algorithm (DrMG) algorithm, which combines distributional RL (like C51) and meta-gradient RL. Specifically, the authors formulate a distributional return in a meta-gradient RL setup and derive a meta-update rule to learn the adaptive distributional return with meta-gradients. Empirically, the proposed algorithm is compared with the existing meta-gradient RL methods on 57 Atari tasks.
strength_and_weaknesses: The strength and weaknesses of this paper are both significant.

## Strength

- The idea of combining distributional RL and meta-gradient RL is natural and intuitive.

- The proposed algorithm DrMG seems to achieve great performance on the Atari benchmark compared to existing meta-gradient RL methods.

- The ablation study considers several important design choices.



## Weakness

The weaknesses mainly come from the experiments.

- The toy experiment "Two Colors Grid-World" is very confusing to me. Specifically,
  
  - In Sec 5.1, this paper mentions "We toggle the reward at each 1e5 steps to investigate the agent’s adaptation ability." What is the exact meaning of "toggle the reward at each 1e5 steps"? According to the appendix, it seems the key collection reward values toggle between the two keys. However, if this is the case, then why a key-agnostic optimal agent could collect a +1.35 expected return? I think it should be 1.5.
  
  - In figure 1, the dashed line denoting the key-agnostic agent is neither 1.5 nor 1.35. What is the exact number of it and how do you get it?
  
  - Why the reward is modified in the middle of training? This is an uncommon rare practice in RL. The authors claim that "Since the RL agents are unaware of the reward toggling during policy training, their adaptation ability is characterized by their efficiency in modeling the return during the training process." I do not really understand this point. Why the "efficiency in modeling the return" needs to be shown in such a weird setting?

- The experiments on Atari are not conducted properly
  
  - First, it seems all the curves come without std. It is unconvincing if the experiments are only run with one random seed, especially when the curves look very unstable. I understand the computation cost of those experiments is high, but it is still necessary to try a least a few more random seeds. In STACX, they use 3 random seeds.
  
  - It is unclear why the experiments shown in figure 3 adopt an early stopping strategy but the experiments shown in figure 2 do not. Given that the full curves (training with 200M) should be available according to figure2, why not show the full curves in figure 3?

- Minor points
  
  - The legends in the figures should be consistent. For example, "MG" in figure 2 but "MetaGrad" in figure 3.
clarity,_quality,_novelty_and_reproducibility: - Clarity
  
  - The proposed method is clearly described and easy to follow.
  
  - The experiments section can be improved with more details and accurate descriptions.

- Quality
  
  - The quality of this paper is ok but can be polished a little bit more.

- Novelty
  
  - To the best of my knowledge, this paper is the first to combine distribution RL with meta-gradient RL.

- Reproducibility
  
  - A lot of implementation details are presented in the appendix, but it would be great if the authors could release the code to the public.
summary_of_the_review: Overall, I feel this paper proposes a good approach that combines distributional RL and meta-gradient RL. However, the empirical evaluation of their approach is not done properly. Given that the idea itself is not extremely fancy, I feel proper evaluation is very important in this case. Therefore, I cannot recommend acceptance.


=========
Update after the rebuttal period:

I previously leaned toward rejection. My main concern is that the experiments are not sufficient to support their claims fully. Given that this paper's technical novelty is not super significant (combining distribution RL and meta-gradient RL), I feel proper evaluation is very important in this case.

However, during the rebuttal period, the authors provided additional experiment results, including more random seeds for each experiment, more baselines, and more ablations. The additional results make the empirical evaluation much stronger. Therefore, my main concern is addressed, and I lean toward acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In the existing meta-gradient studies, the adaptive return is simply formulated in the form of expected cumulative rewards. In this paper, the authors address the meta learning problem of the distributional actor-critic method with quantile distributions by considering the distributional $\lambda$ return and apply the standard meta-gradient method to self-tune a set of hyperparameters. Some of the hyperparameters are quantile-specific, which offers a larger scope of hyperparameters compared to the prior works. This new algorithm outperforms the existing meta-gradient methods in Grid-world and Atari games, achieving the state-of-the-art performance.
strength_and_weaknesses: **Strength**

- This paper provides the first distributional attempt of meta-gradient RL.
- The empirical results are quite promising in Atari 2600 benchmarks.


**Weakness**

- The proposed algorithm appears to be a direct combination of distributional actor-critic and the standard meta-gradient method. The novelty of the algorithm is somewhat limited.
- Comparison with baselines: This paper empirically compares DrMG mainly with the standard MG and a more recent method STACX. However, the most recent baseline Bootstrapped Meta Gradient (BMG) was excluded for comparison. The paper mentioned that “... which is excluded here for comparison because it alters some important experimental settings for 200M evaluation to boost the performance.” Despite this, it still appears feasible to adjust the setting of BMG to ensure a fair comparison. I do not fully get the difficulty here. More justification would be needed.


Additional issues: Some explanations of the methodology are incorrect or confusing. For example: 
- The distributional Bellman operator in Eq (8) appears incorrect (cf. Eq. (5) in [Bellemare et al., 2017]).  
- There shall be some typos in Eq. (14), e.g., the learning rates are missing. 
- The last quality of Eq. (14) would require explanation in more detail.

clarity,_quality,_novelty_and_reproducibility: - Clarity: The paper is overall well-written and easy to follow in most places. 
- Quality: The experimental results are very promising, and the ablation study is quite helpful in explaining the effect of the adaptive return and the lambda return. 
- Novelty: The idea of applying a meta-gradient method to distributional RL has never been substantiated or evaluated before. That said, the overall algorithm design appears to be a rather direct combination of meta gradient and quantile-based distributional RL.
- Reproducibility: This shall be acceptable given that the critical parts of the pseudo code and the hyperparameters are provided in the appendix.
 

summary_of_the_review: This paper proposes an effective meta gradient method to boost the performance of quantile-based distributional RL. Despite that the idea is not very novel, the experimental results on Atari are very promising, and the ablation study verifies the efficacy of the adaptive return.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper is considering meta-gradient RL, algorithms that consider the gradient of the gradient descent update to learn better objectives. The authors are proposing an algorithm called DrMG which is based on two foundational choices: (a) the use of a quantile distribution for the return and (b) the use of a white-box meta-gradient RL model where the loss function depends on two sets of parameters, the low-level parameters theta and the high-level parameters eta. The algorithm used for RL is a actor-critic class algorithm, with the critic approximating a quantile distribution. 
strength_and_weaknesses: + Convincing justification that the estimation through a quantile distribution represented through a uniform mix of Dirac's deltas. 
+ In the evaluation on both a toy problem and on Atari games, the overall performance shows a clear improvement over the comparable algorithms.

- When replacing a numerical value with a distribution, the reader would be naturally interested in how this distribution looks like. Is it Gaussian, skewed, fat tailed, multimodal distribution? The paper makes no effort to investigate this. 
clarity,_quality,_novelty_and_reproducibility: + Clear description of the justification of the proposed approach and how it fits in the state of the art.
+ The primary novelty of the paper is to approximate the distributional value in the value update of the RL, and finding an efficient method to do this.
summary_of_the_review: A novel contribution to the field of model free RL providing faster learning and higher performance than comparable algorithms on the standard benchmarks.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to combine white-box meta-learning methods and
distributional RL. The main benefit of this combination is that the
distributional return can not be adaptive, and perhaps allow for more
effective RL agents. In particular, the authors propose a meta-gradient
through the parameters that define the quantile and its loss. The
distributional meta-update is said to be compatible will almost all
meta-gradient RL work. Experiments are first conducted on a small
non-stationary grid world problem, but the majority of the results are
on Atari 200M, where they beat distributional baselines and a
representative meta-gradient baseline in aggregated score. They note,
however, that the SOTA meta-gradient baseline - STACX - is difficult to
reproduce and do not compete directly.


strength_and_weaknesses: # Strengths

-   Strong empirical results, versus the baselines presented. This is,
    however, somewhat overshadowed by STACX missing as a baseline. It is
    also to be expected that adapting more hyperparameters allows for a
    more flexible learner as it is more general than the baselines
    compared against.

-   The contribution is clear: both meta-gradient RL methods and
    distributional RL are strong methods in current deep RL. Combining
    them together is a clear and interesting research direction.

# Weaknesses

-   Some important baselines and experimental details are missing that
    would help contextualize the merit of distributional meta-gradients.
-   Overall lacking in novelty. While the contribution is clear and
    interesting, combining meta-gradients and distributional RL is
    rather obvious as both methods are orthogonal.
-   The experiments are somewhat limited. Atari 200M is a large and
    difficult problem, and success on this benchmark is commedable.
    Being a large benchmark, however, limits the accuracy of the
    conclusions because of the high degree of variability. Although the
    results are averaged over 5 runs, no error bars are reported and the
    ablation on various games is somewhat ad hoc.

clarity,_quality,_novelty_and_reproducibility: # Detailed Comments (Clarity, Quality, Novetly, Reproducibility)

-   Section 1, Paragraph 1 (Meta-gradient RL): It is limiting to say
    that meta-gradient RL only provides better objectives. While this is
    true in Xu et al. 2022, meta-gradients can also be used to
    meta-learn hyperparameters unrelated to the target or objective,
    such as the learning rate or epsilon for exploration.

-   Section 1, Paragraph 2: "This is essential for RL updates and even
    more critical for meta-gradients that enjoy multiple satisfying
    properties"

    It is not clear what "this" refers to, but if you mean value
    distributions then this is not true nor demonstrated in any
    meta-gradient RL work.

-   "On the other hand, distributional RL methods have good optimization
    properties in the loss form"

    I do not think this is true either. While there are worse options
    than the KL divergence for stability, that does not mean that the KL
    loss in distributional RL has particularly good proprerties in
    comparison to non-distributional RL.

-   Section 2, Paragraph 3: This paragraph is key and really helped me
    understand the goal of this paper more than the abstract or
    Section 1. I would encourage the authors to rework this into the
    introduction.

-   Section 3 " the Bellman operator for quantile distribution does not
    need to perform projection (lose precision) or manually define the
    return boundaries, like C51 (Bellemare et al., 2017) does." This
    seems at odds with the claim that distributional RL is a good method
    for meta-learning because of the additional hyperparameters it
    requires.

-   Section 4 ( Actor-Critic with Distributional λ-Return): this
    majority of this subsection seems better suited to the preliminaries
    section. It is not clear which parts are new contributions or
    relating to previously established work on meta-gradient
    actor-critic.

-   Section 4: "Our proposed distributional adaptive return can
    naturally boost up the scope of self-tuning hyperparameters in the
    meta-gradient RL algorithm."

    Not clear what this means exactly, but the principle of self-tuning
    hyperparameters is not limited in scope in any fundamental way. Using meta-gradients to tune specific hyperparameters related to distributional return is merely an application.

-   Section 5.1 (Two Colors Grid): While the results are compelling, in
    the sense that DrMG adapts quicker than MG, it is not clear whether
    this is due to Distributional meta-gradients or just the
    distributional base algorithm. Unfortunately, the appendix is short
    on details here.

-   Section 5.2 (Atari 2600 200M): Again, the results here are
    compelling, albeit difficult to ascertain their true significance.
    Atari 200M is a large and difficult task, and the reported
    performance seems promising. But, as an evaluation benchmark, it is
    quite limited due to its size and inability to run many seeds with
    confidence intervals.

-   Section 5.3 (Ablating adaptive vs non-distributional return): It is
    true that an important ablation study would compare the importance
    of adaptive and non-adaptive distributional return. However, an even
    more important baseline would be a meta-gradient algorithm that uses a distributional return but does
    not take a meta-gradient through the distributional hyperparameters. Without this very important baseline, it's difficult to tell how important meta-gradients are for distributional RL in particular. 

# Minor Comments

-   Section 3 "computed from the updated model parameters θ"

    Should this be: "computed from the updated model parameters θ'"?

    If not, you should be clear what θ′ is, for example
    θ′ = θ − f(τ, θ, η).

-   Section 4, Distributional Meta-Gradient (usage of α): earlier you
    used α as a step-size so it is confusing to use it for the leak
    v-trace as well.

-   Section 4, "for our method; with STACX and MG it would be 3 × N + 3,
    7 × 3 and 2, respectively." What is N here?

-   Section 5: "We toggle the reward at each 1e5 steps to investigate
    the agent’s adaptation ability. " What does this mean exactly? If
    the blue key gives a reward of 0.7 and the red key gives a reward of
    0.3, does toggling switch the color-reward correspondence.

-   Section 5.1 title: "GIRD-WORLD" -> "GRID-WORLD"


summary_of_the_review: 
# Decision

While there is obvious appeal in combining two currently separate but
successful RL methods (that of distributional and meta-gradient RL), I
think the paper as a whole does not provide enough insight into
challenges and benefits of such a combination. The paper clearly
demonstrates that the proposed method (DrMG) is able to outperform
distributional baselines and a meta-gradient baseline. The paper also
demonstrates that adaptive distributional returns are important in some
Atari games. As a whole though, the Atari benchmark makes sound
empirical comparison difficult due to the fact that no statistical
evidence is presented. In addition, some important baselines are missing
(STACX, which cannot be reproduced to the reported performance as well
as a distributional meta-gradient baseline that does not adapt the
distributional hyperparameters). As the paper currently stands, I am
leaning towards weak reject with the possibility of increasing to weak
accept if some of the gaps in the baselines are addressed.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a general framework of leveraging distributional reinforcement learning (RL) in meta-gradient RL.  The adaptive distributional return learned by the distributional RL method can offer information that is useful in updating meta-parameters b meta-gradient RL.  While the combination of distributional RL and meta-gradient RL might not be super innovative (which is the main weakness of the paper), the proposed method has been validated extensively in the experiments with Atari, which also gives empirical insights into this combination.  This innovation in the fundamental idea together with strong empirical support constitute the major strength of the paper.


justification_for_why_not_higher_score: The proposed approach is a combination of distributional RL and meta-gradient RL, and its algorithmic novelty is rather limited and may not appeal to a broad audience.
justification_for_why_not_lower_score: The reviewers were unanimous in their opinion that the proposed approach is novel, provides new insights, and its effectiveness is strongly supported by experiments.
Decision: Accept: poster