Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper provides theoretical guarantees on multi-step off-policy distributional RL algorithms and derives a DRL algorithm, QR-DQN-Retrace based on QR-DQN which shows empirical improvements over QR-DQN on Atari.
strengths_and_weaknesses: There are two main contributions of this paper:
1. This paper systematically discusses the similarity and difference between multi-step value-based RL and multi-step distributional RL. They also provide theoretical guarantees on multi-step off-policy distributional RL algorithms.
2. This paper derives a novel algorithm, Quantile Regression-Retrace, which leads to a deep RL agent QR-DQN-Retrace that shows empirical improvements over QR-DQN on the Atari-57 benchmark. 

There are some weaknesses of this paper.
1. Although there are some theoretical contributions, the methodological novelty is not that big from the perspective of algorithm architecture compared to the baseline QR-DQN.
2. The empirical part is not that solid while some more comparisons between multi-step value based algorithms and multi-step DRL algorithms are helpful. 
questions: 1. Is it possible to demonstrate the difference between multi-step value based algorithms and multi-step DRL algorithms in an empirical way?
2. As Figure 4 shows, the improvement of "Retrace" over "Uncorrelated" seems to be not significant.
3. How about applying the retrace idea on other DRL algorithms such as IQN, FQF.
4. Can this method ensures the validity of the learned distributions, such as the crossing issues?
Zhou, Fan, Jianing Wang, and Xingdong Feng. "Non-crossing quantile regression for distributional reinforcement learning." Advances in Neural Information Processing Systems 33 (2020): 15909-15919.
limitations: Although there are some theoretical contributions, the algorithm design is not novel enough, and also the empirical part needs to be improved to support the idea of the paper.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors study a multi-step off-policy learning approach to the distributional RL setting. The essential contributions of the paper may be seen as the theoretical and mathematical frameworks necessary to extend one-step distributional RL to an importance-sampled multi-step algorithm. This extension enables off-policy corrections for distributional RL in the same way that the Retrace algorithm enables off-policy corrections for value-based learning. Through a careful analysis, the authors elucidate the fact that such an extension of a Retrace style off-policy update to the distributional RL case is non-trivial. This fact is primarily owed to the nonlinearity, and repeated application of the pushforward operator used to produce the Bellman recurrence relation on the return distribution. The recurrence relation thus establishes a path-dependent relationship between the return distribution and the distributional TD error. This is in direct contrast to the path-independent relationship present in value-based RL methods.

The authors go on to substantiate the proposed methods theoretically and empirically. They evaluate their methods on Atari control problems and compare them against one-step distributional algorithms and uncorrected multi-step distributional algorithms. For each experiment, the authors run their algorithms using two different parametric distribution representations: categorical and quantile regressions. The proposed methods generally outperform the compared methods in either the mean or median score.
strengths_and_weaknesses: Strengths

-The work is, to the best of my knowledge, novel and significant, as it introduces a non-trivial extension of one-step distributional RL to the multi-step off-policy setting.

-The work provides a theoretical basis for the continued study of multi-step paradigms in off-policy reinforcement learning.

-The authors make clear that the extension provided in the paper deviates from the value-based multi-step paradigm. 

-The paper provides adequate clarity of the introduced concepts and is precise with its mathematical notation.



Weaknesses

-The experimental results are not overly convincing that employing the multi-step distributional RL algorithm provides a significant empirical advantage.

-Minor grammatical mistake in line 357.

questions: -Could you please detail exactly in which way the pushforward operations are non-linear? I believe it to be non-obvious, considering that they are constructed using an affine measurable map, and both the composition and inverse of an affine map will be affine.

-Are there particular use cases or real-world consequences of the proposed approach that are worth mentioning? For example, are there specific applications that are now enabled by the work?

limitations: -The authors state in the checklist that they discuss the limitations of their work, although I could not find an explicit statement in the paper. This does not diminish the contributions of the paper.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper investigates the multi-step off-policy distributional RL algorithms. Firstly, the multi-step distributional TD error is defined and the contraction property is also provided. Building this result, the authors pointed out the fundamental difference between multi-step distributional RL and classic RL by proposing a new notion called path-dependence distributional TD error. Finally, a new unbiased version of the multi-step distributional RL algorithm is proposed, including quantile-regression-Retrace and QR-DQN-Retrace, which is slightly better than the QRDQN and the naïve biased version of the multi-step distributional RL algorithm.
strengths_and_weaknesses: ### Originality

Theoretically, the paper provides the first theoretical guarantees on multi-step off-policy distributional RL algorithms (see Proposition 3.2 and 3.3). To the best of my knowledge, the notion of path-dependent distributional TD error is novel. However, combining multi-step distributional RL with quantile regression is straightforward, although a relatively different unbiased version in Line 280 is derived.


### Quality  

The writing quality is good. One minor typo in Line 239, “Speficially” should be “specifically”.

The technical contribution can be further improved. Generally, investigating the multi-step distributional RL may not be somewhat intriguing as some theoretical results in one-step distributional RL can normally be straightforward to extend into the multi-step version. As such, the theoretical result may not be surprising to me, and the design of the QR-based Retrace algorithm may not be novel enough.


###  Clarity  

The paper is well organized and easy to follow. The key confusing point is the comparison between path-dependent distributional TD error and the value-based TD error, although I understand the authors did make some efforts to clarify it. Please refer to Question for my detailed concern.

###  Significance  

One potential limitation of this paper is the proposed algorithm may not achieve significant improvement compared with the baseline “Uncorrected” as suggested in Figure 4. It is easy to expect that the multi-step distributional RL can be significantly better than the one-step and thus it is not surprising that “Retrace” and “Uncorrected” are superior to “one step”. However, as far as I can tell, the contribution in the algorithm part is the proposal of an unbiased version of QR loss (Line 280 and Lemma 5.2), but I can not recognize the improvement of the unbiased version in contrast to its biased version (“Uncorrected”).

questions: 1. Path-dependent distributional TD error. From Line 174 to 180, I feel that $G_{0:t-1}$ can still be canceled out in the definition of distributional TD error so that only R_t is left. I may miss some points, but could the authors provide more explanation about that? The more detailed derivation could help readers to have a better understanding of their discrepancy.

2. To demonstrate the key role of path-dependent distributional TD error, the authors only show that the path-independent operator numerically can be non-convergent. My suggestion is that a more rigorous proof of the non-convergent behavior could be more convincing.

3. The advantage of path-dependent distributional TD error needs to be further explored. As multi-step distributional TD may heavily depend on the past rewards, can it be beneficial for the optimization of RL algorithms? Or it can lead to an instability issue? A deeper explanation is welcome in the future version.

4. What is the motivation of the proposed unbiased QR loss in Line 280 and what is the advantage over the naïve biased version in [26]? It seems that the empirical improvement is not significant.

5. The experimental result across all Atari games can be also provided rather than only the mean and median of average rewards. For instance, authors can present a histogram to illustrate the improvement ratio of the proposed algorithm across all Atari games.

limitations: In the checklist, the authors said they described the limitations of their work. Maybe I missed them, but I have not found any discussion about them in the main context. This paper indeed makes some contribution to the multi-step off-policy distributional RL as claimed in the introduction, but perhaps the main limitation lies in that some results (both theoretically and empirically) are not inspiring or surprising enough that I have not learned too much after carefully reading this paper. For example, Proposition 3.2 may be a straightforward result based on previous results in the one-step case. Moreover, the underlying impact of the path-independent distributional TD error can be deeply investigated. The empirical improvement of the proposed unbiased Retrace algorithm seems to be only slightly better or even worse compared with the uncorrected version, as illustrated in Figure 6.   


In summary, I faithfully thank the pain-taking efforts the authors make in order to make a solid contribution in the multi-step off-policy distributional RL setting, but the limitations may not be addressed easily. My suggestion is that the path-dependence notion can be further probed and the algorithm novelty can be improved to expect better performance. I would be happy to see a better version of this paper in the future.
review_title: null

Meta-review: metareview: The reviewers carefully analyzed this work and agreed that the topics investigated in this paper are important and relevant to the field. Although the reviewers generally expressed positive views on the proposed method, they also pointed out many possible limitations of this paper. On the one hand, one reviewer acknowledged that the authors provided the first theoretical guarantees on multi-step off-policy distributional RL algorithms via a novel algorithm. They argued, however, that the methodological novelty may not be too significant compared to the baseline QR-DQN, and that the experiments could be improved. After reading the authors' rebuttal, this reviewer—although still with an overall positive impression of the quality of this work—said that they do not believe the paper fully shows the particularity of applying the multi-step idea to DRL when compared to some value-based methods. They also believe that the empirical section of the paper was not strong enough. Another reviewer agreed that this is novel and significant work, as it introduces a non-trivial extension of one-step distributional RL to the multi-step off-policy setting. They argued that this paper provides a theoretical basis for the continued study of multi-step paradigms in off-policy reinforcement learning, which is important/significant. This reviewer, however, pointed out that the experimental results were not sufficiently convincing to support the claims that employing the multi-step distributional RL algorithm provides a significant empirical advantage. This reviewer carefully analyzed the authors' rebuttal and appreciated that they clarified the reviewer's original points of confusion. Overall, however, this reviewer agreed with others that the empirical results are not overly convincing. Furthermore, one of the reviewers argued that the notion of path-dependent distributional TD error is novel. They pointed out, as the main limitation of this paper, that investigating the multi-step distributional RL setting "may not be [particularly interesting since] some theoretical results on one-step distributional RL can be straightforward to extend into the multi-step version". They also argued that the unbiased QR-loss was not clearly motivated since the bias for RL may also be useful for exploration. After reading the authors' rebuttal, this particular reviewer updated their score since the authors provided more evidence to strengthen their theoretical and empirical claims. Overall, all reviewers were positively impressed with the quality of this work but brought up many points of contention regarding ways in which the paper could/should still be improved. They encourage the authors to update their work based on their constructive criticisms and, in particular, in a way that tackles the points of contention mentioned in the original reviews and their post-rebuttal comments.
confidence: Less certain
Decision: Accept