Title: Intrinsic Benefits of Categorical Distributional Loss: Uncertainty-aware Regularized Exploration in Reinforcement Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: distributional learning

reinforcement learning

exploration

uncertainty
Abstract: The remarkable empirical performance of distributional reinforcement learning~(RL) has garnered increasing attention to understanding its theoretical advantages over classical RL. By decomposing the categorical distributional loss commonly employed in distributional RL, we find that the potential superiority of distributional RL can be attributed to a derived distribution-matching entropy regularization. This less-studied entropy regularization aims to capture additional knowledge of return distribution beyond only its expectation, contributing to an augmented reward signal in policy optimization. In contrast to the vanilla entropy regularization in MaxEnt RL, which explicitly encourages exploration by promoting diverse actions, the novel entropy regularization derived from categorical distributional loss implicitly updates policies to align the learned policy with (estimated) environmental uncertainty. Finally, extensive experiments verify the significance of this uncertainty-aware regularization from distributional RL on the empirical benefits over classical RL. Our study offers an innovative exploration perspective to explain the intrinsic benefits of distributional learning in RL.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper aims to explain the benefits of (categorical) distributional RL by relating it to entropy regularization, which encourages exploration. Theoretical results are widely provided, and experiments follow.\n\nSTRENGTHS AND WEAKNESSES:\nThis paper presents an interesting perspective on distributional RL by drawing connections to entropy regularization (and thus maxent RL). Both theoretical and experimental components are provided. However, I found the theoretical section somewhat difficult to follow. Some terminology is unclear. For instance, the \"Return Density Decomposition\" paragraph uses heavy notation and probably need clearer explanations. Terms like $p^\\mu_i$, which is the coefficient of the i-th bin, it needs to be strictly defined. The use of the word \"induced\" in line 161, although it is bold, is kind of clear.\n\nBased on my current understanding, the theoretical contributions may be limited in general. Regarding Proposition 1, I may be misunderstanding it, but it appears to be narrow and not referred to in other theoretical results. Proposition 2 seems fairly standard, as it directly applies the KL divergence to histogram representations. Proposition 3 also appears to be straightforward: when $\\Delta$ is approaching 0, the function should reduce to the desired target.\n\nOn the empirical side, the paper is verifying that the \"uncertainty-aware regularization is crucial to the outperformance of CDRL over classical RL...\" (line 335) However I found it difficult to draw this conclusion from the results. According to Figure 3, runs with verifying $\\epsilon$ is worse than C51 while being better than DQN. This could suggest that the gains might come from the general benefits of distributional RL rather than the proposed uncertainty-aware regularization itself. Additional analysis or ablation might help clarify the specific contribution of the regularization component.\n\nQUESTIONS:\nI listed my questions above. My understanding might be inaccurate so please correct me if I was wrong.\n\nLIMITATIONS:\nThe authors discussed the limitations at the end of the paper. I don't see any potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 3, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work revisits the benefits of distributional RL, focusing on categorical distributional RL (CDRL), which uses categorical losses. In Section 4.2, they show that CDRL uses additional information by learning the entire return distribution, compared to standard RL. They also show that the “mean term” in the distributional objective asymptotically behaves like the standard RL objective.\n\nThe paper further presents a convergence analysis for distribution-entropy-regularized policy iteration (DERPI), showing that it enjoys similar guarantees as both standard RL and MaxEnt RL. However, they emphasize a key difference: unlike MaxEnt RL, where exploration is explicitly encouraged, DERPI promotes exploration implicitly through the additional reward signal coming from modelling the return distribution, and encouraging the policy to pursue a \"target\".\n\nThe theoretical insights are supported by experiments on both Atari and MuJoCo environments. Also, this work shows that combining both types of regularized exploration (distributional and MatEnt) can sometimes reach better performances.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\nThis work revisits the benefits of a specific form of distributional RL, i.e., CDRL, and provides a rigorous comparison to standard RL. It clearly shows, by formula, the additional informational used by CDRL. Furthermore, due to the implicit exploration behavior in distributional RL, the paper also theoretical compares to MaxEnt RL. This work makes such comparison clear and also quite elegant.\n\nOverall speaking, this work provides nice insights to the relationships among standard RL, distributional RL, and MaxEnt RL. \n\nAlso, the presentation is smooth and well-structured, with the logical flow of ideas.\n\n## Weaknesses\n\nI don’t have much to complain about. However, \n\n1. The analysis is limited to CDRL, rather than general distributional RL frameworks. While this is explicitly acknowledged in the paper, the readers should be careful when intending to generalize the conclusions beyond CDRL.\n\n2. When using UE, the method appears to introduce some instability during training. It could be worth discussing this behavior in more detail.\n\nQUESTIONS:\n1. There are additional parameters involved, e.g. the parameter $\\\\beta$ in MaxEnt RL. I may have overlooked it, but how is this parameter set in your experiments?\n\n2. Also, when combining the two types of exploration in DSAC, how do you choose the corresponding parameters to balance their contributions? From Figure 4, it seems this combination introduces larger fluctuations in the learning curves. Do you have any insights on whether there are ways to stabilize it?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper attempts to shed light on the improved performance seen in deep RL when using distributional RL, and in particular categorical distributional losses, by decomposing the loss into a mean-related and entropy regularization term. The argument is that this (distribution-matching) entropy regularization term implicitly, but also explicitly in this paper, encourages the policy to visit states/actions where the distribution is poorly estimated.\n\nThe primary contributions include this decomposition, theoretical results supporting this decomposition and the correctness/convergence of an explicitly regularized policy evaluation method, empirical evaluation of the explicit form compared with C51 and DQN, and a study in MuJoCo environments on the combination of this form of regularization with the standard form of entropy regularization.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n* There is actual novel insight here with rigorous theory and informative experiments to back it up.\n* The experimental work asks exactly the right set of questions (for the scope of a conference paper). In Figure 3 we quite clearly see how this explicit form the regularization leads to performance which largely interpolates between C51 and DQN as the strength of regularization is varied. Figure 4 studies a very natural and interesting question of combining with policy entropy regularization, although I feel that this result did not get quite enough discussion for it to fully land.\n* The paper is generally well written and the flow of the paper introduces us to this idea very effectively.\n\nWeaknesses:\n\n* The notation is a little rough. It is precise, but there is a fair bit to keep track of and I found it a bit less intuitive than it could be. In particular, I keep wanting to treat mu as the mean-related term and forgetting it is the opposite. Lots of super- and sub-scripts, there’s a lot to keep track of.\n* An intuitive explanation of the effects of this form of regularization could be improved. The phrase “lags far behind” seemed like it kept getting used, but I really wanted more investigation and discussion of how to understand the effects here. Overall, I came away from reading this able to parrot that concept but lacking a more complete intuition for what that would really mean in terms of affecting behavior.\n\nQUESTIONS:\n* On line 239 the text says “we use uppercase notation…” and then proceeds to give an example “(s_t, a_t)” which is lowercase. Uppercase for random variables is ideal, but this juxtaposition was confusing and I thought maybe a typo.\n* There are other methods that use categorical losses without attempting to estimate the distribution of returns (at least not fully) such as so called TwoHot encodings and HL-Gauss. Could your approach be adapted to analyze these methods as well? Or in these cases do you think that there would not be a related regularization effect?\n* Can you say more about why this form of regularization *should* be expected to be useful, or when it might be counter productive?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper seeks to deepen the understanding of the benefits of the Categorical Distributional Loss. It does so by decomposing the return distribution estimated in distributional RL into the Dirac delta distribution of the mean and the remaining distribution capturing higher-moments information, and then applying the distributional analogue of the neural fitted Q iteration to analyze the objective that categorical distributional RL is minimizing.\n\nThis decomposition and analysis uncovers a connection between distributional RL, whose objective seeks to reach states where the cross-entropy between the estimation of the distribution at the current state and the distribution at the target state is more, and maximum entropy (MaxEnt) RL, whose objective aims to reach states where the entropy of the policy is maximized.\n\nExperiments in Atari and MuJoCo seek to validate this analysis. First, by showing that the performance of Categorical DQN can be smoothly reduced to DQN by varying how much the algorithm focuses on minimizing the cross-entropy term. Second, by comparing exploration caused by the cross-entropy minimization versus the action entropy maximization.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n* The analysis of the paper seems solid\n* The appendix includes a lot of details and is well constructed.\n* The experiments are chosen well to validate the hypothesis\n\n## Weaknesses\n* The way the idea is communicated can be improved further. The core result of the paper is hard to parse unless the reader is familiar with a lot of the related literature. But given the deep analysis of this paper, perhaps that is the audience they are tailoring it for.\n* The experiments on the Atari domain should be replicated in MuJoCo if possible to showcase that the idea holds in discrete and continuous action spaces as well as with value based and actor critic based algorithms.\n* The results in the bottom row of Figure 4 are mixed, but the paper merely presents a hypothesis for the mixed results instead of validating the hypothesis.\n* Minor point: Evaluations with 3 seeds in Atari and 5 seeds in MuJoCo might not be statistically sound, but consistency across the different tasks in these two domains points to some validity.\n* Minor point, another paper that has used a mixture of distributional RL and MaxEnt RL to great success is [1]. Perhaps that paper validates or disproves some of the hypotheses in this one.\n\n### References\n[1] Outracing champion Gran Turismo drivers with deep reinforcement learning., Wurman et al., Nature, 2022\n\nQUESTIONS:\nThe last two subsections of Section 5.1 indicate that the policy seeks states that have a higher discrepancy between the distribution estimate at that state and the return distribution from the next states. However it is not made clear how the policy is affected over predecessor states where this discrepancy might not have propagated yet, and also how that discrepancy might propagate back. It is not simply a question of comparing to MaxEnt RL since there the entropy regularization is not an error term.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}