Normality-Guided Distributional Reinforcement Learning for Continuous Control
Authors: Ju-Seung Byun
Andrew Perrault
Venue: Accepted by TMLR ()

Abstract: Learning a predictive model of the mean return, or value function, plays a critical role in many reinforcement learning algorithms. Distributional reinforcement learning (DRL) has been shown to improve performance by modeling the value distribution, not just the mean. We study the value distribution in several continuous control tasks and find that the learned  value distribution is empirically quite close to normal. We design a method that exploits this property, employing variances predicted from a variance network, along with returns, to analytically compute target quantile bars representing a normal for our distributional value function. In addition, we propose a policy update strategy based on the correctness as measured by structural characteristics of the value distribution not present in the standard value function. The approach we outline is compatible with many DRL structures. We use two representative on-policy algorithms, PPO and TRPO, as testbeds. Our method yields statistically significant improvements in 10 out of 16 continuous task settings, while utilizing a reduced number of weights and achieving faster training time compared to an ensemble-based method for quantifying value distribution uncertainty.

Published in Transactions on Machine Learning Research (06/2025)
Normality-Guided Distributional Reinforcement Learning for Continuous Control
Ju-Seung Byun'
byun.830osu.cm edu
Deportment of Computer Science and Engineering The Ohio State University
Andrew Perrault
perroult. 17 edu
Department of Computer Science and Engineering The Ohio State University
Reviewed on OpenReview: https: // openreview. net/ per/ffrmm? id= =27hb
Abstract
Learning predictive model of the mean return, value function, plays critical role in many reinforcement learning algorithms. Distributional reinforcement learning (DRL) has been shown to improve performance by modeling the value distribution, not just the mean. We study the value distribution in several continuous control tasks and find that the learned value distribution is empirically quite close normal. We design method that exploits this property, employing variances predicted from a variance network, along with returns, to analytically compute target quantile bars representing normal for distributional value fonction. In addition, propose policy updale stralegy based on the correctness as measured by structural characteristics of the value distribution not present in the standard value function. The approach we outline is compatible with many DRL structures. We use two representative on-policy algorithms, PPO and TRPO, as testbeds. Our method yields statistically significant. improvements in out of continuous task settings, while utilizing a reduced number of weights and achieving faster training time compared to an ensemble-based method for quantifying value distribution uncertainly.
1 Introduction
In reinforcement learning, an agent receives rewards by interacting with the environment and updates their policy maximize the cumulative reward or return. The return often has high variance, which may make training unstable. To reduce the variance, actor-critic algorithms Thomas 2014 (chulman et al. 2015b; Mnih et al. 2016 Gu et al. 2016) use learned value function, model of the return that serves a baseline to reduce variance and speed training.
The standard approach to learning a value function estimates a single scalar value for each state. Several recent studies learn value distribution instead, attempting to capture the randomness the interaction between the agent and environment. One method for achieving this involves using multiple value functions (referred as an ensemble) so that these functions collectively represent the value distribution (Lee al. An alternative method requires the value function produce multiple outputs, reprosented quantile
2020
bars Bellemare el al. 2017a Barth-Marm et. al. 2018 Dabney et al. 2017 Singh et al. 2020 Yue et al.
2020 These techniques, known as distributional reinforcement learning (DRL), have proven to enhance stability and expedite the learning process. The target for generating quantile bars is commonly derived from the distributional Bellman optimality operator, and the distributional value function is fitted to the target with the quantile regression (Bellemare et al. 2017a Dabney et al. 2017
"Corresponding author

---

Published in Transactions on Machine Learning Research (06/2025)
Our goal is to derive quantile bars based the Markov chain central limit theorem (MC-CLT) rather than the distributional Bellman optimality operator, and to introduce novel uncertainty measure absent in standard distributional value functions, applying it to policy updates. The MC-CLT indicates that in continuous tasks, the variance of the return keeps decreasing as the timestep goes. Moreover, when sufficient remaining timesteps exist, the return conforms to a normal distribution. We examine the value distribution trained using quantile regression and find, in continuous tasks, the predictions of the distribution closely resemble normal distributions, as described by the MC-CLT and depicted in Figure [I This contrasts with the distributional Bellman optimality operator, which causes the variance of the value distribution to increase over timesteps (Figure 2), due to the multiplication of y < 1 with the next state and action Q value (Equation 4. Section 4.2 provides a detailed example of this variance increase.
In this paper, we assume that the value distribution contimuous tasks is governed by the MC-CLT rather than the distributional Bellman optimality operator. To appropriately guide the distributional value function, we provide analytically computed quantile targets representing normal distributions, grounded in variances sourced from variance network (Nix & Weigend| 1994 Kendall & Gal, (017) and returns. We assess the accuracy of the current predictions made by the distributional value function. We aim to leverage distributional information, prioritizing values that are accurately estimated during policy updates.
As we consider the reliable values for policy updates, we incorporate existing actor-critic algorithms (Tee et al.) 2020 Mai et al. 2022) that factor in uncertainty during policy updates as a comparison. These algorithms typically minimize the influence of samples with high variances. However, some states inherently possess elevated variances. For instance, the true variances are higher near the initial timestep because the expected return from each timestep gets smaller timesteps goes on (Appendix D The question arises: Should states having higher variances have less impact policy updates? Although the predicted variance is correct for a state, its impact could be lessened simply because its variance is high. In contrast, we determine the impact of policy updates by evaluating how closely our distributional value function predicts the quantiles for normals. Given that our distributional value function is fitted to a normal quantile target, a well-trained distributional value function should yield quantiles that closely align with the shape of the normal distribution. This alignment indicates the function's accuracy in reflecting its models' underlying distribution. In Appendix E provide additional discussion on tasks where the normality assumption may not be applicable, well as experiments in the opposite direction, where samples that are not closely aligned are given higher weights for exploration.
In summary, our key contributions are:
We examine that the distributional value function approximates a normal for continuous tasks, and this normal's variance increases with each timestep. To address this, we guide the distributional value function to produce quantiles representing a normal distribution, leveraging variances sourced from the variance network (Nix & Weigend 1994 Kendall & Gal 2017) and sampled returns.
Under the normality assumption, we measure the uncertainty (or correctness) of the value distri- bution for each state, rather than relying the uncertainty derived from multiple value functions (ensemble). We propose an uncertainty-based policy update strategy that assigns a high weight to a correctly predicted state.
We provide an empirical validation that uses PPO (Schulman et al. 2017 and TRPO Schulman et al. 2015a) on several continuous control tasks. We compare our methods to standard algorithms (PPO and TRPO) as well as the ensemble-based approach. We find that our method exhibits better performance than the ensemble- based method in 10/16 tested environments, while using half as many weights and training twice as fast.
2
Related Work
We briefly review related work on distributional reinforcement learning (DRL) and supporting mothods. Bellemare et al. (2017a) studied the distribution of the value network for deep Q-learning and proposed C51, DRL algorithm that uses categorical distribution to model the value distribution. C51 produced
2

---

Published in Transactions on Machine Learning Research (06/2025)
Hopper v2
Hopper v2
Hopper v2
T-500
T-1000
30-
32-
-400
-20
180
2000
260
Quantlle
Walker2d-12
Walkar2d-12
Walker2d-22
1.0
T-1
18-
T+500
10-
T-1000
04
-15.00
Quantle
Quantie
Quantic
Figure 1: Results of trained the distributional value function VD.- with the Huber quantile regression. Each line represents different timesteps: the initial step T 0, the intermediate step T 500, and the last step 7' 1000. Within each cell, the quantile output of yD,= is represented by bars, the corresponding cumulative distribution function (CDF) is depicted in orange, and the computed normal CDF based on the bars is shown in green. The policies are updated with PPO and vD* are updated with the quantile Huber loss. The number of quantile bars is 200, but only 45 are shown for visualization. Notably, for continuous lasks, the distribution predicted by VD,* aligns remarkably well with the normal distribution, even at the final step.
state-of-the-art performance on the Arcade Learning Environment (Bellemare et al. 2013). but required the parameters of the modeling distribution to be fixed in advance. [Danney al. (2017) proposed QR-DQN, which leverages quantile regression and allows for adaptive distribution structure. In addition, they resolved key thcoretical question with KL divergence (DKl), which could lead non-convergence, by replacing it with Wasserstein distance metric and showing convergence of the distributional value network. While these methods focus solely on the mean of the value distribution, our approach seeks to leverage the distributional characteristics through the normality assumption as well.
Several extensions of DRL to continuous lasks have been proposed, including distributed distributional de- terministic policy gradients (D4PG) Marth-Maron et al., 2018), sample-based distributional policy gradient (SDPG) (Singh et al. 2020) and implicit distributional actor-critic (IDAC) (Yue et al.) 2020). Our method is model agnostic and compatible with any existing DRL algorithm that learns the value distribution with single neural network.
Compared to supervised learning, reinforcement learning has an unstable learning process, and uncertainty is often modeled to make RL learning more stable. We can decompose uncertainty into aleatoric and epistemic (Hora] 1996: Kiureghian & Ditlevsen] (209)). Aleatoric comes from the stochasticity of data and is often measured with the attenuation loss function (Nix & Weigend, 1994). Epistemic uncertainty is caused by the limits of generalization of models and can be measured by dropout (Kendall & Gal 2017 or ensemble methods {Lakshminarayanan et al.) 2016). The RL community models these two types of uncertainty to update the model safely or to support sparse reward functions (Bellemare et al. 2016 Strehl & Littman, 2005 Burda el al.) 20118). Clements et al. (2019) penalizes actions with high a aleatoric uncerlainty, and epistemic uncertainty used to guide exploration. To our knowledge, Moerland et al. (2017) is the only previous work assuming a normal distribution for the return distribution, which they use to construct a bootstrap estimate of combined uncertainty. SUNRISE (Lee et al. (2200) measures epistemic uncertainty with ensemble technique and uses it to reduce the gradient of sample with high uncertainty. Mai
3

---

Published in Transactions on Machine Learning Research (06/2025)
et al. 2022 leverage the two types of uncertainty with the combination of ensemble and attenuation loss. We ut utilize quantile bars, which represent the return's distribution, as a proxy for epistemic uncertainty. This proxy is less iesesivee in termss of training compared to ensemble methods, which significantly increase training costs (sce Appendix of Mai et al. (2022)).
3 Preliminaries
3.1 Reinforcement Learning
We consider an infinite-horizon Markov decision process (MDP) PPuterman 2014 Sutton & Barto, 2018)
defined by the tuple .M = (S, P. The agent interacts with an environment and takes an action E A according to policy Te(a, (x)) parameterized by 0 for each state 8, E S at time t. Then, the environment changes the current state St to the next state sc+1 based on the transition probability P(s az). The reward function
A - provides the reward for (st, at). y denotes the discount factor and μ is the initial state distribution for so. The goal of reinforcement learning (RL) algorithm to find a policy my that maximizes the expected cumulative reward, return:
0" = argmax
Σ o ((((,
(1)
ae aet SL+ p(-
In many RL algorithms, a value function (st) is trained to estimate the expected return under the current policy, E| 7 R(se;ap (r)], and updated with the larget:
(T(()) = R(st, + VV where 8+11 a()
(2)
3.2 Distributional Reinforcement Learning
In distributional reinforcement learning (DRL), Bellemare et al. (2017b)
introduced an action-state distri- butional value function QD."(s, a) which models the distribution of returns for agent taking action a and then following policy т. We model the distribution by having DD* output N quantile bars {go, It.... IN-
Similar to the Bellman optimality operator T (Watkins Dayan 1992) (Equation 3) for the state-action
value function Q", Q updated by distributional Bellman optimality operator (Equation 4p:
T*Q" at) = [E[R(ss az)l + + mmxx Q" (5t+1,
B1+1
(3)
where 8411 P(-ls,, a111 = argmax (76411,,
T.Q (t) R(s,, az) +7Q 7 4x+1)
(4)
where St41 P(-(s at), ar+1 argmax,,.. (S(41,
3.3 Markov Chain Central Limit Theorem
We introduce the Markov Chain Central Limit Theorem (MC-CLT) (Jones 2004 (Theorem 1
We leverage
this theorem to argue that returns are normally distributed in some continuous environments.
Theorem (Markov Chain Central Limit Theorem). Let X be a Harris ergodic Markov chain on with stationary distribution p and let f X + R is Borel function. If X is uniformly ergodic and {,,
4

---

Published in Transactions on Machine Learning Research (06/2025)
Hopper-v2
Walker2d-v2
BreskoutkooFramcccc....
Spacemvedeshhfframssato v4
98
Deviatton
013
0.3
030
323
Timesteos
Timesteps
Timestaps
Timestops
Figure 2: The standard deviation computed from each the distributional value functions trained on the distributional Bellman optimality operator (Equation with exponential smoothing. The estimated standard deviation increases as timestep increases.
then for any initial distribution, as nt + oc,
vnlfn [BP(S) 4 o7),
where
In = Σ ((Xi) 0. E/
(5)
tml
0² = Var, +2 Σ Cov (ov,
To relate this to RL, we let f be the reward function and X be the state-action space. the Markov chain induced by a policy Harris ergodic, we expect that the theorem holds (as rewards being a Borel function mild condition). In detail, a return G(s,) is consists of the sum the rewards Ei=, R(s,, a.). Thus, for sufficiently large T, the distribution of G(sz) tends towards a normal distribution. This might not consistently be the case, particularly when / closely approaches T and R(s,,) deviates from a normal for each possible action. Nevertheless, a distributional value function, trained using the Huber quantile regression, outputs distribution that closely represents normal for the final timestep, even for terminal states (Figure 1. It's also worth noting that the presence of transient states can disrupt Harris ergodicity. However, this issue is not prevalent in many continuous control tasks, such as robotics environments in OpenAl Gym, since the optimal policy is stable and repetitive.
4 Approach
In this paper, we introduce method that leverages the distributional property of value distribution, assuming normality in continuous tasks. The objective of this approach to determine the impact of individual samples policy updates. We argue that existing quantile distributional value functions yield estimates where variance increases with timestep. To handle this issue, we guide our distributional value function D.= using quantile bars of a normal distribution derived from variance and return. Furthermore, we discuss how to mcasure the uncertainty (or accuracy) of the current ostimation of DDx for each state and incorporate this measure into policy updates. We also further discuss scenarios where a lask doesn'l I'l satisfy the normality condition in Appendix
4.1 State Distributional Value Function
We first similarly introduce state distributional value function VD. that also outputs quantile bars for stale 8. The distributional value function (s) for the general actor-critic policy gradient is only dependent on the state. To be specific, the target for VPD:(st) R(s;, at) + VD, similar to Equation (4). The quantile bars of (s,) shifted by the amount of R(s,, az). This let us use the actor-critic algorithms such as Policy Optimization (PPO) (Schulman et al. 2017 and Trust Region Policy Optimization (TRPO) (Schulman et al. 2015a).
5

---

Published in Transactions on Machine Learning Research (06/2025)
VD, = (4o 4p 92 93}
N (gvg Lug)
(') 1
16
26
20%
20%
20%
20%
20%
90 91 91vg
92 93
9
qi
9
qi
40
91
92
93
Figure 3: Illustration of how we measure uncertainty, When N = 4, there are 4 quantile bars {90. 91, 92: 93) as an output of yD.. Compute the mean lavg of the quantile bars, and find o, such that P(Z I 4: Qurz (i+1) = *#1 for i E {0, 1.3. We consider VD as approximately N (qavg. lagg) where davg = Eimo 3 Tz. We find the exact quantile location {96- 41 22 d3s of N (qavg: Lugg) then measure how much two types of quantiles are different with mean squared error.
a; = 0.2*
4.2 Analysis of Distributional Value Function
Motivated by Theorem 1, examine the learned distributional value functions VD.= across different states and at several timesteps at convergence (Figure These distributions appear close to normal distributions.
Furthermore, our observation reveals that the learned VD,= and Q exhibit. an increase in variances as timesteps progress, contradicting Theorem 1. The distributional Bellman optimality operator (Bellemare et al. 2017b) causes the variance of the distributional value function to increase with timestep as the discount factor shrinks quantiles. To illustrate, suppose Q*:*(s++1, ac+1) = {go- Q1.... -N-13 } timestep + 1. Then, the target for QD (st,(s at) under the distributional Bellman operator will be {R(se, oe) 79o, R(se, at) "I,,..., (,,,, ae) + 7 i.e., {go, 91. - } is shifted by R(s,, ae) and shrunk by y < 1. Consequently the range of the distribution for timestep is narrower than the range for timestep t + 1.
We test this empirically by training distributional value functions with PPO and QR-DQN using the dis- tributional Bellman optimality operator (Equation 4 and graphing the estimated standard deviation from quantile output for each state (Figure D Indeed, the variance estimates appear to increase with timestep on average. We now propose a method that utilizes the variance network and returns to generate quantiles for the distributional value function.
4.3 Variance of Normal Distribution
Variance network (a introduced by Nix & Weigend (1994) and named by Mai et al. (2022 are designed to capture both the mean and Gaussian error (variance). In line with the normal assumption, the variance network allows us to estimate the variance of Gaussian distribution. We leverage a variance network to estimate variance for each state when computing quantile bars.
In the original variance network, the mean and variance are predicted jointly within single network. We separate these predictions: the variance network oZ solely predicts the variance, while the mean is predicted by vyD like the original actor critic method.
T Σ
(v)(w)
20 (x) | G(s))l Ino
(6)
where (s) is the mean of V, (s) and G(s) is the sum of sampled rewards from the state (return). This separation of prediction climinates the need for additional hyperparameter tuning that would have been required for the simultaneous prediction of mean and variance, as observed in Mai et al. (2022). Although the numerator of Equation 6 is separated from eb the update is performed in the direction of increasing of when the error of the numerator is significant. Conversely, if the error is sufficiently small, of is decreased. 2 is trained to predict variance in manner consistent with the original variance network.

---

Published in Transactions on Machine Learning Research (06/2025)
4.4 Target for Distributional Value Function
We provide the quantile bars of approximated normal distribution to V. with the variance from o above and returns. To compute the target quantile bars, need the mean of the approximated normal distribution. The mean can be TD target or the sum of rewards G(s) from to end. Here, use return G for the sake of notation simplicity. To obtain a quantile target for V DD.=, we first precompute the Z values of the standard normal distribution. For example, when N == 4, Z = { {-0.841, -0.253, 0.253, 0.841), i.c., 1+1 P(X <= [[] 0.2 * (i + 1) = N+I for E 2,33. Note that we use = 8 quantile bars for all experiments practice, as we empirically find that N 2 provides satisfactory performance. Then, we compute the target quantile bars % of (st) such that de = { + oz i E {0, .., N V and fit Vg to o by quantile Huber loss (Aravkin et al. 2014). The quantile Huber loss makes Vo output quantile bars that match the definition of quantiles. The Huber loss (Huber 1964 is as follows:
_u², if |u| < & m(lu) 1n), otherwise.
(u)
(7)
The quantile Huber loss is an asymmetric variant of the Huber loss.
(2 TN Σ e (g((5)) -(..))
1
(8)
where (u,(u) |T: - ( (u)
7 it1 1
for i {0, 1.. --11 1}
+
4.5 Uncertainty Weight
Since V is distributional value function, can use the properties of the distribution to calculate the uncertainty weight for policy updates. We measure the distance between the current quantile output of D and the normal distribution computed based on DD quantile output. We evaluate how close DD is to the target normal distribution to judge whether VD,= generalizes well for the visited states, under the hypothesis that a more accurate DD, will lead to better policy improvement. For example, when DD does not align with normal quantile bars for a state, the discrepancy between VD,r and its target becomes substantial. We discount the impact of such samples having high discrepancy (or high uncertainty) on policy updates to focus more on reliable values that are accurately estimated from the distributional value function.
Figure 3 illustrates this when the number of quantile bars is N = 4 for simplicity. In practice, we use N = 8 quantile bars for all experiments, as we empirically find that N 2 8 provides similar satisfactory performance. If the prediction of D, is reliable, it should be symmetrical with respect to the mean of the quantile bars from our normality assumption, and the location of the quantile bars should follow a normal distribution. We first find the mcan (svg(s) = N Σ -1 {oo, o1....,N, NN-1) for cach quantile such that P(Z <
(s) of the quantile bars and calculate the standard deviations qi qi-qork)
i+1 for i E 10, - 1}. We consider N+1
a;
D. is trying to approximate N ((gavg(s), 22gg) where Javg N E N os.
To measure the uncertainty w for policy updates, we find the locations of each quantile (s). -v (s)} based on ((gagg(s), ( 22g))- The better yD, resembles the normal distribution, the smaller the difference between these two types of quantile bars. Therefore, use the mean sqaured error E Di (
Ni (q.
2, and we compute the uncertainty weight w(s) as follows:
w(s) = o +
(9)
where T is a temperature and a is the sigmoid function. Although Loe et al. (2020) suggest values for T, it seems that finding the reasonable value for T still requires some tuning. We instead set a target uncertainty weight and perform parametric search by adjusting T to achieve that target weight (Appendix B
7

---

Published in Transactions on Machine Learning Research (06/2025)
Algorithm MC-CLT with Uncertainty Weight
Input: policy x, distributional value function Vg variance network og rollout buffer B
2: Initialize π, VD.=, and oi
for i 1 epoch mum do
for
to rollout num do
a, (-1se)
6:
8r+.1 P(-lst,,
7:
Compute N (gavg(s), avgg) with (st) = Qx.
8:
Find g'(s)) from (favg(8), o2gg) avgg)
Compute the mean squared error (
((())))
10:
Store (st, dg, r(st, ae), E, V. o (st) in
11:
if 81+1 is terminal then
12:
Reset env end if end for
13:
14:
15:
Perform the parametric search to find the temperature (Appendix B
16:
Compute target quantiles for VD,r with the stored return and o Minimize (v²(v) and (0)
17:
18:
Optimize x with policy objective scaled by
19:
end for
4.6 Policy Update with Uncertainty Weight
We leverage the uncertainty weights w to update the policy to mainly focus on values for which the current y DD predicts correctly. Each reinforcement learning algorithm has a policy objective function. We scale the objective function with the uncertainty weight to adjust the influence of the gradient for each sample. For TRPO (Schulman et al. 2015a),
no (arl(s))) A Toald (ar (sc)
maximize E, wss)
0
(10)
subject to E, DEL (Tows (188))] <
where ny is a policy and A, is advantage estimator timestep t. The more accurate the prediction of D.sr, the closer the value of w is to 1; hence the gradient is more significant than other inaccurate samples. Note that we normalize the advantages when we sample a batch for optimization in practice. Therefore, the magnitude of the gradient not changed, and we do not need to tune the learning rate as a consequence of introducing the uncertainty weight.
We proposc a normality-guided algorithm with the uncortainty weight to improve policy for PPO and TRPO in Algorithm I but this method can be applied to various policy update algorithms well.
5
Experiments
5.1 Experiment Setups
Our implementation is based on Spinning Up (Achiam| 2018
open-source resource by OpenAl that pro- vides implementations of several reinforcement learning algorithms and tools to help rescarchers get started with deep RL. For our experiments, we chose two representative deep reinforcement learning algorithms, Pol- icy Optimization (PPO) (Schulman et al. 2017] and Trust Region Policy Optimization (TRPO) (Schulman) et al. 2015a), to evaluate our method. The advantage is computed by GAE (Schulman et al.) 2015b).
We use the default hyperparameters such as learning rate and batch size. All policies have a two-layer lanh network with 64 32 units, and all of the value function and the distributional value function has
8

---

Published in Transactions on Machine Learning Research (06/2025)
Table 1: Ablation study of TRPO and PPO. Each entry represents an average return and a standard error, derived from 30 different settings, with each sctting running for 100 episodes. VD, (QR) represents the dis- tributional value function trained with the Huber quantile regression. This table illustrates an improvement trend across the scores with the addition of each component from the baseline to MC-CLT. Also, MC-CLT results generally better than the Ensemble method, while using fewer weights and reduced training time. An asterisk (*) indicates a statistically significant improvement between MC-CLT and Ensemble method at significance level of 0.05
Bipedal Walker
Hopper
HalfCheetah
Ant
TRPO TRPO +V D,= (QR) MC-CLT TRPO w/o w
115 + 3.2
2724 + 31
2749 + 37
2811 + 16
168 + 3.1
2893 + 28
2633 + 35
2962 + 21
186 = 3.1
2835 27
2984 39
3107 = 18
Ensemble TRPO w/ w
180 = 2.7
2742 29
3212 = 44
3368 + 19
MC-CLT TRPO (ours)
195 + 2.6*
2927 + 27
3172 43
3329 +
Swimmer
Walker2d
InvertedDoublePanduumn
LunarLander(C)
TRPO
121 + 0.2
2558 26
7969 + 109
198 3.1
TRPO +VD,* (QR) MC-CLT TRPO w/o w
117 + 0.2
2398 + 25
7878 + 112
227 2.6
118 0.4
2616 + 30
7662 + 114
245 + 2.2
Ensemble TRPO w/ w
116 + 0.2
2834 + 24
8566 + 85
242 + 1.8
MC-CLT TRPO (ours)
118 = 0.2
2904 + 23"
8404 + 93
258 = 1.8*
BipedalWalker
Hopper
HalfCheetah
Ant 1595 + 10
PPO
179 + 2.9
2973 + 27
2902 + 42
PPO D, (QR)
208 + 2.7
2778 30
2574 33
1742 + 11
MC-CLT PPO w/o
215 2.4
3184 23
2870 39
1743 +
Ensemble PPO w/
227 2.2
3208 22
3068 43
1863 + 12
MC-CLT PPO (ours)
236 + 2.3*
3230 22
3110 39
1900 13
Swimmer
Walker2d
InvertedDoubieeendulum
LunarLander(C)
PPO
122 0.1
2198 + 33
8814 + 72
229 + 2.1
PPO +VD (QR)
112 0.8
2037 + 34
8913 + 64
250 + 2.4
MC-CLT PPO w/o
117
2610 + 35
9187 + 41
268 + 1.7
Ensemble PPO w/ w
119
2669 34
8933 64
256 + 1.8
MC-CLT PPO (ours)
122 0.1"
2716 + 33
9225 + 36
278 + 1.2
two-layer ReLU network with 64 64 units 128 128 units for all environments. All networks are method on continuous OpenAl
updated with Adam optimizer (Kingma & Ba 2014h. We evaluate gym Box2D [Brockman et al.] 2016) and MuJoCo tasks (Todorov et al.) (0022), these environments have continuous action spaces and dense reward functions to use the normal approximation of MC-CLT (Theorem D
5.2 Detailed Ablation Analysis and Comparison
There exist three components that separate MC-CLT from the baseline. We perform ablation study to analyze the impact of each. The first component involves substituting the standard value function with the distributional value DD trained with the Huber quantile regression. We refer to this as PPO DD (QR) and TRPO +V (QR). The second is the addition of the normal larget for vD,= but without incorporating the uncertainty weight, denoted as (MC-CLT w/ w). Lastly, our method MC-CLT utilizing the uncertainty weight in policy updates to prioritize more reliable samples (MC-CLT w/ w). Furthermore, we compare our method with the ensemble- based approach with uncertainty weight, as their approach also measures uncertainties and integrates them into policy updates in the same manner. Thus, focuses

---

Published in Transactions on Machine Learning Research (06/2025)
Table 2: Training cost in terms of time and the required number of weights based on the baseline.
Baseline
MC-CLT
Ensemble-based
Time
1
1.5 - 1.9
2.8 - 3.1
# weights
1
2.0 - 2.2
evaluating which type of uncertainty more accurately reflects sample reliability: the one derived from the network's accuracy or the one based on standard deviation.
Table ! represents the experimental results across 8 environments. The first section of the table provides results from policies trained using TRPO, while the second section gives results from policies trained with PPO. As we incrementally add each component from the baseline to MC-CLT, consistent trend of per- formance improvement is observed. Also, our method shows better results compared to the ensemble-based approach in general. In 10 out of the 16 settings, we achieve statistically significant improvements between MC-CLT and the Ensemble method, as indicated by an asterisk (*) in the table, at significance level of 0.05. It's worth noting that the ansemble-based approach with TRPO exhibits better mcan performances three settings: HallCheetah, Ant, and InverteeDDuubleeendduum However, the differences are not stalistically significant.
In addition, MC-CLT utilizes fewer weights and achieves faster training time, as shown in (Table 2 Table 2 presents relative comparison with the other two methods based on the baseline. Specifically, when the training time for PPO is set to I, the MC-CLT method requires 1.5 to 1.9 times more training time compared to the bascline. Similarly, the ensemble-based approach takes 2.8 to 3.1 times more time. Additionally, the MC-CLT method requires approximately 2.0 to 2.2 times more weights, while the ensemble-based approach uses times as many parameters. We use 8 quantile bars for all experiments, as we empirically find that using 8 or more quantile bars provides smiliar satisfactory performance.
While MC-CLT requires hyperparameters for the number of quantile bars and the variance network, it necessitates fewer hyperparameters compared to previous methods (Lee et al. 2020 Mai et al. 2022). We observe satisfactory performance when the number of quantiles is set to 2 8 and the size of the variance network equals to that of the distributional value function. In practice, we fix the number quantiles at 8 for all experiments. Consequently, aside from the aspects controlling uncertainty, MC-CLT shares the same hyperparameters as the bascline method. The detailed hyperparamcter scttings are discussed in Appendix A
5.3 Seeking Accuracy vs. Seeking Exploration
MC-CLT and the ensemble-based methods seek accuracy by reducing weights for samples with high un- certainty. This raises exploration concerns regarding states where the current distributional value function fails to predict accurately. If such states are not given sufficient weight, the policy may never cover them effectively. To address this concern, we conduct additional experiments where states with higher uncertainty receive higher weights to encourage the agent to explore these states more. We compute the weight with w(s) = E * T + 0.5, where E - {s))), and run Algorithm 2 in MC-CLT to find the temperature T' (where left and right are adjusted in the opposite direction from accuracy seeking). Table 3 shows the comparisons between accuracy-see securacy-ceeking and exploration-sceking approaches. In all experiments, the accuracy-seeking approach shows better performance. This is likely because the environments use highly engineered reward functions, providing sufficient feedback without requiring additional exploration. For tasks where discovering novel states important, such as in Atari games (Bellemare al.] (003)) the exploration- seeking approach would have a positive effect on performance (Obando- Ceroo et al. 2023 Schaul et al. 2022
10

---

Published in Transactions on Machine Learning Research (06/2025)
6 Conclusion
We have presented a distributional reinforcement learning (DRL) method in that output quantiles approxi- mate the value distribution as normal distribution under the mild assumption for Markov Chain Central Limit Theorem (Jones 2004h. Existing actor-critic algorithms that assess uncertainty typically employ mul- tiple value functions (ensemble) to estimate variance. Given the presence of states with relatively high variance, we propose an alternative approach to measure uncertainly. Rather than relying on the variance, we evaluate how much the predicted quantiles are close to a normal.
Furthermore, when updating the distribution value function using the distributional Bellman optimality operator, observe that the variance estimates lend to increase with each timestep. The discount. factor of the operator yields variance proportional to timestep. To address this, guide the distributional value function by utilizing analytically computed quantile bars derived from returns and the variance network. Our method prioritizes accurate samples that are well-estimated by the current distributional value function, thereby promoting improved policy performance. Overall, our method exhibits better performance compared to the baselines in our evaluations. Additionally, it leverages a reduced number of weights, leading to faster training times than the ensemble-based method.
Although proposed method has been primarily discussed in continnous tasks, where the assumption is that the return distribution follows normal distribution, it also potentially works in other scenarios, such as lasks with discrete action spaces. Since we obtain the mean of the return through sampling, exactly in original actor-critic methods, it remains unbiased, and the uncertainty is calculated solely based on the deviation from the target that our distributional value function intended to fit. Additional insights and details are provided in Appendix
Reproducibility Statement
We provide the hyperparameters used our evaluations and all source code is available at Atttps://gtthn: SonmshaasaccksMC CLT
Acknowledgement
The authors would like thank the Ohio Supercomputer Center (Cenerr 1987 for providing the computa- tional resources usod in this rescarch.
References
Joshua Achiam. Spinning Up Deep Reinforcement Learning, 2018. OpenAL.
Aleksandr Y. Aravkin, Anju Kambadur, Aurélie C. Lozano, and Ronny Luss. Sparse quantile huber regression for efficient and robust estimation. ArXiv, abs/ 162224202,, 2014.
Gabriel Barth-Marow Matthow W. Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva TB, Alistair Muldal, Nicolas Heess, and Timothy P. Lillicrap. Distributed distributional deterministic policy gradients. CoRR, abs/ 1804.0004.. 2018. URL attp://arxiv. org/ abs/////////////// 086117
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253 279, jun 2013.
Mare G. Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Rémi Munos. Unilying count-based exploration and intrinsic motivation. CoRR, abs/1606.01868, 2016. URL http:
//arxiv.org/aaarl wwwp://///./..oo..r.r//// 01868
Mare G. Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. CoRR, abs/1707.06887,, 201 7a. URL http:///wwww...or//a//////////./
11

---

Published in Transactions on Machine Learning Research (06/2025)
Marc G. Bellemare, Ivo Danihelka, Will Dabney, Shakir Mohamed, Balaji Lakshminarayanan, Stephan Hoyer, and Rémi Munos. The cramer distance as solution to biased wasserstein gradients. CoRR, ab5/1705.10743, 2017b. URL
wwtp://rggbbbbb....ooo 10743
Grog Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. CoRR, abs/ 0t//1666.011540 2016. CRL
01540
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distil- lation. CoRR, 1////1880.128999 2018. URL http:///axiiy wwwrriiv..orrvae. http//abb//111111/11
Ohio Supercomputer Center. Ohio supercomputer center, 1987. URL http://osc..etu/ark::/1 f5s1ph73
William R. Clements, Bastien Van Delft, Benoîl-Marie Robaglia, Reda Bahi Slaoui, and Sébastien Toth.
Estimating risk and uncertainty in deep reinforcement learning, 2019. URL httpa////xxvvvor/aaaaaa
1905. 096388
Will Dabney, Mark Rowland, Mare G. Bellemare, and Rémi Munos. Distributional reinorrcemet learning with quantile regression. CoRR, abs/1710.10044, 2017. CRL http://aaravv.oo///aa////.....00
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic
methods. CoRR, abs/ abs/1180.09947,, 2018. URL
Shixiang Gu, Timothy Lillicrap, Zoubin Ghahramani, Richard E' Turner, and Sergey Levine. Q-prop: Sample- efficient policy gradient with an off-policy critic. arXiv preprint arXiv:1611.02247. 2016.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. CoRR, abs/ abs/1801.01290, 2018. URL http:
//arxiv. httpg//bs11801..0222
Stephen C. Hora. Aleatory and epistemic uncertainty in probability elicitation with an example from haz-
ardous waste management. Reliability Engineering & System Safety, 54(2):217-223,, 1996. ISSN 0951-8320.
doi: http::///bbboooy//00000//////--0 atticiaepp1/005518299000007 Treatment of Alcatory and Epistemic Uncertainty.
URL BtRpss//:://ww acwencadirect..cmmm
Peter Huber. Robust Estimation of a Location Parameter. The Annals of Mathematical Statistics, 35(:): 73 101, 1964. doi: 00221//aom//177700333 URL https::/ac..oorg//// /ttp://pp/////.ooo.oo/////////// 10t0//0000..2444//m///7777777222
Galin L. Jones. On the Markov chain central limit Cheorem. Probabilily Trobubbiity Surreyy,e.. (n9 320, 2004. doi: 10.144/1599588410000005
URL httpl//pe///aaaoo//////////////0
Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? CoRR, abs/1703.04977, 2017. URL http://arriv..og//aa///00..0999
Diederik P. Kingma and Jimmy Ba. Adam: A mothod for stochastic optimization. CoRR, abs/ tabs/14H 1412.6980, uni-trier
2014. URL http://hhwaarttate/////////aaaaa
Armen Der Kiureghian and Ove Ditlevsen. Aleatory or epistemic? does matter? Structural Safety, (22):105-111, 2009. ISSN 0167-4730. doi: http:r//l...ogg//.0000/sssttttte 000000000...00000000000000 URL
Risk Acceptance and
Risk Communication.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncer- tainty estimation using deep ensembles, 2016. URL
Kimin Leo, Michael Laskin, Aravind Srinivas, and Pieter Abbeel. SUNRISE: A simple unified framework for ensemble learning in deep reinforcement learning. CoRR, abs/2007.04933 2020. URL https: Inttps://arxi: ://axivv/
org/abs/2000 04938
12

---

Published in Transactions on Machine Learning Research (06/2025)
Vincent Mai, Kaustubh Mani, and Liam Paull. Sample efficient deep reinforcement learning via uncertainty estimation. CoRR, abs/2201.00 666, 2022. URL httl./ttees//arrvv.oooo/////////
Volodymyr Maih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukeuoglu. Asynchronous methods for deep reinforcement learning. In Inter- national conference on machine learning, pp. 1928-1937. PMLR, 2016.
Thomas M. Moerland, Joost Broekens, and Catholijn M. Jonker, Efficient exploration with double uncertain value networks. CoRR, abs/ abs/1711.10789, 2017. URL 10tp://2rax....rrbbbb///1/11999
D.A. Nix and A.S. Weigend. Estimating the mean and variance of the target probability distribution. In Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94), volume 1, pp. 55 60 vol.1, 1994. doi:
Johan Obando-Ceron. Mare Bellemare, and Pablo Samuel Castro. Small batch deep reinforcement learn- ing, 2023. URL
wttp/b/////b//ooo/oe 03882
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
Tom Schaul, André Barreto, John Quan, and Georg Ostrovski. The phenomenon of policy churn, 2022. URL
https://arxiv: http::///aaxww..o///aa///////// wrgg/ba////0a// 2aabba/2206.000 007300070000
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Francis Bach and David Blci (ods.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 1889 1897, Lille, France, 07-09 Jul 2015a. PMLR. URL http:///rroceednnss.oo mlr. www.spresseeeeeee v37 www//schulmmnn5.hhr..
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint orXiv:1506002458 2015b.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimiza- tion algorithms. CoRR, abs/ Abs/1707.06347 2017. URL
Rahul Singh, Keuntack Lee, and Yongxin Chen. Sample-based distributional policy gradient. CoRR, 0tee/2001.026552 2020. URL
TCML, pp. 856-863 2005. URL wwwpa/wwwww//////////0/00////
Alexander L. Strehl and Michael L. Littman. A theoretical analysis of model-based interval estimatiinn In 11411101111111111111111111
Richard S Sutton and Andrew G Barlo. Reinforcement learning: An introduction. MIT press, 2018.
Philip Thomas. Bias in natural actor-critic algorithms. In International conference on machine learning, pp. 441-448. PMLR, 2014.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RS] International Conference Intelligent Robots and Systems, pp. 5026-5033 2012. doi: 10.1109 10.1109/TROS.20 FRRSS.002.63866
Christopher JCH Walkins and Peter Dayan. Q-learning. Machine learning, 8(3):279 292, 1992.
Yuguang Yue, Zhendong Wang, and Mingyuan Zhou. Implicit distributional reinforcement learning. CoRR, abs/2000.00619, 2020. URL hattps:/aaat //tp//i//vv/./////////eae/aaaav/ http///xxxvoooaaaa/a///aaaaa hwtp://07..0000 06159
13

---

Published in Transactions on Machine Learning Research (06/2025)
A Hyperparameters
Deep reinforcement learning algorithms pose a significant challenge for evaluation duc to the inherent insta- bility and stochasticity during training. To ensure fair comparison with other algorithuss, we attempted maintain a consistent set of hyperparameters as much as possible. We follow Spinning Up (Achiam] 2018)) default scttings for learning rate, optimizer, etc. All policies have two-layer tanh network with 64 32 units, and all of the value function and the distributional value function has two-layer ReLU network with 64 x 64 units or 128 x 128 units for all environments.
Although Lee et al. 2020 suggest the formula to compute the uncertainty weight w(s)= = o + Wmin: it seems that finding the reasonable value for T not intuitive for various tasks. (Note that E is computed as the normality error of the distributional value function our method and replaced with the standard deviation of the value functions in Lee et al. (2020)). We instead set a target uncertainty weight and perform parametric scarch by adjusting T to achieve that target weight (Appendix B This process substitutes T with the target weight wtar a hyperparameter. Wtar is chosen from Wtar E (0.85, 0.9), and Wmin is chosen from Wmin E (0.4,0.. 0.5, 0.6].
MC-CLT has the number of quantile output nodes of the distributional value function as a hyperparameter. We examine how performance affected by the number of quantile bars, with values such as N 4, 12, and 20. We observe satisfactory performance when the number of quantile bars set N 8, so we fix the number of quantiles at 8 for all experiments. Also, the ensemble-based method introduces the ensemble size hyperparameter. As Lee et al. (020)) and Mai et al. (2022) demonstrate that ensemble size 5 shows the sufficient results, we set the ensemble size to 5 for these models.
In order to ensure a fair comparison with the ensemble methods, we selected 3 best configurations, each of which is trained with ten different seeds. The numbers reported in the tables correspond to the results from the last 10 runs.
B Algorithms In Detail
Instead of manually tuning T for each task, we dynamically adjust T' such that the average uncertainty lies within the range [Wtar - C, Wtar + c]. The following algorithm illustrates the parametric search employed to find such T. While MC-CLT utilizes the error set E, the ensemble-based methods leverage the standard deviation set. The term 2(1 - Wmin) ensures that each uncertainty weight lies within the range (Wmin: 1.0).
Algorithm 2 Parametric search to find the temperature T
Input: target weight Wtars minimum weight Wmins error set. E Initialize left 0, right 212
3: while left < right do
4: T= (left right)/2
5: W = 2(1 Wmin (-FE TT) Wmin
6: Compute the mean uncertainty weight Wavg = Avg(W'
7: if Wavg is the range [Wear - e, Wtar e] then
Use T to compute the uncertainty weight break
10: else if Wavg 2 Waar + then
11: left=
12: else
13:
right =
14: end if
15:
end while
14

---

Published in Transactions on Machine Learning Research (06/2025)
BipedalWalker-v3
Hopper-v2
0.90
Stanndard Deviation
6.17
K.1
6.4
Timestops
Timestops
HumanoidStanddup-22
Swimmer-v2
Standard Devlation
Standard Devlation
150
0:229
0:00
686
Timesteps
200
Timesteps
80
105
Figure 4: Changes in standard deviation over timestep. We train ensemble value functions and monitor the changes in standard deviations. The graphs indicate that the standard deviation consistently decreases as the timestop progresses.
C Variance Network
In practice, the variance network (Kendall & Gal 2017) trained to predict log variance Ino instead of predicting variance directly. This avoids a division by zero or a negative value the log term Equation (6). However, after predicting log variance, it is necessary to retrieve the actual variance value through the exponential operation, i.c., of exp(Inog).
However, in cases of environments having the large observation space, such as Ant-v2 or HumanoidStandup- v2, we frequently observe overflow issues during the exp operation, particularly in the early training epochs. Thus, we directly predict the variance by limiting the minimum value o² oa) We set = 0.0001, and this resolves the overflow issue for the large observation space environments. We also compared the outcomes of the two methods on tasks with smaller observation space. We couldn't see any meaningful difference between the two methods, we conducted experiments for all environments by directly predicting the variance.
Standard Deviation Change
Cortain states display relatively higher variances compared to others. For instance, fragile control tasks like Walker2d-v2, the states close to the terminal state typically exhibit higher variances. This is because once the agent overcomes the terminal states, several more successful actions incrcasing the sum of rewards (return). Additionally, the closer the state is to the initial timestep, the higher the variance. This correlation is apparent when comparing the changes in return from the initial and intermediate timesteps.
likely to be generated,
The graphs (Figure have been obtained by applying exponential smoothing to the standard deviation of the trained value functions (the ensemble-based method), Notably, the standard deviation is observed decrease the timestep.
15

---

Published in Transactions on Machine Learning Research (06/2025)
Mean
Figure 5: The green line depicts a non-normal complex distribution. The blue bars represent normal quantile bars, which are analytically computed using the mean of the complex distribution, with variance obtained from a variance network. The orange line illustrates the line version of the normal distribution derived from the quantile bars.
Motivated by these observations, propose a novel approach. Rather than decreasing the uncertainty weight of states with higher variance, we suggest a method for measuring the accuracy of the value function prediction. We then use this measure to obtain the uncertainty weight.
E
Normality Discussion
This section addresses scenarios where tasks deviate from the normality condition. Our method does not compromise the policy update, even when the normality condition not satisfied. Since we first compute the mean of the target quantiles from sampled episodes and then analytically compute the quantile bars (the variance is derived from the variance network), the mean represented by these quantile bars is unbiased.
In detail, suppose complex green line distribution D is the true return distribution of state under a policy as shown in Figure 5 A standard value function V*, which predicts a single value (return) (*((s) for a state, estimates the mean of this complex distribution D, indicated by the dotted line, through sampled episodes.
Rather than just outputting the single value, our distributional value function D, is trained to output blue quantile bars for s under the same policy n, representing normal distribution (the orange line). Although (s) represents the orange normal distribution, distinct from the green one, the mean of these quantile bars is also located al the dotted line position because the sampling mechanism are both the same. In other words, A - (s(s) = (s),,, where V (s) = {go 91 (s..... qs)) Given that this mean is used to update the policy, this alignment confirms that our approach without uncertainty weights is unbiased.
The question arises: What happens when we use the uncertainly weights to update the policy lasks devialing from the normality ?ondition? In this case, we argue that our method becomes a special case of Exploration by Random Network Distillation (RND) (Burda et al., 2018), in which random neural network is provided, and another neural network is fitted to this random network. While the model is fitted, exploration bonus is computed based the differences between the two models. Differing from RND's approach of providing a random neural network, we provide normal quantile target, expecting that VD will also produce quantile bars indicative of a normal distribution. We then measure how closely D predictions mirror the shape of a normal distribution.
Both approaches rely on the premise that the difference diminishes with sufficient learning. The distinction between RND and our method is that RND utilizes uncertainty bomus added to the reward, secking novelty, whereas our approach is a risk-averse strategy by reducing the impact of gradients from high- uncertainty samples. These different scarch strategics can be chosen depending on the problem. Given that the reward function in control lasks, such as MuJoCo, is highly engineered, being risk-averse is more effective than seeking novelty, as the feedback from the reward function is sufficient, unlike sparse reward settings. We actually conduct experiments to chock whether encouraging the agent visit high-uncortainty states is beneficial for our environments (Table B), and find that seeking exploration does not outperform in environments with highly engineered rewards.
16

---

Published in Transactions on Machine Learning Research (06/2025)
Table Seeking Accuracy vs. Seeking Exploration Table: We compute the uncertainty weight (our method) basod on Equation g However, this raises concerns regarding exploration. To address this, we conduct an experiment using Equation (2) to compule the exploration weight, ensuring that stales with high uncertainty are visited more frequently. Each entry represents average return and standard error, derived from 30 different settings, with cach setting running for 100 episodes.
BipedalWalker
Hopper
HalfCheetah
Ant
Exploration TRPO MC-CLT TRPO (ours)
34 3.3
2328 +
2459 32
1663 +
195 + 2.6
2927 27
3172 43
3329 + 20
Swimmer
Walker2d
InvertedDoubleePeddunnn
LunarLander((1
Exploration TRPO MC-CLT TRPO (ours)
36 2.2
1086 + 33
7628 + 112
179 + 4.4 258 1.8
118 + 0.2
2904 + 23
8404 = 93
Bipeda/Walker
Hopper
HalfChootah
Ant
Exploration PPO MC-CLT PPO (ours)
70 + 3.6
2727 + 28
2756 = 37
1315 + 10
236 + 2.3
3230 + 22
3110 + 39
1900 + 13
Swimmer
Walker2d
InvertodDoobboeenddulnn LunarLander(C)
Exploration PPO
44 + 0.8
916 + 24
9027 + 54
230 + 2.8
MC-CLT PPO (ours)
122 0.1
2716 33
9225 + 36
278 + 1.2
F
Limitation
Our current method is primarily effective in stochastic continuous tasks with dense rewards based the principle of MC-CLT. However, in environments where value distribution is non-normal, alternative meth- ods may be more effective for accurately capturing these non-normal value distributions. It's important to note that any limitations in our current approach don't lead to significant failures, as detailed in Appendix +
G Potential Research Directions
The use of our method does not exclude the possibility of utilizing the ensemble technique. In fact, by combin- ing our method with the ensemble, it becomes possible to leverage multiple distributional value functions. However, simply averaging the uncertaintics from each function has shown no performance improvement compared to our method alone. Besides weighting policy updates, exploring alternative uses of uncertainties from each distributional value function would be an intriguing avenue for future research. For instance, methods like Soft Actor-Critic (SAC) (Haarnoja et al. 2018) or Twin Delayed DDPG (TD3) (Fujimoto al. 2018). where the minimum value is selected among two Q functions, the Q function with lower uncertainty could be chosen to utilize more accurate value because our uncertainty weight represents the accuracy of the network.
17


Reviews:
[{"review_id": "mjdO1eDMfH", "reviewer": "Reviewer_5uVb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The observation of the normality distribution for continuous tasks and the corresponding design provides a new perspective for distributional reinforcement learning.\n2. The idea of the proposed approach is compatible with many DRL structures and can be applied to different policy update algorithms.\n3. The proposed approach uses fewer weights and has a shorter training time, reducing computational costs and resource consumption. And it outperforms the ensemble-based method in most tested environments, significantly improving policy performance with statistically significant improvements in 10/16 settings.\n\nWeaknesses:\nThere is no clear weakness.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper observes that the value distribution in continuous tasks is close to normal under certain conditions and use this property to design a normality-guided method for updating the distributional value function, which provides quantile targets conforming to a normal distribution to solve the variance problem caused by the traditional distributional Bellman optimality operator. Experiments are conducted using PPO and TRPO algorithms on multiple continuous control tasks and compared with standard algorithms and an ensemble-based method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "NA", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "cbNJoSPoKV", "reviewer": "Reviewer_Vrpn", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. The idea itself looks interesting and reasonable. The authors first identified the simlarity to a normal distribution and then were able to measure the uncertainty, which was used to derive weighted based updates.\n2. The authors conducted empiricall studies on a few MuJoCo domains, and the experiments demonstrated the improvement compared with other methods.\n3. This paper is presented well and mostly easy to follow.\n\nWeaknesses\n1. One claim the authors claimed is that the policy updates need to focus on the states the current value function predicts correctly. I can see the reasoning behind it. On the other hand, I have some concerns regarding the states the current value fuction fails to predict: If such states are not given enough weights, the policy would never cover them well. Subsequently, exploration in this algorithm will be an issue.\n2. Some ablation studies are necessary to validate a few claims.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors built on the successes of distributional reinforcement learning, and emprically observed that the distribution for the value distrbution is close to a normal distribution. Based on this observation, they proposed to separately estimate the variance for each state, and subsequently designed a weighted policy update strategy. The experiments demonstrated the superior performance of the proposed algorithm.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Please address the first comment regarding the weakness above. Perhaps you can run more experiment on other MuJoCo domains?\n2. When measuring uncertainty, why did you choose N=4 bars? How does this number affect the overall performance?\n3. The authors mentioned \"correct locations\" just above Equation (9). To my understanding, they are still estimated from samples so why are they \"correct\"?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "05G4291LnG", "reviewer": "Reviewer_CHAg", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The proposed method appears to outperform the baselines in several tasks in terms of final performance, while simultaneously using significantly less weights. The authors also claim that their method achieves this high performance with much less computations, but I could not find those results on the main paper. As far as I am aware, the method and the motivation are novel.\n\nThere are some important points that are not clear to me at this point:\n- The paper start by showing that the distribution of returns closely relates to a normal, empirically, and recalls a theorem that as far as I understand is supporting that hypothesis. Does this finding contradict the currently used methods for distributional reinforcement learning? The arguments in Section 4.2 appear to hint that way, but this was not clear in the paper. \n- In light of the above, the goal of the paper is not clear. If the goal is solely to provide a method that outperforms the currents methods for distributional reinforcement learning, it should be stated earlier in the paper. \n\nAs the biggest weakness of the paper, I would point that it is not very clear. For instance, in Figure 3, we don't have any scale or values on the x and y axis, so it is very hard to understand and validate the plot. Figures 1 and 2 are also difficult to interpret, despite the extensive description. I believe the notation may be a bit dense and that the paper would benefit from also describing the results in a more abstract way. I also think the setting is confusing: the MDP is introduced as infinite-horizon, but there is reference to \"done\" states, which are not described. Additionally, if the MDP is infinite-horizon, I don't understand very well what is meant by later states, or states in later time steps, and remaining time-steps.\n\nMinor:\n- Figure 2 appears on page 5, while only being referred in page 7, whereas Figure 3 appears on page 6 while being referred on page 5 (so I believe the figures should be switched, and appear on top of the page they are referred to). \n- There is a reference to a p-value of 0.05, when I believe it is meant a significance level of 0.05.\n- I also don't think it is clear for a general reader what is Spinning Up (page 8).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper focuses on distributional reinforcement learning, and proposes a method that assumes normality of the distribution of returns in continuous environments. Leveraging the assumption, the method is shown to outperform vanilla and distributional actor-critic algorithms on several continuous environments.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "I think the paper needs to improve significantly in terms of clarity and motivation. At the moment, neither the setting nor the objective of the paper seem clear to me, and it is somehow hard to follow and evaluate the work.", "broader_impact_concerns": "I don't predict special ethical implications of the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: As mentioned above, after addressing reviewer feedback, the paper clearly presents a simple, yet novel, approach to distributional RL that the RL community can leverage for further advancements. The majority of reviewers support acceptance, with which I agree.
Decision: Accept (poster)