Year: 2021

Abstract:
Many exploration strategies are built upon the optimism in the face of the uncertainty (OFU) principle for reinforcement learning. However, without considering the aleatoric uncertainty, existing methods may over-explore the state-action pairs with large randomness and hence are non-robust. In this paper, we explicitly capture the aleatoric uncertainty from a distributional perspective and propose an information-theoretic exploration method named Optimistic Value Distribution Explorer (OVD-Explorer). OVD-Explorer follows the OFU principle, but more importantly, it avoids exploring the areas with high aleatoric uncertainty through maximizing the mutual information between policy and the upper bounds of policy’s returns. Furthermore, to make OVD-Explorer tractable for continuous RL, we derive a closed form solution, and integrate it with SAC, which, to our knowledge, for the first time alleviates the negative impact on exploration caused by aleatoric uncertainty for continuous RL. Empirical evaluations on the commonly used Mujoco benchmark and a novel GridChaos task demonstrate that OVDExplorer can alleviate over-exploration and outperform state-of-the-art methods.

heading: 1 INTRODUCTION
text: The exploration and exploitation trade-off is critical in reinforcement learning (Thompson, 1933; Sutton & Barto, 2018). Many exploration strategies have been proposed in the literature (Osband et al., 2016; Lillicrap et al., 2016; Martin et al., 2017; Pathak et al., 2017; Ciosek et al., 2019). Among these strategies, those following the Optimism in the Face of Uncertainty (OFU) principle (Auer et al., 2002) provide efficient guidance for exploration (Chen et al., 2017; Ciosek et al., 2019). Generally, OFU-based methods regard the uncertainty as the result of insufficient exploration for the state-action pair, and refer to such uncertainty as epistemic uncertainty. Besides, there is another uncertainty called aleatoric uncertainty, which captures environmental stochasticity:
• Epistemic uncertainty (a.k.a. parametric uncertainty) represents the ambiguity of the model arisen from insufficient knowledge, and is high at those state-action pairs seldom visited (Dearden et al., 1998; Osband et al., 2016; Moerland et al., 2017).
• Aleatoric uncertainty (a.k.a. intrinsic uncertainty) is the variation arisen from environment randomness, caused by the stochasticity of policy, reward and/or transition probability, and is characterized by return distribution. (Bellemare et al., 2017; Moerland et al., 2017).
In the heteroscedastic stochastic tasks where the environment randomness of different state-action pairs differs, those OFU-based methods may be inefficient without properly tackling aleatoric uncertainty (Nikolov et al., 2019). As shown in Figure 1, if the epistemic uncertainty estimation is disturbed due to the volatility caused by aleatoric uncertainty, the exploration strategy (i.e., policy A) optimistic about such estimated uncertainty may lead to explore areas with high aleatoric uncertainty. Visiting such areas results in unstable and risky transitions. In the real world, the aleatoric uncertainty can be caused easily, for example, unpredictable wind can shift the trajectory of an robot’s action. If such aleatoric uncertainty is not modelled, the RL agent may be trapped because the state transitions in such area can be wrongly considered novel and worth exploring due to the high uncertainty. This issue, to explore overly the state-action pairs visited frequently but with high
aleatoric uncertainty, is referred to as the over-exploration issue. Intuitively, such issue could be solved by avoiding exploring optimistically about aleatoric uncertainty (i.e., policy B in Figure 1).
The similar concern has been raised in the discrete RL, where Nikolov et al. (2019) proposes to use the Information-Directed Sampling (IDS) to avoid the over-exploration issue in the environments with heteroscedastic noise. However, IDS needs to calculate information-regret ratio for each action, thus applying IDS for continuous RL with explosive or even infinite action space could be non-trivial and ineffective. Meanwhile, many advanced continuous RL approaches suffer from the over-exploration issue. Soft Actor-Critic (SAC) is a well performed continuous RL algorithm, but does not account for efficient exploration beyond maximizing policy entropy (Haarnoja et al., 2018). OAC improves the exploration follow-
ing OFU principle, but it ignores to avoid exploration towards higher aleatoric uncertainty (Ciosek et al., 2019). Therefore, an effective exploration method to address the over-exploration issue for continuous RL is of urgent need.
To address the over-exploration issue, we propose that the characterization of aleatoric uncertainty and a new exploration principle are necessary. The new principle need to properly trade-off between the epistemic and aleatoric uncertainties, so as to explore seldom visited state-action pairs while avoiding trapped in areas with high aleatoric uncertainty. In this paper, we propose a general information-theoretic exploration approach OVD-Explorer, which enriches the OFU exploration principle with a complement: “avoid areas with high aleatoric uncertainty”. Furthermore, to guide the exploration following the new principle, OVD-Explorer maximizes the mutual information between policy and corresponding upper bounds. Notable, the upper bounds represent the best return that policy can reach, formulated using Optimistic Value Distributions (OVD), and the value distribution characterises the aleatoric uncertainty. From the theoretical derivation, we show that maximizing such mutual information urges the OVD-Explorer to guide exploration towards the areas with higher epistemic uncertainty and alleviate over-exploration issue by avoiding the areas with lower information gain, i.e., the areas that have higher aleatoric uncertainty but low epistemic uncertainty.
To make OVD-Explorer tractable for continuous action space, we derive its closed form, and propose a scheme to incorporate OVD-Explorer with any policy-based RL algorithm. Practically, we demonstrate the exploration benefits based on SAC. Evaluations on many challenging continuous RL tasks, including a toy task GridChaos, tasks in Mujoco as well as their stochastic variants are conducted, and the results verify our analysis. To the best of our knowledge, OVD-Explorer firstly addresses the negative impact of aleatoric uncertainty for exploration in continuous RL.

heading: 2 PRELIMINARIES

heading: 2.1 DISTRIBUTIONAL RL
text: To model the randomness in the observed long-term return, which characterises aleatoric uncertainty, distributional RL methods (Bellemare et al., 2017; Dabney et al., 2018b;a) are used to estimate Q-value distribution rather than expected Q-value (Mnih et al., 2015). In our paper, we focus on quantile regression used in QR-DQN (Dabney et al., 2018b), where the distribution of Q-value is represented by the quantile random variable Z. Z maps the state-action pair to a uniform probability distribution supported on the values at all corresponding quantile fractions. Based on value distribution estimator parameterized as θ, we denote the value at quantile fraction τi as Ẑi(s, a, θ) given state-action pair (s, a).
Similar to the Bellman operator in the traditional Q-Learning (Watkins & Dayan, 1992), the distributional Bellman operator (Bellemare et al., 2017) T πD under policy π is given as:
T πDZ(st, at) D = R(st, at) + γZ(st+1, at+1), at+1 ∼ π(·|st+1). (1)
Notice that T πD operates on random variables, D = denotes that distributions on both sides have equal probability laws. Based on operator T πD , Dabney et al. (2018b) propose QR-DQN to train quantile estimations via the quantile regression loss (Koenker & Hallock, 2001), which is denoted as:
LQR(θ) = 1
N N∑ i=1 N∑ j=1 [ρτ̂i(δi,j)], (2)
where TD error δi,j = R(st, at) + γẐi(st+1, at+1; θ̄) − Ẑj(st, at; θ), the quantile Huber loss ρτ (u) = u ∗ (τ − 1u<0), and τ̂i means the quantile midpoints, which is defined as τ̂i = τi+1+τi2 .

heading: 2.2 DISTRIBUTIONAL SOFT ACTOR-CRITIC METHOD
text: Distributional Soft Actor-Critic (DSAC) (Ma et al., 2020) seamlessly integrates distributional RL with Soft Actor-Critic (SAC) (Haarnoja et al., 2018). Basically, based on the Equation 1, the distributional soft Bellman operator T πDS is defined considering the maximum entropy RL as follows:
T πDSZ(st, at) D = R(st, at) + γ[Z(st+1, at+1)− αlogπ(at+1|st+1)], (3)
where at+1 ∼ π(·|st+1), st+1 ∼ P(·|st, at). Then, the quantile regression loss differs from Equation 2 on δi,j , by extending clipped double Q-Learning (Fujimoto et al., 2018) based on the maximum entropy RL framework, to overcome overestimation of Q value estimation:
δki,j = R(st, at) + γ[ min k=1,2 Ẑi(st+1, at+1; θ̄k)− α log π(at+1|st+1; φ̄)]− Ẑj(st, at; θk), (4)
where θ̄ and φ̄ represents their target networks respectively. The actor is optimized by minimizing the following actor loss, the same as SAC,
Jπ(φ) = E st∼D, t∼N [ log π(f(st, t;φ)|st)−Q(st, f(st, t;φ); θ) ], (5)
where D is the replay buffer, f(s, ;φ) means sampling action with re-parameterized policy and is a noise vector sampled from any fixed distribution, like standard spherical Gaussian, and Q value is the minimum value of the expectation on certain distributions, as
Q(st, at; θ) = min k=1,2
Q(st, at; θk) = 1
N min k=1,2 N−1∑ i=0 Ẑi(st, at; θk). (6)

heading: 3 METHODOLOGY
text: We introduce a general exploration method, named OVD-Explorer, to achieve efficient and robust exploration for continuous RL. Overall, OVD-Explorer estimates the upper bounds of the policy’s return, and uses the bounds as criteria to find a behavior policy for online exploration from the information-theoretic perspective. Innovatively, to avoid over-exploration, OVD-Explorer particularly takes the aleatoric uncertainty into account, leveraging the optimistic value distribution (OVD) to formulate the upper bounds of policy’s return. Then, OVD-Explorer derives the exploration policy via maximizing the mutual information between policy and corresponding upper bounds for an optimistic exploration. Meanwhile, the derived behavior policy avoids being optimistic to aleatoric uncertainty, thus avoiding over-exploration issue.
In the following, we first give the intuition together with the theoretical derivation of OVD-Explorer. Then we practically describe how we formulate OVD based on uncertainty estimation. Lastly, analysis are given to further illustrate why OVD-Explorer can explore effectively and robustly.

heading: 3.1 OPTIMISTIC VALUE DISTRIBUTION GUIDED EXPLORATION
text: Most OFU-based algorithms follow the exploration principle: “select the action that leads to areas with high uncertainty”. However, this principle is incomplete as the high uncertainty may also be caused by the aleatoric uncertainty, which misleads the exploration direction. Hence, we complement the principle by introducing a constrain: “Not only select the action that leads to areas with
high uncertainty, but also avoid the ones that only lead to the area with high aleatoric uncertainty”. Notable, this new principle intuitively describes a good exploration ability, and we will illustrate how to quantitatively measure a policy’s exploration ability in the later section. By following this principle, OVD-Explorer derives the behavior policy for exploration.
Intuitively, each policy π in the policy space Π has different exploration ability (at state s), denoted by Fπ(s). Given current state s, OVD-Explorer aims at finding the exploration policy πE with the best exploration ability by solving the following optimization problem:
πE = arg max π∈Π
Fπ(s). (7)
We propose to quantitatively measure the policy’s exploration ability from an information-theoretic perspective by measuring the mutual-information of multi-variables as follows:
Fπ(s) = MI(Z̄π(s, a0), · · · , Z̄π(s, ak−1); π(·|s)|s), (8) where π(·|s) is the action random variable, and Z̄π(s, ai) the random variable describing the upper bounds of return that action ai can reach under policy π at state s. Here, ai ∈ A denotes any legal action, thus k could be infinite in continuous action space. The higher Fπ(s) is, the better ability that policy π has in exploring towards higher epistemic uncertainty while simultaneously avoiding higher aleatoric uncertainty, which satisfies the principle we raised. (see analysis in Section 3.3.)
Now we state the Theorem 1 to measure the mutual information above in continuous action space. Theorem 1. The mutual information in Equation 8 at state s can be approximated as:
Fπ(s) ≈ 1 C E a∼π(·|s)
z̄(s,a)∼Z̄π(s,a)
[ ΦZπ (z̄(s, a)) log ΦZπ (z̄(s, a))
C
] . (9)
Φx(·) is the cumulative distribution function (CDF) of random variable x, z̄(s, a) is the sampled upper bound of return from its distribution Z̄π(s, a) following policy π, Zπ describes the current return distribution of the policy π, and C is a constant (See proof in Appendix B.1).
Theorem 1 reveals that Fπ(s) is only proportional to ΦZπ (z̄(s, a)), by maximizing which the policy’s exploration ability can be improved. Concretely, given any policy πθ (parameterized by θ), the derivative ∇θΦZπθ (z̄(s, a)) can be measured, thus gradient ascent can be iteratively performed to derive a better exploration policy. (The complete closed form solution is given in later Section 4)
Note that, to perform above optimization procedure, we need to formulate two critical components in ΦZπ (z̄(s, a)): 1 the return of the policy Zπ(s, a) and 2 upper bound of the return of the policy Z̄π(s, a). Therefore, we first details the formulations in Section 3.2, and then analyze why maximizing ΦZπ (z̄(s, a)) can derive a better exploration in Section 3.3.

heading: 3.2 FORMULATING THE RETURN OF POLICY AND CORRESPONDING UPPER BOUND
text: Now, we introduce the formulation of the return of policy Zπ(s, a) and corresponding upper bound Z̄π(s, a). As mentioned before, most OFU-based methods neglect the aleatoric uncertainty when formulating the return and upper bound of return (Mavrin et al., 2019; Chen et al., 2017), resulting in the over-exploration issue. Therefore, OVD-Explorer particularly takes into account the aleatoric uncertainty, leveraging distributional RL paradigm to characterize aleatoric uncertainty (Bellemare et al., 2017; Dabney et al., 2018b;a). OVD-Explorer uses two value distribution estimators Ẑ(s, a; θ1) and Ẑ(s, a; θ2) parameterized by θ1 and θ2, as ensemble estimators to formulate Z̄π and Zπ in different ways. Unless stated otherwise, the (s, a) is omitted hereafter to ease notation.
Formulation of Z̄π . The Z̄π denotes the upper bounds of value that policy π can reach via different actions. We propose Gaussian distribution with optimistic mean value for formulation as follows, and accordingly refer to it as Optimistic Value Distribution (OVD):
Z̄π(s, a) ∼ N (µZ̄(s, a), σ2aleatoric(s, a)), (10) where µZ̄(s, a) and σ2aleatoric(s, a) is mean value and variance, respectively. Notable, Chen et al. (2017) discovers the optimisticity is beneficial for better estimating the upper bound, which motivates us to optimistically estimate the µZ̄(s, a) by considering epistemic uncertainty as follows:
µZ̄(s, a) = µ(s, a) + βσepistemic(s, a), s.t. { µ(s, a) = Ei∼U(1,N)Ek=1,2Ẑi(s, a; θk) σ2epistemic(s, a) = Ei∼U(1,N)vark=1,2Ẑi(s, a; θk) ,
(11)
where µ(s, a) represents the Q-value estimation, σepistemic(s, a) is the epistemic uncertainty, β is a hyper-parameter controlling the magnitude of epistemic uncertainty, U is uniform distribution, N is the number of quantiles, and Ẑi(s, a; θk) is the value of the i-th quantile drawn from Ẑ(s, a; θk).
Moreover, the aleatoric uncertainty should be considered as it will affect the policy’s performance, thus we propose to model it as the variance of Z̄π . In practice, the aleatoric uncertainty σaleatoric(s, a) can be captured by value distribution estimators (Clements et al., 2019) as follows:
σ2aleatoric(s, a) = vari∼U(1,N) [ Ek=1,2Ẑi(s, a; θk) ] . (12)
Leveraging optimistic value estimations (via epistemic uncertainty) together with explicitly modeling aleatoric uncertainty, the upper bound of the value Z̄π can be effectively formulated as an optimistic Gaussian distribution. Such an upper bound value estimation provides a useful guidance for OVD-Explorer to derive the behavior policy for online exploration.
Formulation ofZπ . Zπ estimates the value obtained following policy π. Inspired by TD3 (Fujimoto et al., 2018), to alleviate overestimation, we propose to formulate Zπ in a pessimistic way. In practice, Zπ can be measured in two ways. First, similar to formulating Z̄π in Equation 10, Zπ can also be formulated as Gaussian distribution as follows:
Zπ(s, a) ∼ N (µZπ (s, a), σ2aleatoric(s, a)), s.t. µZπ (s, a) = µ(s, a)− βσepistemic(s, a), (13)
where µ(s, a), σaleatoric(s, a) and σepistemic(s, a) are the same defined in Equation 11. Differently, σepistemic(s, a) is subtracted from µ(s, a), which reveals the pessimistic value estimation of Zπ .
Another way is to formulate Zπ as multivariate uniform distribution as follows:
Zπ(s, a) ∼ U{zπi (s, a; θ)}i=1,...,n, s.t. zπi (s, a; θ) = min k=1,2 Ẑi(s, a; θk), (14)
where each quantile value zπi (s, a; θ) equals to the minimum estimated value among ensemble estimators (i.e., zi(s, a; θk). As such, the estimated value of Zπ is relative pessimistic.
Adopting pessimistic value estimation avoids overestimation issue, thereby the value distribution Zπ can be more accurately formulated. Additional, Gaussian distribution helps more when the environment randomness follows a unimodal distribution, and multivariate uniform distribution can be more flexible and suitable for scenarios with multi-modal distributions. (see more in Section 5.3.)

heading: 3.3 ANALYSIS OF OVD-EXPLORER
text: This section analyzes how OVD-Explorer optimistically explores the informative areas and avoids the over-exploration issue. According to Theorem 1, the exploration policy OVD-Explorer derives can maximize Fπ(s), which is proportional to CDF value ΦZπ (z̄(s, a)). In the following, Figure 2(a) and 2(b) illustrate such CDF values for different actions (shaded area), and Table 1 shows how uncertainties affect the exploration.
In these cases, the value distributionZπ(s, a) is specified as a Gaussian distribution (in Equation 13), and the sampled optimistic value z̄(s, a) is specified as the mean of OVD µZ̄(s, a) (in Equation 11). Specifically, at state s, we assume that the means of Zπ at actions a1 and a2 are the same for ease of clarification.
Basically, Figure 2(a) shows OVD-Explorer can achieve more optimistic exploration. We assume the aleatoric uncertainty at a1 and a2 are the same, but epistemic uncertainty is higher at a1, causing µZ̄(s, a1) > µZ̄(s, a2). Thus the CDF value is larger at a1, which means the action with higher epistemic uncertainty is preferred.
More crucially, Figure 2(b) shows that OVDExplorer could guide to avoid the area with higher aleatoric uncertainty given the equal optimism. We assume that the epistemic uncertainty at a1 and a2 are the same, causing their optimistic value estimation to be equal to µZ̄(s, a), but aleatoric uncertainty is lower at a1, i.e., PDF curve of Zπ(s, a1) is ”thinner and taller”. Thus the CDF value is larger at a1, meaning that the action with lower aleatoric uncertainty is preferred.
In the limit of t→∞, the epistemic uncertainty tend to be 0, and the aleatoric uncertainty estimated in Equation 12 converges to represent the true environment randomness. This is comparable to the second case (Figure 2(b)), and OVD-Explorer tends to choose the one with lower aleatoric uncertainty, which is a great advantage of our method over other OFU methods.
To summarize, the exploration guided by OVD-Explorer trades off between two criteria: exploring the areas with higher epistemic uncertainty, to ensure exploring optimistically, and avoiding the areas with higher aleatoric uncertainty, to avoid over-exploration caused by high environment randomness.

heading: 4 OVD-EXPLORER FOR MODERN RL ALGORITHMS
text: For continuous action space, the argmax operator in Equation 7 is intractable. To address that, in this section, we derive the behavior policy πE in closed form and state the scheme to incorporate it with existing policy-based algorithms.
First, we denote the policy learned by any policy-based algorithm as the target policy πT . To avoid the gap between training data collected using the behavior policy πE and the target policy πT , we need to constrain the difference between πE and πT , thus we derive πE in the vicinity of πT .
Second, to derive exploration πE for modern RL algorithms with stochastic policy based on OVDExplorer, where both the exploration πE = N (µE ,ΣE) and target policy πT = N (µT ,ΣT ) are Gaussian distributions, we introduce the following proposition: Proposition 1. The OVD-Explorer behavior policy πE = N (µE ,ΣE) is as follows:
µE = µT + αEZ̄π [ m× ∂z̄(s, a)
∂a |a=µT
] , (15)
and ΣE = ΣT . (16) In specific, α is the step size controlling the exploration level and m = log ΦZπ(s,µT )(z̄(s,µT ))
C + 1 (see proof in Appendix B.2).
The expectation EZ̄π can be estimated by sampling K samples, then Equation 15 is simplifies as:
µE = µT + αm
K K∑ i=1 ∂z̄i(s, a) ∂a |a=µT . (17)
Algorithm 1 summarizes the overall procedure of OVD-Explorer, including the formulation of Z̄π and Zπ (Line 2 and line 3) and behavior policy generation (Line 4). The generated behavior policy can be integrated with any modern policy-based RL algorithms to render the stable and wellperformed algorithm. Please see Appendix C for the entire pseudo-code and details about how the OVD-Explorer-based behavior policy is incorporated with DSAC.

heading: 5 EXPERIMENTS
text: To reveal the consistency between our theoretical analysis and the performance of OVD-Explorer algorithm, we conduct experiments to address the following questions:
Algorithm 1 The behavior policy (i.e., exploration policy) derived from OVD-Explorer. Input: Current state st, current value distribution estimators θ1, θ2, current policy network φ. Output: Behavior policy πE .
1: Obtain target policy from policy-based RL algorithm πT (·|st;φ) ∼ N (µT (st;φ), σT (st;φ)) 2: Derive OVD Z̄π(st, µT (st;φ)) using Eq. 10. 3: Construct value distribution of behavior policy Zπ(st, µT (st;φ)) using Eq. 13 or 14. 4: Calculate the mean of behavior policy distribution µE using Eq. 17. 5: return πE ∼ N (µE , σT (st;φ))
RQ1 (Exploration): Can OVD-Explorer explore optimistically while avoiding over-exploration simultaneously? RQ2 (Performance): Can OVD-Explorer handle complex and even stochastic tasks? RQ3 (Sensitivity to α): Is OVD-Explorer sensitive to α that controls exploration magnitude?

heading: 5.1 EXPERIMENT SETUP
text: Baselines include SAC (Haarnoja et al., 2018), DSAC (Ma et al., 2020), and DOAC, which is the distributional variant of OAC (Ciosek et al., 2019). We implement Gaussian and quantile formulations of Zπ as in Equation 13 and 14, which are denoted in the following as OVDE G and OVDE Q, respectively. We test OVD-Explorer on a novel task GridChaos and several tasks in Mujoco (Todorov et al., 2012) including the stochastic variants.
The appendix gives more details, including environment settings, implementation details, hyperparameters settings, and computing infrastructure used, as well as more experiment evaluations, including tasks with different noise scales, tasks with different episode horizon, the sensitivity to β and ablation study on the pessimistic formulation of Zπ(s, a).

heading: 5.2 EXPLORATION IN GRIDCHAOS (RQ1)
text: To illustrate the exploration pattern of OVD-Explorer and show the advantage of OVD-Explorer over DSAC and DOAC, we evaluate OVD-Explorer on a novel continuous and stochastic control task called GridChaos.
Figure 3(a) shows the map of GridChaos, in which we control the cyan triangle (agent) aiming to reach the fixed dark blue goal. The state is the current coordinate, and the action is a two-dimensional vector including the movement angle and distance. One episode terminates when the agent reaches the goal or maximum steps (i.e., MAX STEP, typically 100). Also, it receives a +100 reward when reaching the goal otherwise 0. The reason why it is chaos is that the randomness of the transition is heterogeneous in different parts of this environment. Table 2 shows the environment settings. Moreover, Figure 3(b) shows that OVDExplorer can reach the goal faster, with more efficient exploration.
Figure 3(c) shows the values of uncertainty estimation and exploration objective (mutual information) taken four different actions at the state shown in Figure 3(a) and at training epoch 1249. Basically, Figure 3(c) illustrates that the estimated aleatoric uncertainty of left is higher than that of right, indicating that OVD-Explorer models aleatoric uncertainty properly. Further, OVD-explorer encourages to explore the right side at that time, since the value of exploration objective (in green) is highest. It implies that OVD-explorer tends to explore areas with higher epistemic uncertainty and avoiding higher aleatoric uncertainty, which is in accordance with our exploration principle. On the other hand, if high epistemic uncertainty is considered only like in OAC(Ciosek et al., 2019), the
agent would be guided to the left. Then the agent may be trapped in the left side due to the high aleatoric uncertainty, and it is the possible reason about why OAC fails tackling such heteroscedastic stochastic task. In appendix, we show more analysis about the exploration pattern, please refer to Appendix E.2 and Appendix E.10.
Besides, we conduct the evaluation when the randomness is high around the goal, as well as many other noise settings, the results enhance the ability of OVD-Explorer. More experiments that evaluate the performance on several other noise settings, as well as more detailed value histogram on different epoch in training process can be found in Appendix.

heading: 5.3 PERFORMANCE ON MUJOCO TASKS (RQ2)
text: To demonstrate the performance of OVD-Explorer more generally, we evaluate it on several Mujoco tasks. For 5 standard Mujoco tasks 1, the transition is deterministic and the randomness is only from the stochastic policy. For 5 noisy tasks, the heteroscedastic Gaussian noise is added in Mujoco tasks. Specifically, in each state transition on noisy tasks, Gaussian noise of different scales is randomly injected following a certain probability. Overall, we have the following findings.
Basically, OVD-Explorer can perform stably in the standard tasks with slight or little randomness, guiding efficient exploration. From the results2 in standard Mujoco tasks in table 3, it is shown that in the relatively easy tasks, i.e., Hopper-v2, Reacher-v2, and HalCheetah-v2, OVD-Explorer does not obtain much gain beyond the baselines, which seems that these tasks are not profoundly demanding for exploration. In the high-dimension tasks, i.e., Ant-v2 as shown in Figure 4(a), OVD-Explorer significantly outperforms DSAC and DOAC. It means that the aleatoric uncertainty caused by policy indeed degrades the performance of DOAC, where the aleatoric and epistemic uncertainties are not distinguished.
Besides, OVD-Explorer can avoid the impact of heteroscedastic aleatoric uncertainty on exploration and thus improve robustness. As the results of noisy Mujoco tasks shown in Table 3, DOAC is worse than DSAC in most cases, which means that heteroscedastic aleatoric uncertainty causes significant degrades of DOAC. Simultaneously, OVD-Explorer significantly outperforms DOAC and DSAC, especially in Noisy Ant-v2 as shown in Figure 4(b).
Furthermore, OVDE G takes the Gaussian prior and concretely obtains better estimation of critic in the noisy tasks, while OVDE Q performs better in the standard tasks due to the flexibility of quantile distribution. Generally, the difference between OVDE G and OVDE Q is that they use different formulations of Zπ(s, a). Theoretically, the value function distribution in OVDE Q is more flexible and should perform better than OVDE G, as the reported results on five standard Mujoco tasks show. On the other hand, OVDE G performs better in the noisy tasks, and it is because that the transition probability is Gaussian in the stochastic Mujoco tasks.
1https://github.com/openai/gym/tree/master/gym/envs/mujoco 2Here, the reported results may be slightly different from previously reported results, partly due to the statistic approach, and partly due to the implementation. Nevertheless, the patterns of these baselines (e.g., DSAC outperforms SAC in the vast majority of cases) are consistent with previous results. The details about the implementation of baselines are described in the Appendix D.2.

heading: TASK EPOCH SAC DSAC DOAC OVD-EXPLORER G OVD-EXPLORER Q
text: 5.4 SENSITIVITY TO α (RQ3)
The step size α in Equation 15 controls how much the behavior policy derived from OVD-Explorer, i.e. πE , is far away from the target policy πT , which can be essential for exploration benefit. To investigate the appropriate range of it, we test several α value on Noisy Ant-v2 task using OVDE G, and the result is shown in Figure 4(c). If α is quite small, OVD-Explorer degenerates to DSAC and implies little exploration. In contrast, if α is larger, the performance becomes worse because of the huge gap between behavior policy and target policy. The result demonstrates that α should be taken in a suitable range to facilitate more adequate exploration. In our experiments, we uniformly use α to be equal to 0.05 to show the performance of the OVD-Explorer. In addition, we find that a smaller α leads to higher gains when the task is much more difficult, and the details can be found in Appendix E.6. We also verify the sensitivity of OVD-Explorer to β in Appendix E.7 and found that there is a broad range of settings for β, which can lead to well performance.

heading: 6 CONCLUSION
text: This paper proposes an information-theoretic exploration method OVD-Explorer, which introduces a novel measurement of exploration ability, i.e., the mutual information between the policy and the upper bounds of return. By maximizing the mutual information, OVD-Explorer is able to derive the behavior policy, that follows the OFU principle, and further avoids exploring the areas with high aleatoric uncertainty. Integrated with SAC, OVD-Explorer addresses the negative impact of aleatoric uncertainty for exploration in continuous RL for the first time.

heading: A RELATED WORK
text: In this work, we consider the exploration strategy under the principle of Optimism in the Face of Uncertainty (OFU) (Auer et al., 2002), especially in the heteroscedastic stochastic environment. We aim to improve exploration efficiency, and alleviate the over exploration issue caused by aleatoric uncertainty.
Basic exploration strategies, like -greedy (Sutton & Barto, 2018), noise perturbation (Lillicrap et al., 2016), entropy regularization (Mnih et al., 2016) and stochastic policy (Haarnoja et al., 2018), lead to undirected exploration through random perturbations. With the increasing emphasis on exploration efficiency in RL, various exploration methods have been developed. One kind of methods uses intrinsic motivation to stimulate agent to explore, such as count-based novelty (Martin et al., 2017; Ostrovski et al., 2017; Bellemare et al., 2016; Tang et al., 2017; Fox et al., 2018), prediction error (Pathak et al., 2017), reachability (Savinov et al., 2019) and information gain on environment dynamics (Houthooft et al., 2016). Some recent methods, originating from tracking uncertainty, guide efficient exploration under the principle of OFU, such as Thompson Sampling (Thompson, 1933; Osband et al., 2016), IDS (Nikolov et al., 2019; Clements et al., 2019) and other customized methods (Moerland et al., 2017; Pathak et al., 2019).
The base of OFU methods is to model epistemic and aleatoric uncertainties in RL. Bootstrapped DQN (Osband et al., 2016) has become the well-used approach for capturing epistemic uncertainty (Kirschner & Krause, 2018; Ciosek et al., 2019), and distributional RL methods (Bellemare et al., 2017; Zhou et al., 2020; Dabney et al., 2018a;b) are used for capturing aleatoric uncertainty. However, most traditional OFU methods do not distinguish the two types of uncertainty, which can easily lead the naive solution to favor actions with higher variances in stochastic tasks, i.e., over-exploration issue.
To address that, Mavrin et al. (2019) study how to take advantage of value distribution for efficient exploration under both types of uncertainty, proposing Decaying Left Truncated Variance (DLTV) based on QR-DQN. Besides, Nikolov et al. (2019) and Clements et al. (2019) propose to use Information Direct Sampling (Kirschner & Krause, 2018) for efficient exploration in RL, which formulates epistemic and heteroscedastic aleatoric uncertainty and maximizes information gain on globally optimal action to explore informative state-action pairs. However, such methods are complicated when deriving a behavior policy and is limited to discrete control.
Meanwhile, there is not any strategy that can help the well-performed continuous RL algorithms (Haarnoja et al., 2018; Ciosek et al., 2019; Ma et al., 2020) to address aleatoric uncertainty when exploration. OAC (Ciosek et al., 2019) proposes exploration bonus guided by the upper bound of Q estimation to facilitate exploration based on Soft Actor-Critic (SAC) (Haarnoja et al., 2018). Nevertheless, OAC ignores the potential impact of the aleatoric uncertainty, which may cause misleading exploration. Our proposed OVD-Explorer is a novel exploration strategy, which can guide agent to explore towards higher epistemic uncertainty, and also avoid the areas with high aleatoric uncertainty, improving the robustness of exploration especially facing heteroscedastic aleatoric uncertainty.
To capture aleatoric uncertainty, OVD-Explorer models the value distribution and uses mutual information to guide exploration following the principle of OFU, measuring the correlations between the policy distribution and upper bounds distribution of return. There are some other informationtheoretic exploration strategies using mutual information, such as VIME (Houthooft et al., 2016), which measures the information gain on environment dynamics, and EMI (Kim et al., 2019), which generates intrinsic reward using prediction error of representation learned by mutual information. Those methods can solve sparse reward problem very well by using intrinsic reward. Nevertheless, those exploration methods use mutual information neither on the value distribution, nor for OFUbased exploration. Besides, unlike the mechanisms used in measuring mutual information, such as variational inference (Hinton & van Camp, 1993; Houthooft et al., 2016) and f-divergence (Nowozin et al., 2016; Kim et al., 2019), we find the correlation between policy and upper bounds of return through uncertainty as shown in Theorem 1, thus we can directly derive the close form exploration policy.

heading: B PROOFS

heading: B.1 PROOF OF THEOREM 1
text: In order to prove the Theorem 1, we first propose the following lemma about Fπ(s).
Lemma 1. The mutual information of Z̄π(s, a0), · · · , Z̄π(s, ak−1) and π(·|s) at state s is: Fπ(·|s) = ∫ a∼π(·|s) E z̄(s,a)∼Z̄π(s,a) [ p(a|z̄(s, a), s) log p(a|z̄(s, a), s) π(a|s) ] da, (18)
where p(a|z̄(s, a), s) represents the posterior probability distribution of policy given current state s and the sampled upper bound of return z̄(s, a).
Proof. For simplicity, we assume that the size of action space is k = 2, and the actions are denoted as a0 and a1, a0 6= a1. Then we derive the mutual information among three random variables MI(Z̄π(s, a0), Z̄
π(s, a1), π(·|s)), where the action sampled from π(·|s) is either a0 or a1. Considering the formula for the mutual information, Fπ(s) is derived as follows:
Fπ(·|s) = MI(Z̄π(s, a0), Z̄π(s, a1);π(·|s)|s) = ∑
a∼π(·|s) z̄(s,a0)∼Z̄π(s,a0) z̄(s,a1)∼Z̄π(s,a1)
[ p(a, z̄(s, a0), z̄(s, a1)) log p(a, z̄(s, a0), z̄(s, a1))
π(a|s)p(z̄(s, a0), z̄(s, a1))
]
= ∑
a∼π(·|s) z̄(s,a0)∼Z̄π(s,a0) z̄(s,a1)∼Z̄π(s,a1)
[ p(a|z̄(s, a0), z̄(s, a1))p(z̄(s, a0), z̄(s, a1)) log
p(a|z̄(s, a0), z̄(s, a1)) π(a|s)
]
= ∑
a∼π(·|s) E
z̄(s,a0)∼Z̄π(s,a0) z̄(s,a1)∼Z̄π(s,a1)
[ p(a|z̄(s, a0), z̄(s, a1)) log
p(a|z̄(s, a0), z̄(s, a1)) π(a|s)
] ,
where the posterior distribution p(a|z̄(s, a0), z̄(s, a1)) is the probability of choosing action a on the condition of the samples from upper bounds of action a0 and a1.
Considering that in the decision-making process, the probability of action a0 is independent to the upper bound of other actions, such as z̄(s, a1), which means that p(a0|z̄(s, a0), z̄(s, a1)) = p(a0|z̄(s, a0)). Therefore, the above equation can be further reduced as follows.
Fπ(·|s) = E z̄(s,a0)∼Z̄π(s,a0)
[ p(a0|z̄(s, a0), s) log
p(a0|z̄(s, a0), s) π(a0|s) ] + E z̄(s,a1)∼Z̄π(s,a1) [ p(a1|z̄(s, a1), s) log p(a1|z̄(s, a1), s) π(a1|s) ] .
It is easy to extend to infinite-action case. Based on that for any action ai ∈ [a0, ak−1] and k →∞, the conditional probability p(ai|z̄(s, a0), z̄(s, a1), ..., z̄(s, ak−1)) = p(ai|z̄(s, ai)), Fπ(·|s) can be simplified as following,
Fπ(·|s) = ∫ a∼π(·|s) E z̄(s,a)∼Z̄π(s,a) [ p(a|z̄(s, a), s) log p(a|z̄(s, a), s) π(a|s) ] da. (19)
Lemma 1 tells that the mutual information Fπ(st) is in direct proportion to p(a|z̄(s, a), s), which measures how much it is worth acting under the current policy π(a|s) when the upper bound is known.
Next, to measure the posterior probability p(a|z̄(s, a), s), we use a general and practically effective approach (Wang & Jegelka, 2017; Belakaria et al., 2020; Perrone et al., 2019; Li et al., 2020) of approximating the posterior probability given upper bound value.
Specifically, we approximate p(a|z̄(s, a), s) using the prior that zπ(s, a) ≤ z̄(s, a) with given policy π(s, a), since z̄(s, a) is the upper bound of zπ(s, a). Hence, we use the indicator function 1zπ(s,a)≤z̄(s,a) to truncate the policy π(s, a), and utilize the constant C to normalize the probability, as is shown in the following equation.
p(a|z̄(s, a), s) ≈ 1 C π(a|s)Ezπ(s,a)∼Zπ(s,a)
[ 1zπ(s,a)≤z̄(s,a) ] .
Here, Ezπ(s,a)∼Zπ(s,a) [ 1zπ(s,a)≤z̄(s,a) ] = ΦZπ(s,a)(z̄(s, a)), where Φx is the cumulative distribution function (CDF) of x, Z̄π and Zπ are the random variables, whose distributions describe the randomness of the returns, and z̄(s, a) is the value of random variable Z̄π .
Therefore, the posterior probability can be measured as follows,
p(a|z̄(s, a), s) ≈ 1 C π(a|s)ΦZπ (z̄(s, a)). (20)
In our method, we do not use the commonly used mechanisms about mutual information such as neural network estimation (Belghazi et al., 2018) and upper bound estimation (Cheng et al., 2020). Instead, we can find the correlation between random variables as shown in Equation 20, which helps to derive mutual information directly.
According to Lemma 1 and Equation 20, we can give the proof of Theorem 1 in the following.
Proof. By Combining Lemma 1 and Equation 20, Fπ(s) can be further derived as follows.
Fπ(s) = ∫ a∼π(·|s) E z̄(s,a)∼Z̄π(s,a) [ p(a|z̄(s, a), s) log p(a|z̄(s, a), s) π(a|s) ] da
≈ ∫ a∼π(·|s) E z̄(s,a)∼Z̄π(s,a) [ 1 C π(a|s)ΦZπ (z̄(s, a)) log π(a|s)ΦZπ (z̄(s, a)) Cπ(a|s) ] da
= ∫ a∼π(·|s) E z̄(s,a)∼Z̄π(s,a) [ 1 C π(a|s)ΦZπ (z̄(s, a)) log ΦZπ (z̄(s, a)) C ] da
= 1
C E
a∼π(·|s) z̄(s,a)∼Z̄π(s,a)
[ ΦZπ (z̄(s, a)) log ΦZπ (z̄(s, a))
C
]
Here, the last equality follows from Theorem 1.

heading: B.2 PROOF OF PROPOSITION 1
text: Proof. Similar to (Ciosek et al., 2019), we set the covariance matrix of behavior policy πE is that of target policy πT , i.e., ΣE = ΣT . Hence, the OVD-Explorer problem is simplified as:
µE = arg max µ F̂(s)
= arg max µ , Ē Zπ
[ ΦZπ (z̄(s, µ)) log ΦZπ (z̄(s, µ))
C
] (21)
To ensure that the behavior policy samples actions around the target policy, we derive the πE upon mean µT of target policy πT . In specific, we firstly obtain the gradient of F̂(s, µ) at πT , which is given as follows:
∇aF̂π(s, µ)|µ=µT = EZ̄π [ m̂× ∂z̄(s, a)
∂a |a=µT
] (22)
Algorithm 2 OVD-Explorer for DSAC 1: Initialise: Value networks θ1, θ2, policy network φ and their target networks θ̄1, θ̄2, φ̄, quantiles
number N, target smoothing coefficient (τ ), discount (γ), an empty replay pool D 2: for each iteration do 3: for each environmental step do 4: at ∼ πE(at, st) according to Algorithm 1 5: D ← D ∪ {(st, at, r(st, at), st+1)} 6: end for 7: for each training step do 8: for i = 1 to N do 9: for j = 1 to N do
10: calculate δki,j , k = 1, 2, following Eq. 4 11: end for 12: end for 13: Calculate LQR(θk), k = 1, 2 using δki,j following Eq. 2 14: Update θk with∇LQR(θk) 15: Calculate Jπ(φ), following Eq. 5 16: Update φ with∇Jπ(φ) 17: end for 18: Update target value network with θ̄k ← τθk + (1− τ)θ̄k, k = 1, 2 19: Update target policy network with φ̄← τφ+ (1− τ)φ̄ 20: end for
where m̂ is given as:
m̂ = φZπ(s,µT )(z̄(s, µT ))(log ΦZπ(s,µT )(z̄(s, µT ))
C + 1), (23)
and φ(x) is the probability distribution function (pdf). Hence, the µE is given as follows: µE = µT + αEZ̄π [ m× ∂z̄(s, a)
∂a |a=µT
] , (24)
where α is the step size controlling exploration level and m = log ΦZπ(s,µT )(z̄(s,µT ))
C + 1.

heading: C ALGORITHM 2: OVD-EXPLORER FOR DSAC
text: In this section, we show the whole algorithm of our implementation of OVD-Explorer based on DSAC in Algorithm 2. All the code can be found in the supplementary material.

heading: D MORE DETAILS ABOUT THE EXPERIMENTS
text: D.1 GRIDCHAOS
GridChaos is an environment built on OpenAI’s Gym toolkit, whose map is shown as in Figure 3(a) and Figure 5. In this section we illustrate more details in addition to Section 5.2.
The movable cyan triangle and the fixed symmetric dark blue goal are two parts split from square, and the goal is to make the triangle embedded in the goal to recover the original square, which is to say that it is an isosceles triangle whose base side is equal to the height. The triangle is always initialised randomly in the cyan rectangle, and the black line in the map represents the wall, where the triangle will be
adsorbed once hits the wall. The state transition is stochastic, and we add Gaussian noise to the action resulting in Gaussian transition probability.
To represent the location of the triangle, we establish a Cartesian coordinate system using the centroid of the map as the origin, as shown in Figure 5. Then the coordinates of the triangle are represented by the midpoint of the altitude of the triangle which is shown as the red point in the triangle. In the case shown in Figure 5, the initial coordinate of the triangle (agent) is in the negative half of the x-axis and the task target is in the first quadrant.

heading: D.2 BASELINES
text: Ma et al. (2020) shows the performance of DSAC and TD4, which is the distributional extension of TD3 (Fujimoto et al., 2018), and can also be used to capture epistemic and aleatoric uncertainty. Moreover, DSAC outperforms TD4 on Mujoco tasks as shown in Ma et al. (2020), so we evaluate only on SAC and DSAC, and further implement OVD-Explorer based on DSAC to develop the exploration ability.
SAC. The SAC (Haarnoja et al., 2018) implementation is mainly based on OAC repository, and the results in Ant-v2 and Hopper-v2 are similar to reported results by OAC. Our SAC report a better result than OAC’s implementation for SAC on HalfCheetah-v2, which is because the high variance of this environment as explained as in OAC.
DSAC. The DSAC (Ma et al., 2020) implementation is based on our implementation of SAC, except that the distributional Q function in the DSAC repository is used instead of the traditional Q function in SAC. As it is based on SAC, we set the hyper-parameters of DSAC to be consistent with SAC to ensure the fair comparison, which also results in the different reported results from original paper of DSAC. In our results, DSAC can guarantee an absolute advantage over SAC in most cases, which is consistent with the previous conclusion.
DOAC. The DOAC implementation is mainly based on our implementation of DSAC as well as the open source code of OAC. As DSAC shows great advantage due to the distributional value estimation, to ensure a fair comparison, we extend OAC (Ciosek et al., 2019) to its distributional version, i.e., DOAC, by replacing the exploration process of DSAC by the behavior policy derived by OAC. We set the hyper-parameters the same as used by OAC in Mujoco3, and our results of DOAC on Ant-v2 and HalfCheetah-v2 are significantly better than that OAC reported.
3That is given by the open source code, where βUB is 4.66 and δ is 23.53
D.3 IMPLEMENTATION
Our implementation of OVD-Explorer is based on the open source code of OAC 4, also refer to the code of DSAC 5 as well as softlearning 6. All experiments are performed on NVIDIA GeForce RTX 2080 Ti 11GB graphics card.
The training process of OVD-Explorer and DOAC are the same as in DSAC, except for the different behavior policy used, while OVD-Explorer and DOAC enrich the experience replay with the data using the the derived exploration policies, respectively. To ensure the fair comparison, the hyperparameters for training process of baselines and OVD-Explorer are the same. Besides, we have three hyper-parameters associated with OVD-Explorer as mentioned before, including α that controls the exploration level, β that determines the magnitude of uncertainty we use, as well as C that is the normalization factor. The hyper-parameters in our experiments are shown in Table 4.

heading: E MORE EXPERIMENT STUDY
text: E.1 RUNTIME ANALYSIS
Figure 6 shows the time consumption of algorithms relative to SAC. As can be seen, the distributional value estimation used in DSAC, DOAC and our methods introduces extra time consumption distinctly. Nevertheless, the relative time consumption of OVDE G and OVDE Q to SAC is 1.21 and 1.17, respectively, which means that OVD-Explorer spends about 20% more time than SAC to achieve up to nearly 100% performance gain as shown in Figure 6(b). This demonstrates the extra time consumption is well worth it. Besides, the time consumption of OVDE Q is close to that of DSAC, only with a larger variance, which indicates that the additional time consumption of OVDE Q is minimal while performing better exploration.

heading: E.2 ANALYSIS ABOUT OVD-EXPLORER’S ADVANTAGE IN THE CASE OF GRIDCHAOS
text: With the heatmap of the visiting frequency of agent during exploration, and the heatmap about the uncertainty estimation, we can visually analyze the patterns and advantages of OVD-Explorer.
4https://github.com/microsoft/oac-explore 5https://github.com/xtma/dsac 6https://github.com/rail-berkeley/softlearning
The distinctly different exploration patterns can be easily found. Figure 7(a), 7(b) and 7(c) present the state visiting frequency of OVD-Explorer, DSAC and DOAC, respectively. We can see that OVD-Explorer explores directly to the right half, where the environmental randomness is lower, whereas DSAC and DOAC are both stuck in the left half with higher environmental randomness.
Furthermore, we show how OVD-Explorer could explore directly without being trapped by the randomness through the estimated uncertainty. In specific, Figure 7(d) shows that the aleatoric uncertainty estimated by OVD-Explorer is consistent with environment settings, where the environment noise is higher on the left half. Figure 7(e) shows the ratio of estimated epistemic uncertainty and aleatoric uncertainty (i.e., epistemic-aleatoric ratio) of OVD-Explore, and higher ratio means higher epistemic uncertainty or lower aleatoric uncertainty, which is exactly the direction OVD-Explorer explores . The ratio is larger on the right half, which means that OVD-Explorer can avoid being stuck in the left half. Meanwhile, Figure 7(f) presents the estimated uncertainty in DOAC, which is larger on the left half. As DOAC encourages exploring area with relatively large estimated uncertainty, it explains why DOAC is stuck in the left half.

heading: E.3 EVALUATION ON SEVERAL OTHER NOISE SCALE IN GRIDCHAOS
text: OVD-Explorer can explore efficiently in heteroscedastic stochastic environment by considering differently about epistemic and aleatoric uncertainty for exploration as shown in Section 5.2. To further empirically prove its strength, we test OVD-Explorer in GridChaos with different noise scales in four quadrants, and the result is shown as Figure 8 and Table 5. We can find that OVD-Explorer can perform well in all those different noise injection of environment.
Concretely, from the results, the following observations deserve to be noticed. First, OVD-Explorer can significantly achieve better average return in all those settings, especially when the noise is set high, and can learn to reach the goal faster (see column about FRG). It shows the ability of OVDExplorer to guide agent explore against higher aleatoric uncertainty on the left side (the second
and third quadrants). Second, for the task without noise as shown in row E, which means the state transition is deterministic, OVD-Explorer still learns quickly. The results in row E show the inherently high difficulty of this task, not only because of the very sparse reward, but also the gate leading to the goal is set very small (the width of the gate is only 30% of the length of agent, i.e., the base of the isosceles triangle, which means that at the doorway the agent can only move a very small distance horizontally, otherwise it would be adsorbed to the wall and immobile).

heading: E.4 EVALUATION IN GRIDCHAOS WHEN THE ALEATORIC UNCERTAINTY IS HIGH AROUND GOAL
text: In the previous experiments in GridChaos, the noise (i.e., aleatoric uncertainty) near the goal is set lower. In such situation, OVD-Explorer, which follows the principle of OFU and further avoids exploring areas with higher aleatoric uncertainty, could bring significant advantage. Such setup of heterogeneous noise is reasonable, because in real life, the goal or optimal policy is always not expected to be highly stochastic.
Nevertheless, the evaluation about tasks with the existence of high randomness in the target region is valuable, so we conducted the following experiment in GridChaos, where the environment randomness in the right half (first and fourth quadrants), where the target is located, was set larger. The results are shown in Table 6. Note that we use OVDE(P) to denote the usual implementation that pessimistically estimates the value distribution (i.e., using Equation 13). Besides, OVDE(M) denotes the implementation that does not pessimistically estimate the value distribution (i.e., we modify the mean of Gaussian distribution Zπ in Equation 13 from the lower bound to expected value of the Q estimation µ(s, a) as in Equation 11.).
Our experimental findings are mainly the following three aspects.
Firstly, when facing extremely high aleatoric uncertainty around the goal (see row F), which causes the interaction around goal to be very unstable, chaotic and disorder, OVD-Explorer would strongly discourage exploring such a area, and thus performance would be damaged. In contrast, DSAC and DOAC have no restriction on aleatoric uncertainty, and high randomness may instead increase the probability of achieving the goal.
Second, in most cases (see G, H, I, J), OVD-Explorer always can guide better exploration and achieve better performance than DSAC and DOAC, especially when the noise is negligible (see row J). This reflects the fact that our exploration objective (the mutual information shown in Equation 8) makes great sense, achieving an appropriate trade-off between avoiding high aleatoric uncertainty and being optimistic about high epistemic uncertainty.
Third, an interesting finding is that OVD-Explorer may perform better by turning off the pessimistic estimation facing higher aleatoric uncertainty around the goal (see column OVDE(M)). This suggests that excessive pessimism is unnecessary if there is a need to explore areas with high aleatoric uncertainty.
Overall, from the results in Table 5 and Table 6, OVD-Explorer is able to tackle most of the cases quite well. When there is high randomness around the goal, OVD-Explorer has a shortcoming that it will inevitably slow down the efficiency of reaching the goal, because it limit the exploration towards such area. Fortunately, this shortcoming can be mitigated by turning off the pessimistic estimation.

heading: E.5 EVALUATION OF STATISTICAL SENSE

heading: TASK EPOCH SEED DSAC DOAC OVD-EXPLORER G OVD-EXPLORER Q
text: To counteract the randomness from a statistical perspective, we conduct all experiments for 5 trails with different seeds (typically 0-5), and report the average results with standard deviation. Next, to verify that the 5 trails are sufficient to mitigate the statistical randomness, we run other 5 runs (seeds are set as 5, 6, 7, 8, 9, respectively) for those algorithms on Ant-v2, and show the results in the following. Note that the first row of results is from our previously reported results, which is the same as Table 3, and the second row show the results new. The experimental results in Table 7 show that the results of 5 trials are sufficiently representative of the overall level, while the performance of OVD-Explorer undoubtedly stays ahead.

heading: E.6 STUDY ON EPISODIC HORIZON
text: Section 5.3 has shown great advantage of OVD-Explorer over DSAC and DOAC in stochastic Mujoco tasks, which limits the length for an episode to 100 steps. To further empirically verify the efficiency of OVD-Explorer, we test on Noisy Ant-v2 task with different maximum episodic length setup. Our results in Figure 9 show that OVD-Explorer can significantly perform better than baselines in different maximum episodic length (i.e., 250, 500, 750 and 1000). Noting that longer maximum episodic length renders higher difficulty of solving tasks, especially for the high-dimensional tasks demanding exploration. In specific, we have the following two conclusions.
Firstly, the exploration should be more conservative in the harder tasks, where we should set smaller α in OVD-Explorer. In Figure 9(a), (b) and (c), α is set to 0.05 by default, while we can find that the advantage of OVD-Explorer G decreases gradually with the increasing of the difficulty of tasks. Further, if α is set to 0.005, then there is a substantial performance improvement as shown in Figure 9(d). Also, as shown in Figure 9(e), both OVD-Explorer methods perform well when the task episodic horizon is 750 with α set to 0.005. On the hardest task we tested, i.e., the Noisy Ant-v2 with horizon 1000 as shown in Figure 9(f), OVD-Explorer gain remarkable performance, while α is set smaller as 0.001.
Secondly, OVD-Explorer Q is more stable than OVD-Explorer G, which is consistent with the conclusion in Section 5.3. We can find from Figure 9(a), (b) and (c) that OVD-Explorer Q performs stably better while OVD-Explorer G degrades. OVD-Explorer G is better in easier task with horizon 100, which is due to the Gaussian prior of noise. But when the task becomes harder, the prior helps less, and OVD-Explorer Q shows the advantage of more flexibly modeling aleatoric uncertainty and thus the performance is more stable.
E.7 STUDY ON HYPER-PARAMETERS β
OVD-Explorer is sensitive to α, as shown in Section 5.4. There is another hyper-parameter β, which controls the scale of uncertainty quantification as shown in Equation 11 and Equation 13, further having an impact on Z̄π and Zπ . To evaluate its sensitivity about β, we conduct the experiment on Noisy Ant-v2 task using OVD-Explorer G, and the result is shown in Figure 10. The results demonstrate that there is a broad range of settings for β, which can lead to well performance.

heading: E.8 ABLATION STUDY ON VALUE DISTRIBUTION ESTIMATION
text: As mentioned in Section 3.2, we estimate Zπ pessimistically to alleviate over-estimation. Also, as mentioned in Appendix E.4, the pessimism is unnecessary if there is a need to explore areas with high aleatoric uncertainty. In the following, to investigate the benefit of pessimistic estimation in general case, we compare the performance of OVD-Explorer to the modified versions that use normally estimated Zπ . Our results show that pessimistic estimation can mostly be better than that using normal estimation.
For OVDE G (mean), we modified the mean of Gaussian distribution Zπ in Equation 13 from the lower bound to average value of the Q estimation µ(s, a) as in Equation 11. For OVDE Q (mean), we modified the zπi (s, a) in Equation 14 from the minimum value to the average value, i.e., zπi (s, a; θ) = Ek=1,2Ẑi(s, a; θk).
As shown in Figure 11, both OVDE G (mean) and OVDE Q (mean) perform worse than the pessimistic version. To draw a conclusion, pessimistic estimate is indeed required in general cases. Only when there is a need to explore areas with high aleatoric uncertainty, is such pessimistic estimation worth being turned off.

heading: E.9 THE PERFORMANCE COMPARED WITH RND
text: RND also follows OFU principle, modeling uncertainty based on network distillation and using it as an intrinsic motivation signal to facilitate agent exploration. For the sake of fairness, we implement RND based on DSAC, denoted by DSAC+RND in the following, and evaluate it on 3 standard Mujoco tasks and 3 noisy Mujoco tasks. We show the results in the following.

heading: TASK EPOCH DSAC DSAC+RND OVD-EXPLORER G OVD-EXPLORER Q
text: For the complex task Ant-v2, RND brings much greater improvement by facilitating exploration based on the DSAC. For the other simpler tasks, RND does not bring significant performance improvement. This experiment demonstrates to some extent the effectiveness of RND exploration on several Mujoco tasks, but at the same time, our algorithm OVD-Explorer is still better.

heading: E.10 ANALYSIS ABOUT EXPLORATION PROCESS OF OVD-EXPLORER
text: In the following, we further verify from statistical analysis that OVD-Explorer is in compliance with our raised exploration principle, i.e., OVD-Explorer can achieve better trade-off between optimistic exploration and effectively avoiding exploring the areas with high aleatoric uncertainty. We show the values of uncertainty estimations and our exploration objective (mutual information) at different stages during the training processes of two trials with different noise settings in figures.
In Figure 12, the environment noise is set lower around the goal, noting that the darker background color in the map represents higher aleatoric uncertainty, and the red dot represents the coordinate of the current state. The performance under this trail is given and the agent hardly ever reaches the goal before the 1000th epoch. Therefore, in the early stage, the aleatoric uncertainty is inaccurate and remains very low, as the value distribution shows little divergence. The figure also shows that our exploration objective (in green) is high when the epistemic uncertainty is high. So before the 1000th epoch, the exploration is guided by epistemic uncertainty, which follows the OFU principle. Later, once the goal has been explored, the aleatoric uncertainty is properly modelled, i.e., the aleatoric uncertainty towards left is larger than right at current state (see epoch 1240). Then the mutual information value towards left is lower than right, although the epistemic towards left is higher. It
indicates that OVD-Explorer can property balance epistemic uncertainty and aleatoric uncertainty, and effectively avoid to explore the areas with higher aleatoric uncertainty.
In Figure 13, the environment noise is set higher around the goal. At the stage before the 1000th epoch, the exploration guidance is similar to Figure 12. The agent would hardly estimate the accurate aleatoric uncertainty without obtaining any reward. In the later stage, at the 1240th epoch, OVDExplorer suggests exploring to the right, even though it has been recognized that the environmental uncertainty on the right is high. This is because the epistemic unertainty dominates under the mutual information. In contrast, at the 1249th epoch, when the action towards right has been explored much, the significant higher aleatoric uncertainty towards right dominates. Therefore, following the mutual information, the action towards left is preferred. This demonstrates the trade off that OVD-Explorer make, which satisfies our raised exploration principle.
up down left right 0.0
0.5
1.0 Training epoch = 20
up down left right
Training epoch = 400
up down left right
Training epoch = 800
Type Epistemic uncertainty Aleatoric uncertainty Mutual information
up down left right 0.0
0.5
1.0 Training epoch = 1000
up down left right
Training epoch = 1240
up down left right
Training epoch = 1249
Type Epistemic uncertainty Aleatoric uncertainty Mutual information
up down left right 0.0
0.5
1.0 Training epoch = 20
up down left right
Training epoch = 400
up down left right
Training epoch = 800
Type Epistemic uncertainty Aleatoric uncertainty Mutual information
up down left right 0.0
0.5
1.0 Training epoch = 1000
up down left right
Training epoch = 1240
up down left right
Training epoch = 1249
Type Epistemic uncertainty Aleatoric uncertainty Mutual information

Review summaries:
The paper proposes the so-called Optimistic Value Distribution Explorer (OVD-Explorer) to deal with over-exploration issue in reinforcement learning (RL). It is claimed that the proposed algorithm is tractable for continuous action space and can avoid exploring  the areas with high aleatoric uncertainty. The paper tests the performances of OVD-Explorer and some of its competitors on some common environments.

This paper introduces OVD-Explorer, an efficient exploration algorithm that is able to detect and avoid areas with high aleatoric uncertainty. Actually, the proposed algorithm explores state-action pairs that have not been visited frequently (high epistemic uncertainty) and at the same time avoids areas with high aleatoric uncertainty. In order to achieve this behavior, OVD-Explorer maximises the mutual information between policy and corresponding upper bounds. Theoretical results show that the over-exploration issue can be tackled through the maximisation of such mutual information. Empirical analysis has been conducted on a toy task and at five MuJoCo environments including their stochastic variants. 

This paper proposes a new information-theoretic exploration approach, called OVD-Explorer, for deep reinforcement learning with continuous action space. The OVD-Explorer is capable of exploring epistemic uncertainty while avoiding aleatoric uncertainty. The paper also contains extensive experiments to evaluate the performance of OVD-Explorer.

The paper is concerned with the problem of over-exploration in RL. The authors propose to capture the aleatoric uncertainty during exploration and propose a new exploration method called Optimistic Value Distribution Explorer (OVD-Explorer). OVD-Explorer is designed to explore optimistically while avoiding the areas with high aleatoric uncertainty. The authors propose a new measure for policy's exploration ability which aims at maximizing the mutual information between policy and policy return upper bounds. OVD-Explorer achieves good empirical performance, outperforming other methods.