OVD-Explorer: A General Information-theoretic Exploration Approach for Reinforcement Learning
Authors: Jinyi Liu
Zhi Wang
YAN ZHENG
Jianye HAO
Junjie Ye
Chenjia Bai
Pengyi Li
Venue: ICLR 2022 Submitted (2022)
Keywords: Exploration
Uncertainty
Reinforcement Learning

Abstract: Many exploration strategies are built upon the optimism in the face of the uncertainty (OFU) principle for reinforcement learning. However, without considering the aleatoric uncertainty, existing methods may over-explore the state-action pairs with large randomness and hence are non-robust. In this paper, we explicitly capture the aleatoric uncertainty from a distributional perspective and propose an information-theoretic exploration method named Optimistic Value Distribution Explorer (OVD-Explorer). OVD-Explorer follows the OFU principle, but more importantly, it avoids exploring the areas with high aleatoric uncertainty through maximizing the mutual information between policy and the upper bounds of policy's returns. Furthermore, to make OVD-Explorer tractable for continuous RL, we derive a closed form solution, and integrate it with SAC, which, to our knowledge, for the first time alleviates the negative impact on exploration caused by aleatoric uncertainty for continuous RL. Empirical evaluations on the commonly used Mujoco benchmark and a novel GridChaos task demonstrate that OVD-Explorer can alleviate over-exploration and outperform state-of-the-art methods.

Under review as a conference paper at ICLR 2022
OVD-EXPLORER: A GENERAL INFORMATION- THEORETIC EXPLORATION APPROACH FOR REIN- FORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Many exploration strategies are built upon the optimism in the face of the uncer- tainty (OFU) principle for reinforcement learning. However, without consider- ing the aleatoric uncertainty, existing methods may over-explore the state-action pairs with large randommess and hence are non-robust. In this paper, we explic- itly capture the aleatoric uncertainty from distributional perspective and propose an cntemmation-thooetiie exploration method named Optimistic Value Distribution Explorer (OVD-Explloer). OVI--Explorer follows the OFU principle, but more importantly, it avoids exploring the areas with high aleatoric uncertainty through maximizing the mutual information between policy and the upper bounds pol- icy's returns. Furthermore, make OVD- DVD--xxlorer tractable for continuous RL, we derive a closed form solution, and integrate it with SAC, which, to our knowl- edge, for the first time alleviates the negative impact exploration caused by aleatoric uncertainty for continuous RI. Empirical evaluations on the commonly used Mujoco benchmark and novel GridChaos task demonstrate that OVD- Explorer can alleviate overeexploration and outperform state-of-the-ar methods.
INTRODUCTION
The exploration and exploitation trade-off is critical in reinforcement learning (Thompson 1933
Sutton & Barto 2018). Many exploration strategies have been proposed in the literature (Osand ct al. 2016; Lillicrap et al., 2016 Martin et al. 2017; Pathak et al.) 2017 Ciosek et al. 2019)
Among these strategies, those following the Optimism in the Face of Uncertainty (OFU) principle
(Auer et al. 2002) provide efficient guidance for exploration (Chen et al., 2017 Ciosek al. 2019).
Generally, OFU-based methods regard the uncertainty as the result of insufficient exploration for the state-action pair, and refer to such uncertainty as epistemic uncertainty. Besides, there is another uncertainty called aleatoric uncertainty, which captures environmental stochasticity:
Epistemic uncertainty (a.k.a. parametric uncertainty) represents the ambiguity of the model arisen from insufficient knowledge, and is high at those state-action pairs seldom visited (Dearden et al. 1998 Osband al. 2016 Moerland et al. 2017).
Aleatoric uncertainty (a.k.a. intrinsic uncertainty) is the variation arisen from environment randomness, caused by the stochasticity of policy, reward and/or transition probability, and is characterized by return distribution. (Bellemare al. 2017 Moerland al. 2017).
In the heteroscedastic stochastic tasks where the environment randomness of different state-action pairs differs, those OFU-based methods may be inefficient without properly tackling aleatoric un- certainty (Nikolov al.] (019). As shown in Figure [ if the epistemic uncertainty estimation disturbed due to the volatility caused by aleatoric uncertainty, the exploration strategy (i.e., policy A) optimistic about such estimated uncertainty may lead to explore areas with high aleatoric uncer- tainty. Visiting such areas results in unstable and risky transitions. In the real world, the aleatoric uncertainty can be caused easily, for example, unpredictable wind can shift the trajectory of an robot's action. If such aleatoric uncertainty is not modelled, the RL agent may be trapped because the state transitions in such area can be wrongly considered novel and worth exploring due to the high uncertainty. This issue, to explore overly the state-action pairs visited frequently but with high

---

Under review as a conference paper at ICLR 2022
aleatoric uncertainty, is referred to the over-exploration issue. Intuitively, such issue could be solved by avoiding exploring optimistically about aleatoric uncertainty (i.e., policy B in Figure
The similar concern has been raised in the discrete RL, where Nikolov et al. (2019) proposes to use the Information-Directee over-exploration issue in the environments with het- eroscedastic noise. However, IDS needs calculate information-regrel ratio for each action, thus apply- ing IDS for continuous RL with explosive or even in- finite action space could be non-trivial and ineffec- tive. Meanwhile, many advanced continuous RI. ap- proaches suffer from the over-exploration issue. Soft Actor-Critic (SAC) is well performed continuous RI. algorithm, but does not account for efficient explo- ration beyond maximizing policy entropy (Haarnoja eet al. 2018 OAC improves the exploration follow- ing OFU principle, but it ignores to avoid exploration towards higher aleatoric uncertainty (Ciosek al. 2019). Therefore, effective exploration method to address the over-exploration issue for continuous RL is of urgent need.
higher
Aleatoric uncertainty
lower
Sampling (IDS) to avoid the
Policy A
Palicy
agent
Figure I: An intuitive example. Red/green lines denote good/bad exploration policies.
To address the over-exploration issue, propose that the characterization of aleatoric uncertainty and a new exploration principle are necessary. The new principle need to properly trade-off be- to explore seldom visited state-action pairs
tween the epistemic and aleatoric uncertainties, while avoiding trapped in areas with high aleatoric uncertainty. In this paper, we propose a gen- exploration approach OVD-Explorer, which enriches the OFU exploration
eral information-theore principle with a complement: "avoid areas with high aleatoric uncertainty". Furthermore, to guide exploration following the new principle, OVD OVD-Explorer maximizes the mutual information be- tween policy and corresponding upper bounds. Notable, the upper bounds represent the best return that policy can reach, formulated using Optimistic Value Distributions (OVD), and the value distri- bution characterises the alcatoric uncertainty. From the thcoretical derivation, show that maxi- mizing such mutual information urges the OVD-Explorer to guide exploration towards the areas with higher epistemic uncertainty and alleviate wvereexpooaation issue by avoiding the areas with lower information gain, i.e., the areas that have higher aleatoric uncertainty but low epistemic uncertainty.
To make OVD-EExllorer tractable for continuous action space, we derive its closed form, and pro- pose scheme incorporate OVD-Exxllore with any policy-based RI. algorithm. Practically, demonstrate the exploration benefits based on SAC. Evaluations on many challenging continuous RL tasks, including a toy task GridChaos, tasks in Mujoco as well their stochastic variants are conducted, and the results verify our analysis. To the best of our knowledge, OVD- OVD-Explorer firstly addresses the negative impact of aleatoric uncertainty for exploration in continuous RL.
2
PRELIMINARIES
2.1 DISTRIBUTIONAL RL
To model the randomness in the observed long-term return, which characterises aleatoric uncer- tainty, distributional RL methods (Bellemare et al., 2017 Dabney et al. 2018b a) are used esti- mate Q-value distribution rather than expected Q-value (Miih et 2015). In our paper, we focus on quantile regression used in QR-DQN (Dabney et al.) 2018b), where the distribution of Q-value represented by the quantile random variable Z. Z maps the state-action pair to uniform probability distribution supported on the values at all corresponding quantile fractions. Based on value distri- bution estimator parameterized as 0, we denote the value at quantile fraction т; as 2,(8, a, A) given state-action pair (s. a).
Similar to the Bellman operator in the traditional Q-Learning /aakiins & Dayan butional Bellman operator (Bellemare al. 2017) To under policy is given
1992). the distri-
0441 7ss, at) 2 R(s,, a.) + (8 0141), a++1 ~ x(-
2

---

Under review as a conference paper at ICLR 2022
Notice that TB operates on random variables, denotes that distributions on both sides have equal probability laws. Based operator 73. Dabney et al. (2018b) propose QR-DQN to train quantile estimations via the quantile regression loss (Koenker Hallock 2001), which is denoted as:
N
N Ptt (5:g))
(QR
= N
(2)
1=
where TD error dzj = R(st, at) + ( 2z(8t ax; 0), the quantile Huber loss Pr (u) u* (T luca), and i; means the quantile midpoints, which is defined f; -:41
2.2 DISTRIBUTIONAL SOFT ACTOR-CRITIC METHOD
Distributional Soft Actor-Critic (DSACC (M et al. 2020 seamlessly integrates distributional RI. with Soft Actor-Critic (SAC) (Haarmoja al. 2018). Basically, based on the Equation [ the distri- butional soft Bellman operator Tös is defined considering the maximum entropy RL as follows:
(s D R(s,, az) + (5(+11, - (log
where a++1 (1+1), St+1 P(+)s;,a;) az). Then, the quantile regression loss differs from Equa- tion 2 on ótg, by extending clipped double Q-Learning (Fujimoto et al. 2018)) based on the maxi- mum entropy RL framework, to overcome overestimation of Q value estimation:
R(s,, ax) + 7 min (:65:+11444::8 a log (ac++1554+1
(4)
k=1.2
where 9 and ō represents their target networks respectively. The actor is optimized by minimizing the following actor loss, the same as SAC,
E (x) ScNDEEMN
[ogT(((se Q(st, f(s,, e: 0);
(5)
where D the replay buffer, f(s, e; ②) means sampling action with re- re-parameterized policy and is a noise vector sampled from any fixed distribution, like standard spherical Gaussian, and Q value is the minimum value of the expectation on certain distributions,
1 N 1=12
Q(5r, ac; 0) min Q(5r, de; 0x) k=1
min
Σ
(1 azi 0x)-
(6)
3 METHODOLOGY
We introduce general exploration method, named OVD-Explorer, to achieve efficient and robust exploration for continuous RL. Overall, OVD-Explorer estimates the upper bounds of the policy's s return, and uses the bounds criteria to find behavior policy for online exploration from the information-theoretie perspective. Innovatively, to avoid everexxploration, OVD-Explorer particu- larly takes the aleatoric uncertainty into account, leveraging the optimistic value distribution (OVD) to formulate the upper bounds of policy's return. Then, OVD-Explorer derives the exploration pol- icy via maximizing the mutual information between policy and corresponding upper bounds for optimistic exploration. Meanwhile, the derived behavior policy avoids being optimistic to alcatoric uncertainty, thus avoiding over-exploration issue.
In the following, we first give the intuition together with the theoretical derivation of OVD-Explorer. Then we practically describe how formulate OVD based on uncertainty estimation. Lastly, anal- ysis are given to further illustrate why OVD-Explorer can explore effectively and robustly.
3.1 OPTIMISTIC VALUE DISTRIBUTION GUIDED EXPLORATION
Most OFU-based algorithms follow the exploration principle: "select the action that leads to areas with high uncertainty". However, this principle is incomplete the high uncertainty may also be caused by the aleatoric uncertainty, which misleads the exploration direction. Hence, we comple- ment the principle by introducing constrain: "Not only select the action that leads to areas with
3

---

Under review as a conference paper at ICLR 2022
high uncertainty, but also avoid the ones that only lead to the area with high aleatoric uncer- tainty". Notable, this new principle intuitively describes a good exploration ability, and we will illustrate how to quantitatively measure policy's exploration ability in the later section. By follow- ing this principle, OVD-Explorrr derives the behavior policy for exploration.
Intuitively, each policy in the policy space II has different exploration ability (at state s), denoted by F"(s). (s). Given current state 8, OVD-Explorer aims at finding the exploration policy ae with the best exploration ability by solving the following optimization problem:
E arg max F (s).
sEIl
We propose to quantitatively measure the policy's s exploration ability from an information- thhooretic perspective by measuring the mutual-information of multi-variables as follows:
FT (s) MI(Z (s)).... - (s, ak-1); [--ss)ss),
(8)
where T T-|ss) is the action random variable, and (s, a,) the random variable describing the upper bounds of return that action a, can reach under policy π at state s. Here, a; A denotes any legal action, thus could be infinite in continuous action space. The higher F (s) is, the better ability that policy π has in exploring towards higher epistemic uncertainty while simultaneously avoiding higher aleatoric uncertainty, which satisfies the principle we raised. (see analysis in Section 3.3
Now state the Theorem I measure the mutual information above in continuous action space. Theorem 1. The mutual information Equation at state 8 can be approximated as:
& z2 (..s
(*) C
E
D2 (z(s log
(9)
C
x(*.a)=-2* (R.a)
(2(+) is the cumulative distribution function (CDF) of random variable x. E(8, is the sampled upper bound of return from its distribution 7. (s, a) following policy n, 7" describes the current return distribution of the policy n, and C is a constant (See proof in Appendix(...
Theorem D reveals that (s) is only proportional to Z (ā(s, a)), by maximizing which the pol- icy's 's exploration ability can be improved. Concretely, given any policy To (parameterized by 0), the derivative V 25 (ž(s, a)) can be measured, thus gradient ascent can be iteratively performed to derive a better exploration policy. (The complete closed form solution is given in later Section
Note that, to perform above optimization procedure, we need to formulate two critical components in 2z (2(s, a)): the return of the policy (* a) and 2 upper bound of the return of the policy Z (s, a). Therefore, we first details the formulations in Section 3.2 and then analyze why maxi- mizing 2- (z(s,a)) a)) can derive a better exploration in Section 3.3
3.2 FORMULATING THE RETURN OF POLICY AND CORRESPONDING UPPER BOUND
Now, we introduce the formulation of the return of policy Z* a) and corresponding upper bound 2" (s, a). As mentioned before, most OFU-based methods neglect the aleatoric uncertainty when formulating the return and upper bound of return (Mavrin et al.) 2019; Chen et al., 2017). result- ing in the over-exploration issue. Therefore, OVD-Explorer particularly takes into account the aleatoric uncertainty, leveraging distributional RI. paradigm to characterize aleatoric uncertainty [Bellemare et al., 2017 Dabney al. 2018b ap. OVD-Explorer uses two value distribution estima- tors 2ss, a; 02) and (ss, a: 02) parameterized by 01 and f22 as ensemble estimators to formulate Z" and z" in different ways. Unless stated otherwise, the (s, a) is omitted hereafter to ease notation.
Formulation of 2=. The 2t denotes the upper bounds of value that policy и can reach via different actions. We propose Gaussian distribution with optimistic mean value for formulation as follows, and accordingly refer to it as Optimistic Value Distribution (OVD):
2= a) ~ ((z2(s, a astattori a));
(10)
where (2 and 2 aeeatoric (s, is mean value and variance, respectively. Notable, Chen et al. 2017) discovers the optimisticity is beneficial for better estimating the upper bound, which moti- vates us to optimistically estimate the (12(s, a) by considering epistemic uncertainty as follows:
{ u(s, = -E Ex 0² oppissemic (s, a) E, va 02)
(s, a) = (=(s a) + B0 epistemmi (2, a), s.t.
(11)
4

---

Under review as a conference paper at ICLR 2022
Table 1: The comparison about scenarios. Note that for the ease of clarity, without causing ambiguity, we omit part of the notation in the table.
Fig.2 2
Aleatoric uncertainly
Epistenic uncertainty
Optimistic value estimation 2. (s.a)
CDF value ((a)
Action a
(8)
da (a(u:) (a()))
(a(01] (x(u2)
22 0s) pzlee d2)
P(ai) 4(a2)
d1
(b)
(a (ala))
(a())) (a(a))
12 12 u2la,
+(a1) ((a)
where ja(s, a) represents the Q-value estimation, Depistemic (8, a) the epistemic uncertainty, / is a hyper-parameter controlling the magnitude of epistemic uncertainty, / is uniform distribution, N is the number of quantiles, and (s,s, a; 0x) is the value of the i-th quantile drawn from (1s, a; 0s).
Moreover, the aleatoric uncertainty should be considered as will affect the policy's performance, thus propose to model it the variance of 2T, In practice, the alcatoric uncertainty alcaatori (5, a) can be captured by value distribution estimators (Clements et al. 2019)) as follows:
2 aeeatooic (s, ((1,-)((1.N)) 2,==1,2.(s,, a0:k]
(12)
Leveraging optimistic value estimations (via epistemic uncertainty) together with explicitly mod- eling aleatoric uncertainty, the upper bound of the value 2" can be effectively formulated as an optimistic Gaussian distribution. Such an upper bound value estimation provides useful guidance for OVD-Explorer to derive the behavior policy for online exploration.
Formulation of 7" 7 estimates the value obtained following policy æ. Inspired by TD3 (Fujimooo et al. 2018), to alleviate overestimation, we propose to formulate 2" in a pessimistic way. In practice, 6T can be measured in two ways. First, similar to formulating 2 Equation 10 Z can also be formulated Gaussian distribution as follows:
("(8,(a) a) NNZzz (8, a), aulestooic (8,a)), s.l. (1z=(8,0) ((s, _0 epiistemiic (8, a),
(13)
where pa(s, a). O alleatoric (s, a) and appistemic a) are the same defined in Equation 11 Differently, Jepistemic (s, a) is subtracted from pls, a), which reveals the pessimistic value estimation of Z"
Another way is to formulate 2 as multivariate uniform distribution as follows:
as, u1= (8, a)))))......... S.l. 2, (s. a;
min (s,,,,; k=1,2
(14)
where each quantile value z (s, a; 0) equals to the minimum estimated value among ensemble esti- mators (i.e., (s, a; 02). As such, the estimated value of 2 relative pessimistic.
Adopting pessimistic value estimation avoids overestimation issue, thereby the value distribution 2" can be more accurately formulated. Additional, Gaussian distribution helps more when the environment randomness follows a unimodal distribution, and multivariate uniform distribution can be more flexible and suitable for scenarios with multi-modal distributions. (see more in Section 5.3
3.3 ANALYSIS OF OVD-EXPLOORER
This section analyzes how OVD-Explorer optimistically explores the informative areas and avoids the over-exploration issue. According to Theorem [] the exploration policy OVD-Exxlorer derives maximize F (s), which is proportional to CDF value Φ (ā(s, In the following, Fig- ure ((a and 2bb illustrate such CDF values for different actions (shaded area), and Table [ shows how uncertainties affect the exploration.
In these cases, the value distribution (s a) is specified as Gaussian distribution (in Equation 13 and the sampled optimistic value æ(s, a) is specified as the mean of OVD uz(s, a) (in Equation II] Specifically, at state we assume that the means of 2" at actions and are the same for ease of clarification.
Basically, Figure ((a) shows OVD-Exxlooee can achieve more optimistic exploration. We assume the aleatoric uncertainty at 01 and ag are the same, but epistemic uncertainty is higher at 01 causing a > (z(s,a2), 42). Thus the CDF value is larger at a,, which means the action with higher epistemic uncertainty is preferred.
5

---

Under review as a conference paper at ICLR 2022
More crucially, Figure 2(b) shows that OVD- Explorer could guide to avoid the area with higher aleatoric uncertainty given the equal op- timism. We assume that the epistemic un- certainty aj and 02 are the same, causing their optimistic value estimation to be equal to (2(s, a), but aleatoric uncertainty is lower at d1, i.e., PDF curve of Zd (s. 01) is "thinner and taller". Thus the CDF value is larger a1 meaning that the action with lower aleatoric un- certainty preferred.
Photeiiitty Demiity
2-144) (1)
Phobbiiliyy Develly
ZGA) 2(00)
(y(88) (2(5)) (a)
(2(,,))
3.0
(b)
Figure 2: How OVD-Explorer explores (a) op- timistically about epistemic uncertainty and (b)
pessimistically about aleatoric uncertainty.
In the of & -> the epistemic uncertainty tend to be 0, and the aleatoric uncertainty esti- mated in Equation 12 converges to represent the true environment randomness. This is comparable to the second case (Figure 26). and OVD-Explorer tends to choose the one with lower aleatoric uncertainty, which is great advantage of our method over other OFU methods.
To summarize, the exploration guided by OVD-Ex DVD-xpoorrer trades off between two criteria: exploring the areas with higher epistemic uncertainty, to ensure exploring optimistically, and avoiding the areas with higher aleatoric uncertainty, to avoid over-exploration caused by high environment randomness.
OVD-EXPLORER FOR MODERN RL ALGORITHMS
For continuous action space, the argmax operator in Equation 7 is intractable. To address that, in this section, we derive the behavior policy TE in closed form and state the scheme to incorporate it with existing policy-based algorithms.
First, we denote the policy learned by any policy-based algorithm as the target policy Tr. To avoid the gap between training data collected using the behavior policy Tg and the target policy Tr. we need to constrain the difference between y and Ty, thus we derive Th in the vicinity of कт.
Second, to derive exploration Tæ for modern RL algorithms with stochastic policy based on OVD- Explorer, where both the exploration TE Niuu, Σς) and target policy Gaussian distributions, we introduce the following proposition:
N N(UT, Er) are
Proposition 1. The Explorer behavior policy = ~NEE, DE) is as follows:
05(s,a) a)
PE = pT + a.E
x
da a=
(15)
and
Σg ET.
(16)
In specific, or is the step size controlling the exploration level and m log 4 (o (B(e,rr)) (see proof in Appendix B.2.
The expectation Ez, can be estimated by sampling samples, then Equation 15 is simplifies as:
am K
K
02,(8, a) a=
ME = ppr +
(17)
da
Algorithm summarizes the overall procedure of OVD-Explorer, including the formulation of n and z (Line 2 and line 3) and behavior policy generation (Line 4). The generated behavior pol- icy can be integrated with any modern posicy-based RL algorithms to render the stable and well- performed algorithm. Please see Appendix [ for the entire pseudo-code and details about how OVD -VD--xplorer-basee behavior policy is incorporated with DSAC.
5 EXPERIMENTS
To reveal the consistency between our theoretical analysis and the performance of OVD-Explorer algorithm, conduct experiments to address the following questions:

---

Under review as a conference paper at ICLR 2022
Algorithm The behavior policy (i.e., exploration policy) derived from OVD-Explorer.
Input: Current state 8t, current value distribution estimators 0,, 02. current policy network ø.
Output: Behavior policy Tg.
1: Obtain target policy from volicy-based RL algorithm TT (T((-((;))) (N(pr()))) or(s
2: Derive OVD (s;, (s;; 6)) using Eq.
3: Construct value distribution of behavior policy Z (= ( Φ)) using Eq. 13 or
4: Calculate the mean of behavior policy distribution μe using Eq.
5: return E
RQ1 (Exploration): Can OVD-Explorer explore optimistically while avoiding over-exploration simuitaneously?
RQ2 (Performance): Can OVD-Explorer handle complex and even stochastic tasks?
RQ3 (Sensitivity to a): Is OVD--Explorer sensitive that controls exploration magnitude?
5.1 EXPERIMENT SETUP
Baselines include SAC (Haarnoj et al. 2018), DSAC eM al. 2020), and DOAC, which is the dis-
tributional variant of OAC (Closek et 2019). We implement Gaussian and quantile formulations of 2d as in Equation 13 and 14] which are denoted in the following as OVDE_ G and OVDE Q. re- spectively. We test OVD-Explorer on a novel task GridChaos and several tasks in Mujoco (Todorov et al. 2012) including the stochastic variants.
The appendix gives more details, including environment settings, implementation details, hyper- parameters settings, and computing infrastructure used, as well as more experiment evaluations, including tasks with different noise scales, tasks with different episode horizon, the sensitivity and ablation study on the pessimistic formulation of 2 2²'(s.0. a).
5.2 EXPLORATION IN GRIDCHAOS (RQ1)
To illustrate the exploration pattern of OVD-Explorer and show the advantage of OVD-Explorer over DSAC and DOAC, we evaluate OVD-Explorer on novel continuous and stochastic control task called GridChaos.
Figure 3(a) shows the map of GridChaos, in which we control the cyan triangle (agent) aim- ing to reach the fixed dark blue goal. The state is the current coordinate, and the action is two-dimensional vector including the move- ment angle and distance. One episode termi- nates when the agent reaches the goal max- imum steps (i.c., MAX STEP, typically 100). Also, it receives +100 reward when reach- ing the goal otherwise O. The reason why it is chaos is that the randomness of the transition is heterogeneous in different parts of this envi- ronment. Table 2 shows the environment set- tings. Moreover, Figure 3(b) shows that OVD- Explorer can reach the goal faster, with more efficient exploration.
Table 2: Settings of GridChaos.
Value
Description
Ohservation[0) Observation[]] Action[O]
[-1, 1]
X-coordinate
[-1, 1]
Y-coordinate
[-1,1]
Degree, mapped to
[-T,, m]
Action[1]
[-1,
Distance, mapped to [0, MAX_STEP]
Noisc_0
0.5
Variance of Gaussian noise in the left half of the map (default)
Noise 1
0.1
Variance of Gaussian noise in the right half of the map (default)
Figure 3(c) shows the values of uncertainty estimation and exploration objective (mutual informa- tion) taken four different actions at the state shown in Figure (3a] sically, Figure ((c illustrates that the estimated aleatoric uncertainty of left is higher than that of right, indicating that OVD-Explorer models aleatoric uncertainty properly. Further, OVD-explorer encourages to explore the right side at that time, since the value of exploration objective (in green) is highest. It implies that OVD-explorg tends to explore areas with higher epistemic uncertainty and avoiding higher aleatoric uncertainty, which is in accordance with our exploration principle. On the other hand, if high epistemic uncertainty is considered only like in OAC(Ciosek et al. 2019, the
and and at training epoch 1249. Ba-

---

Under review as a conference paper at ICLR 2022
Training epoch 1249
DSAC OYDE DOAC
Aveaag Raturn
Type
1.
Epistemic uncertainty
Aleatoric uncertainty
20
Mutual information
563
up
down
left
right
Number Training Epoch
(A)
(b)
(c)
Figure 3: GridChaos. (a) The cyan triangle move reach the goal the top right. (b) The performance on it. (c) The values about uncertainty and exploration objective (mutual information).
agent would be guided to the left. Then the agent may be trapped in the left side due to the high aleatoric uncertainty, and it is the possible reason about why OAC fails tackling such heteroscedastic stochastic task. In appendix, we show more analysis about the exploration pattern, please refer to Appendix E.2 and Appendix E0.0
Besides, we conduct the evaluation when the randomness is high around the goal, as well as many other noise settings, the results enhance the ability of OVD-Explorer, More experiments that evaluate the performance on several other noise settings, as well as more detailed value histogram on different epoch in training process can be found in Appendix.
5.3 PERFORMANCE ON MUJOCO TASKS (RQ2)
To demonstrate the performance of OVD OVD-Explorer more generally, we evaluate it on several Mujoco tasks. For standard Mujoco tasks] the transition is deterministic and the randomness is only from the stochastic policy. For 5 noisy tasks, the heteroscedastic Gaussian noise is added in Mujoco tasks. Specifically, each state transition on noisy tasks, Gaussian noise of different scales is randomly injected following certain probability. Overall, have the following findings.
Basically, OVD-Explorer can perform stably in the standard tasks with slight or little randomness, guiding efficient exploration. From the results?) in standard Mujoco tasks in table is shown that in the relatively easy tasks, ie.. Hopper-v2, Reacher-v2, and HalCheetah-v2 OVD-Explorer does not obtain much gain beyond the baselines, which seems that these tasks not profoundly demanding for exploration. In the high-dimension tasks, i.e. Ant-v2 as shown in Figure 4(a) OVD-Explorer significantly outperforms DSAC and DOAC. It means that the aleatoric uncertainty caused by policy indeed degrades the performance of DOAC, where the alcatoric and epistemic uncertainties are not distinguished.
Besides, OVD-Explorer can avoid the impact of heteroscedastic aleatoric uncertainty on exploration and thus improve robustness. As the results of noisy Mujoco tasks shown in Table DOAC is worse than DSAC in most cases, which means that heteroscedastic aleatoric uncertainty causes significant degrades of DOAC. Simultaneously, OVD-Explorer significantly outperforms DOAC and DSAC, especially in Noisy Ant-v2 as shown in Figure 4(b)
Furthermore, OVDE_G takes the Gaussian prior and concretely obtains better estimation of critic in the noisy tasks, while OVDE_Q performs distribution. Generally, the difference between OVDE_G and OVDE_Q is that they use different formulations of 7 (s, a). Theoretically, the value function distribution in OVDE and should perform better than OVDE as the reported results on five standard Mujoco tasks show. On the other hand, OVDE G performs better in the noisy tasks, and it is because that the transition probability is Gaussian in the stochastic Mujoco tasks.
perffrrms better in the standard tasks due to the flexibility of quantile
more flexible
'htps://github.co 'https//gitubb.oo//oeaaaaa/ htwpg//wwww...oooooooooooooeonne
Here, the reported results may be slightly different from previously reported results, partly due to the statistic approach, and partly due to the implementation. Nevertheless. the patterns of these baselines (e.g.. DSAC outperforms SAC in the vast majority of cases) are consistent with previous results. The details about the implementation of haselines descrihed in the Appendix D.2.
8

---

Under review as a conference paper at ICLR 2022
Table 3: Comparisons of algorithms on five standard and five noisy tasks in Mujoco. We report the averaged performance and standard deviation of runs. Each trail uses the mean undiscounted episodic return over the last 8% epoch (or at most the last 100 epoch) avoid bias, and the total epoch number is shown in column epoch. The maximum value of each row is shown in bold.
DSAC
DOAC
OVD-EXPLORER.G
OVD-EXPLORER.Q
ANT-V2
2500
2706.2
6206.9112024
6
7160.6:761.2
7590. 1594,9
HALFCHEETAB-
2500
12373.8=860.9
13890.0134244.4
12977.011:0.4
14054,5=1579.5
14792.49997.4
HOPPER-V2
1250
2751. 8-775.7
2199.
2215.1+557..1
22299.
2619,3+457.0
REACHER-V2
250
-21 021.622..
-11.9.0.5
-1988+1.7
-11.622.4
-10.8
INVDSPENDULUX-V2
300
9344.0 2:2.44
9359. 9389.61:011
5109.143638.7
9128. 06:660.6
9351.3=16.3
N-ANT-V2
2500
261.5:157.6
117.99=11.96
450.34158.44
N-HALICHISTTB--2
1250
351 91 +6668
431. 59++35 68
417,47=390
445 28 52.52
429 (199663+33444 A5
N- X-HOPPER-V2
1250
207 007.062:19 49
244.53:44.71
242.7A=7.87
252 19
237.68+13..
N.PUSHER-V2
1250
.46.92 12.12
.2..3112.46
9.006
-
N-IVIDMPINNUTLIMA22
300
9.34. 36 1.91
932.78-44.002
496.61 106.61-205.00
933.674 1.57
634.64-
500
6000
Average Retumn
400
340
320
300
4000
SAC
SAC
DSAC
DSAC
2000
DDAC
DOAC
280
OVDE
OVDE_G
OVDE_0
OVDE_9
240
500
2000
0.0
Number Training Epoch
Number Training Epoch
alpha
(a) Ant-v2
(b) Noisy Ant-v2
(c) Noisy Ant-v2
Figure 4: Training curves on (a) Ant-v2 and (b) Noisy Ant-v2. The x-axis indicates the number of training epochs, while the y-axis is the evaluation result represented by the average episode return. The shaded region denotes the half standard deviation of average evaluation over seeds. Curves are smoothed uniformly for visual clarity. (c) Sensitivity a. The x-axis indicates different a settings, while the y-axis is the evaluation result represented by average episode return in the last 100 epoch before total 1250 epoch. Error bars indicate half standard deviation of average evaluation over seeds. The 9 different a values are 0.0005, 0.01. 0.025, 0.05, 0.075, 0.1, 0.25, 0.4, 0.8, respectively.
5.4 SENSITIVITY TO (RQ3)
The step size a in Equation 15 controls how much the behavior policy derived from OVD-Explore iee we, is far away from the target policy ær, which can be essential for exploration benefit. To investigate the appropriate range of it, we test several a value on Noisy Ant-v2 task using OVDE_G, and the result is shown in Figure [[c) If a is quite small, OVD-Explorer degenerates to DSAC and implies little exploration. In contrast, if is larger, the performance becomes worse because of the huge gap between behavior policy and target policy, The result demonstrates that a should be taken in suitable range to facilitate more adequate exploration. In experiments, uniformly use a to be equal to 0.05 to show the performance of the OVD-Explorer. In addition, we find that a smaller a leads to higher gains when the task is much more difficult, and the details be found in Appendix F.6) We also verify the sensitivity of OVD-Explorer to ,3 in Appendix H.7 there broad range of settings for 8, which can lead well performance.
F.7land and found that
CONCLUSION
This paper proposes an information- information-theoretic theorrtic exploration method OVD-Explorer, which introduces a novel measurement of exploration ability, i.e., the mutual information between the policy and the upper bounds of return. By maximizing the mutual information, OVD-Explorer is able to derive the behavior policy, that follows the OFU principle, and further avoids exploring the areas with high aleatoric uncertainty. Integrated with SAC, OVD-Oxplooe uncertainty for exploration in continuous RL for the first time.
addresses the negative impact of aleatoric

---

Under review as a conference paper at ICLR 2022
REFERENCES
Peter Auer, Nicolò Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Mach. Learn., (7(22-))225--222 2002.
Syrine Belakaria, Aryan Deshwal, and Janardhan Rao Doppa. Max-value entropy search for multi- objective bayesian optimization with constraints. CoRR, abs/2009.0172. 2020.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, R. Devon Hjelm, and Aaron C. Courville. Mutual information neural estimation. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmüssan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 530- 539. PMLR, 2018.
Mare G. Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Rémi Munos. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Infor- mation Processing Systems, pp. 1471- 1471-1479, 2016.
Mare G. Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Re- search, pp. 449 458. PMLR, 2017.
Richard Y Chen, Szymon Sidor, Pieter Abbeel, and John Schulman. Ucb exploration via q ensembles. arXiv preprint 1rXiv:1706.0102. 2017.
Pengyu Cheng, Weituo Hao, Shuyang Dai, Jiachang Liu, Zhe Gan, and Lawrence Carin. CLUB: A contrastive ratio upper bound of mutual information. In Proceedings of the 37th Inter- national Conference on Machine Learning. ICML 2020, 13-18 July 2020, Virtual Event, vol- 119 of Proceedings of Machine Learning Research, pp. 1779-1788 PMLR, 2020. URL wwwps/essvvvlee /vlssvvvvii 9/ 9/ ch/ng20b. html
Kamil Ciosek, Quan Vuong, Robert Loftin, and Katja Hofmann. Better exploration with optimistic actor critic. In Advances in Neural Information Processing Systems 32: Anmual Conference Neural Information Processing Systems 2019, NeurIPS 2019, 8-14 December 2019. Vancouver,
BC, Canada, pp. 1785-1796, 2019.
William R. Clements, Marie Robaglia, Bastien Van Delft, Reda Bahi Slaoui, and Sébastien Toth. Estimating risk and uncertainty in deep reinforcement learning. CoRR, abs/ 2019.
Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distri- butional reinforcement learning. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmäässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 1104-1113 PMLR, 2018a.
Will Dabney, Mark Rowland, Mare G. Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In Proceedings of the Shirty-Secondl AAAI Conference Artifi- cial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (1AAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pp. 2892-2901. AAAI Press, 2018b.
Richard Dearden, Nir Friedman, and Stuart J. Russell. Bayesian q-learning In Proceedings of the Fifteenth National Conference on Artificial Intelligence and Tenth Innovative Applications of Artificial Intelligence Conference, AAAI 98, IAAI 98, July 26-30, 1998, Madison, Wisconsin, USA, pp. 761-768, 1998.
Lior Fox, Leshem Choshen, and Yonatan Loewenstein. Dora the explorer: Directed outreaching reinforcement action-selection. In International Conference on Learning Representations, 2018.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Proceedings of the 35th International Conference Machine Learning, SCML2018 Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 Proceedings of Machine Learning Research, pp. 1582-1591. PMLR, 2018.
10

---

Under review as a conference paper at ICLR 2022
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference Machine Learning. ICML 2018, Stockholmsmüssan. Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 1856- 1865. PMLR, 2018.
Geoffrey E. Hinton and Drew van Camp. Keeping the neural networks simple by minimizing the description length of the weights. In Proceedings of the Sixth Annual ACM Conference on Compu- tational Learning Theory, COLT 1993, Santa Cruz, CA, USA, July 26-28, 1993, pp. 5-13. ACM, 1993.
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel, Vime: Variational information maximizing exploration. In Advances in Neural Information Processing Systems, pp. 1109-1117, 2016.
Hyoungseok Kim, Jaekyeom Kim, Yeonwoo Jeong, Sergey Levine, and Hyun Oh Song. EMI: exploration with mutual information. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 3360-3369, 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
Johannes Kirschner and Andreas Krause. Information directed sampling and bandits with het- eroscedastic noise. In Conference On Learning Theory, COLT 2018, Stockholm, Sweden, 6-9 July 2018, volume 75 of Proceedings of Machine Learning Research, pp. 358-384. PMLR, 2018.
Roger Koenker and Kevin Hallock. Quantile regression. Journal of economic perspectives, 15(4): 143-156, 2001.
Shibo Li, Wei Xing, Robert M. Kirby, and Shandian Zhe. Multi-fidelity bayesian optimization via deep neural networks, In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
Timothy Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In 4th International Conference on Learning Representutions, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
Xiaoteng Ma, Qiyuan Zhang, Li Xia, Zhengyuan Zhou, Jun Yang, and Qianchuan Zhao. Distribu- tional soft actor critic for risk sensitive learning. CoRR, abs/2004. 14547, 2020.
Jarryd Martin, Suraj Narayanan Sasikumar, Tom Everitt, and Mareus Hutter. Count-based explo- ration in feature space for reinforcement learning. In Proceedings of the Twenty-Sixth Interna- tional Joint Conference on Artificial Intelligence, pp. 2471-2478, 2017.
Borislav Mavrin, Hengshuai Yao, Linglong Kong, Kaiwen Wu, and Yaoliang Yu. Distributional rein- forcement learning for efficient exploration. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 4424-4434, PMLR, 2019.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wier- stra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nat., 518(7540):529-533 2015.
Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International Conference Machine Learning, pp. 1938-1937, 2016.
11

---

Under review as a conference paper at ICLR 2022
Thomas M. Moerland, Joost Broekens, and Catholijn M. Jonker. Efficient exploration with double uncertain value networks. CoRR, abs/ abss/171..1079, 2017.
Nikolay Nikolov, Johannes Kirschner, Felix Berkenkamp, and Andreas Krause. Information- directed exploration for deep reinforcement learning. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. In Advances in Neural Information Processing Sys- tems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 271-279, 2016.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped DQN. In Advances in Neural Information Processing Systems 29: Annual Confer- ence on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 4026-4034, 2016.
Georg Ostrovski, Marc G. Bellemare, Aäron van den Oord, and Rémi Munos. Count-based ex- ploration with neural density models. In Proceedings of the 34th International Conference on Machine Learning, pp. 2721-2730, 2017.
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In Proceedings of the 34th International Conference on Machine Learming, pp. 2778-2787 2017.
Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via disagreement. In Proceedings the
the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach. California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 10062-077. PMLR, 2019.
Valerio Perrone, laroslav Shcherbatyi, Rodolphe Jenatton, Cédric Archambeau, and Matthias W. Seeger. Constrained bayesian optimization with max-value entropy search. CoRR, abs/1910.07003, 2019.
Nikolay Savinov, Anton Raichuk, Damien Vincent, Raphaël Marinier, Mare Pollefeys, Timothy P. Lillicrap, and Sylvain Gelly. Episodic curiosity through reachability. In 7th International Con- ference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. Open- Review.net, 2019.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. #exploration: A study of count-based exploration for deep reinforcement learning. In Advances in Neural Information Processing Systems, pp. 2753-2762, 2017.
William R Thompson. On the likelihood that one unknown probability exceeds another view of the evidence of two samples. Biometrika, 25(3/4):285-22 294, 1933.
Emanuel Todorov. Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based con- trol. 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2012, Vilamoura, Algarve, Portugal, October 7-12, 2012, pp.
IEEE, 2012.
Zi Wang and Stefanic Jegelka. Max-value entropy search for efficient Bayesian optimization. vol- ume 70 of Proceedings of Machine Learning Research, pp. 3627-3635, International Convention Centre, Sydney, Australia, 06-11 11 Aug 2017. PMLR.
Christopher J. C. H. Watkins and Peter Dayan. Technical note qeleaminng Mach. Learn., 8:279-292, 1992.
Fan Zhou, Jianing Wang, and Xingdong Feng. Non-crossing quantile regression for distributional reinforcement learning. In Advances in Neural Information Processing Systems 33: Annual Con- ference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
12

---

Under review as a conference paper at ICLR 2022
A RELATED WORK
In this work, we consider the exploration strategy under the principle of Optimism in the Face of Uncertainty (OFU) (Auer et al., 2002), especially in the heteroscedastic stochastic environment. We aim to improve exploration efficiency, and alleviate the over exploration issue caused by aleatoric uncertainty.
Basic exploration strategies, like e-greedy (Suttoon Barto 2018). noise perturbation (Lillicrap et al, 2016), entropy regularization (Mnih et al. 2016) and stochastic policy (Haarnoja et al. 2018), lead to undirected exploration through random perturbations. With the increasing emphasis on explo- ration efficiency in RL., various exploration methods have been developed. One kind of methods uses intrinsic motivation to stimulate agent to explore, such as count-based novelty (Martin et al.)
2017 Ostrovski et al., 2017, [ellemare al. 2016 Tang et al. 2017 Fox et al. 2018). prediction error Pathakk et al., (2017). reachability (Savinov et al., 2019) and information gain on environment dynamics (Houthooft al., 2016). Some recent methods, originating from tracking uncertainty, guide efficient exploration under the principle of OFU, such as Thompson Sampling (Thompson 1933 Oshand al.] 2016), IDS (Nikolov et al. 2019 Clements al. 2019) and other customized
methods (Moerland et al. 2017 Pathak et al. 2019).
The base of OFU methods is to model epistemic and aleatoric uncertainties in RL. Bootstrapped DQN (Osband et al. 2016) has become the well-used approach for capturing epistemic uncertainty
Kirschner& & Krause 2018 Ciosek et al. 2019), and distributional RL methods (Bellemare et al.)
2017 Zhou al.] 20200] Dabney al. 2018a
used for capturing aleatoric uncertainty. How- ever, most traditional OFU methods do not distinguish the two types of uncertainty, which can easily lead the naive solution to favor actions with higher variances in stochastic tasks, i.e., over-exploration
issue.
To address that, Mavrin al. (2099) study how to take advantage of value distribution for effi- cient exploration under both types of uncertainty, proposing Decaying Left Truncated Variance (DLTV) based on QR-DQN. Besides, Nikolov al. (2019) and Clements al. (2019) propose to use Information Direct Sampling (Kirschner & Krause 2018)) for efficient exploration in RL, which formulates epistemic and heteroscedastic aleatoric uncertainty and maximizes information gain on globally optimal action to explore informative state-action pairs. However, such methods are complicated when deriving a behavior policy and limited to discrete control.
Meanwhile, there is not any strategy that can help the well-performed continuous RL algorithms (Haarnoj al., 2018, Ciosek et al. [019 Ma et al., (2200) to address aleatoric uncertainty when exploration. OAC (Ciosek et al. 2019) proposes exploration bonus guided by the upper bound of Q estimation to facilitate exploration based on Soft Actor-Critic (SAC) (Haarnoja et al., 2018). Nevertheless, OAC ignores the potential impact of the aleatoric uncertainty. which may cause mis- leading exploration. Our proposed OVD-Explorer a novel exploration strategy, which can guide agent explore towards higher epistemic uncertainty, and also avoid the areas with high aleatoric uncertainty, improving the robustness of exploration especially facing heteroscedastic aleatoric un- certainty.
To capture aleatoric uncertainty, OVD-Explorer models the value distribution and uses mutual in- formation to guide exploration following the principle of OFU, measuring the correlations between the policy distribution and upper bounds distribution of return. There are some other information- theoretic exploration strategies using mutual information, such as VIME (Houthooft et 2016), which measures the information gain on environment dynamics, and EMI (Kim et al. 2019) which generates intrinsic reward using prediction error of representation learned by mutual information. Those methods can solve sparse reward problem very well by using intrinsic reward. Nevertheless, those exploration methods use mutual information neither on the value distribution, nor for OFU- based exploration. Besides, unlike the mechanisms used in measuring mutual information, such as variational inference (Hinton & van Camp] 1993] Houthooft et al., 2016) and f-divergence (Nowozin et al. 2016 Kim et al. 2201)), we find the correlation between policy and upper bounds of return through uncertainty as shown in Theorem [| thus we can directly derive the close form exploration policy.
13

---

Under review as a conference paper at ICLR 2022
B PROOFS
B.1 PROOF OF THEOREM 1
In order to prove the Theorem we first propose the following lemma about F (s).
Lemma 1. The mutual information of (=, (s, ak-1) and T(-1s) at state s is:
F (-1s) = ( ( E
((,
p(alε(s, a), s) log <((ss)
da,
(18)
2(8,0)=Z" (s,a)
where p(a 2(s, a), s) represents the posterior probability distribution of policy given current state and the sampled upper bound of return a).
Proof. For simplicity, we assume that the size of action space is A: = 2, and the actions are denoted og and an, ag # a1 Then derive the mutual information among three random variables MI (8, ao), 2d (s, a1), (-]))), where the action sampled from r(-[s) is either ag or 01
Considering the formula for the mutual information, F (s) is derived as follows:
(=((1)) MI 2n (", on), 2n (7 (1)01))=((s))())
5(8, an) a1)) log ((0, ((0)))p(3(s.00))
Σ
a
3(5,ag)~2* (s.0o) =(8,01) (8.01)
p(a|z(s, ((s,))
Σ
a1)) log
(aass)
5(s,ag)~2* (8,00) 3(5,01)~2""(xn))
au), au), a1)) log p(a|a(s, a0), a())
Σ
E (s.00) a-w((1s) ((9.a))~2"(c) (8.01)
(a(s))
where the posterior distribution p(a]ā(s, on), I(8 a.) ) is the probability of choosing action a on the condition of the samples from upper bounds of action do and a1
Considering that in the decision-making process, the probability of action ag independent to the upper bound of other actions, such 5(s, a1), which means that p(aol=(ss, ao), (1)) p(ao) (5 an)). Therefore, the above equation can be further reduced as follows.
p(aol as)
F (+((s))
E 3(8 3(s.ag) (c.
plaol an), 8) log
<(aols)
(a,)3((,a1)) <(a s)
p(arlE(s, (1) log
=(e,0))~2 (s..a))
It is easy to extend infinite-action case. Based on that for any action a; [ao, a 1] and +
the conditional probability p(a,lā(s, ag), ø(s, az), ..., E(8, (x-)) = p(a,)a(s, a))). F (-(ss) can be simplified following,
plalā(s,a), a), s) log (a)=
F (+(ss)
E 2* (s.a)
da.
(19)
[{als))
Lemma I tells that the mutual information F (s:) is in direct proportion to p(a|ž(s, a), which measures how much worth acting under the current policy (als) when the upper bound is known.
14

---

Under review as a conference paper at ICLR 2022
Next, measure the posterior probability 2( a). use general and practically effective approach (Wang & Jegelka 2017 Belakaria et al., 2020; Perrone et al. 2019 Li et al. 2020 of approximating the posterior probability given upper bound value.
Specifically, we approximate p(alz(s,a), using the prior that 2" a) S E(s, a) with given pol- icy =(s,a), since 2(s, a) is the upper bound of a). Hence, ( to truncate the policy (s, a), and utilize the constant C to normalize the probability, is shown in the following equation.
the indicator function
p(alz(s,a),s) R Ba (x.a) (33663))))
Here, (sx.)))))((().a) [1 tion function (CDF) of 2, z" and z" are the random variables, whose distributions describe the randomness of the returns, and ø(s, a) the value of random variable z*.
(z (E(s,a))) (E,s) where 4a is the cumulative distribu-
Therefore, the posterior probability can be measured as follows.
p(alz(s,a),s) 2 (Z(ss))2
(20)
In method, we do not the commonly used mechanisms about mutual information such as neural network estimation (Belghazi et al. 2018) and upper bound estimation (Cheng et al. 2020).
Instead, we can find the correlation between random variables shown in Equation 20 which helps to derive mutual information directly.
According to Lemma and Equation 20 can give the proof of Theorem in the following.
Proof. By Combining Lemma and Equation 20 F (s) can be further derived as follows.
/ ((s.a)=2"(s.a))
p(a)=(8,a), 8)
F (6)
p(alā(s, 8) log
da
(an
<(a 8)
1
log π T 22 (5(s,
E
C T(ass) (Z(s,a)) (((),))
da
(awx(-ls) 20
Cm(aas)
4z- (2(s,o))
C T(ass) (z a)) log
da
(owe(11s) (8) 2mm.a)22mmm (s,a)
1
Φ (3(s. a))
E
2 (2(s, log
C
(-
C
a(s,a)~2
(e.aa
(8.0)
Here, the last equality follows from Theorem
B.2 PROOF OF PROPOSITION
Proof. Similar (Ciosek et al. 2019), we the covariance matrix of behavior policy mæ is that of target policy nr, l.e., DE Σr. Hence, the OVD-Explorer problem is simplified as:
#E = arg max ((s)
arg max, E (E(s(E(s, log 22- C
(21)
To ensure that the behavior policy samples actions around the target policy, derive the Fg upon mean μa of target policy सT. In specific, firstly obtain the gradient of F(ss uJ) at nr, which is given as follows:
V. FT (8,11))pppp = Ezz m x 0z(s, a)
(22)
Oa
l==pT
15

---

Under review as a conference paper at ICLR 2022
Algorithm 2 OVD-Explorer for DSAC
1: Initialise: Value networks 0,, 02. policy network & and their target networks 0,, 02. o. quantiles number N, target smoothing coefficient (7), discount (y), empty replay pool D
2: for each iteration do
3: for each environmental step do (a, 8t) according to Algorithm +- DU {(8t, 0z, ((t,, 8t+1)
4:
5:
6:
end for
7:
for each training step do
8:
for iori=: to N do
9:
for = 1 to do
10:
calculate 62 1, following Eq.
11:
end for end for
12:
13:
Calculate COR (O)(0 = 1, using of following Eq
14:
Update ON with V (O)
Calculate J= ()), following Eq. 5
16:
Update @ with (o)
end for
18:
Update target value network with 0g -0: + (1 -
10 k: 1,2
Update target policy network with @ f- TΦ - T)
20: end for
where n is given as:
+ zz (s.rr) HT)) + 1).
= 02 (x.,,)) (T))
(23)
and (xx) is the probability distribution function (pdf). Hence, the ue given follows:
05(s, a)
eE PT - alE 2
(24)
On
a=
where a is the step size controlling exploration level and log (((orr)()(()))) + 1.
C ALGORITHM 2: OVD-EXPLORER FOR DSAC
In this section, we show the whole algorithm of our implementation of OVD-Explorer based on DSAC in Algorithm 2
All the code be found in the supplementary material.
D MORE DETAILS ABOUT THE EXPERIMENTS
D.1 GRIDCHAOS
GridChaos is an environment built on OpenAl's Gym toolkit, whose map is shown in Fig- ure 3(a) and Figure 5 In this section we illus- trate more details in addition Se2
The movable cyan triangle and the fixed sym- metric dark blue goal are two parts split from square, and the goal is to make the triangle embedded in the goal to recover the original square, which is to say that it is an isosceles triangle whose base side is equal to the height. The triangle is always initialised randomly in the cyan rectangle, and the black line in the map represents the wall, where the triangle will be
4
Figure 5: Map of GridChaos.
16

---

Under review as a conference paper at ICLR 2022
Table 4: OVD-Exxlorer parameters
Parameter
Value
Training
Discount
0.99
Target smoothing coefficient
5e-3
Learning rate
3e-4
Optimizer
Adam
(Kingma & Ba 2015)
Batch size
256
Quantiles amount
20
Replay buffer size
1.0 x 10 for Mujoco tasks
1.0 X 10 for other tasks
Environment steps per epoch
1.0 X 10 for Mujoco tasks
1.0 x 10 for other tasks
Exploration
Exploration ratio
0.05
Uncertainty ratio
3.2
Normalization factor
C 0.5
adsorbed once hits the wall. The state transition is stochastic, and we add Gaussian noise to the action resulting in Gaussian transition probability.
To represent the location of the triangle, we establish a Cartesian coordinate system using the cen- troid of the map the origin, shown in Figure 5 Then the coordinates of the triangle are repre- sented by the midpoint of the altitude of the triangle which is shown as the red point in the triangle. In the shown in Figure 5 the initial coordinate of the triangle (agent) is in the negative half of the x-axis and the task target is in the first quadrant.
D.2 BASELINES
Ma et al. (2020) shows the performance of DSAC and TD4, which is the distributional extension of TD3 Fujimoto al. 2018), and can also be used to capture epistemic and aleatoric uncertainty. Moreover, DSAC outperforms TD4 on Mujoco tasks as shown in Ma et al. (2020), so we evaluate only on SAC and DSAC, and further implement OVD-Explorer based on DSAC to develop the exploration ability.
SAC. The SAC (Haarnoja et al. 2018) implementation is mainly based on OAC repository, and the results in Ant-v2 and Hopper-v2 are similar to reported results by OAC. Our SAC report better result than OAC's implementation for SAC on HalfCheetah-v2, which is because the high variance of this environment explained in OAC.
DSAC. The DSAC Ma et al. 2020) implementation based on our implementation SAC, except that the distributional function in the DSAC repository is used instead of the traditional Q function in SAC. As it is based on SAC, we set the hyper-parameters parameters of DSAC to be consistent with SAC to ensure the fair comparison, which also results in the different reported results from original paper of DSAC. In our results, ASAC can guarantee an absolute advantage over SAC in most cases, which is consistent with the previous conclusion.
DOAC. The DOAC implementation is mainly based on our implementation of DSAC as well as the open source code of OAC. As DSAC shows great advantage due the distributional value estimation, to ensure fair comparison, version, i.e., DOAC, by replacing the exploration process of DSAC by the behavior policy derived by OAC. We set the parammtarsese the same as used by OAC in Mujocd") and our results of DOAC on Ant-v2 and HalfCheetah-v2 are significantly better than that OAC reported.
extend OAC (Ciosek et al.) 2019 to its distributional
Thha given by the open code, where Sup is 4.66 and 5 is 23.53
17

---

Under review as a conference paper at ICLR 2022
D.3 IMPLEMENTATION
Our implementation of OVD--Explorer is based the open source code of OAC also refer the code of DSACE as well as softlearning All experiments are performed on NVIDIA GeForce RTX 2080 Ti 11GB graphics card.
The training process of OVD-Explorer and DOAC are the same as in DSAC, except for the different behavior policy used, while OVD-Explorer and DOAC enrich the experience replay with the data using the the derived exploration policies, respectively. To ensure the fair comparison, the hyper- parameters for training process of baselines and OVD-Explorer are the same. Besides, we have three hyper-parameters associated with OVD-Fxplorer mentioned before, including that controls the exploration level, / that determines the magnitude of uncertainty we use, as well as C that the normalization factor. The payper-paameeee
experiments are shown in Table 4
E MORE EXPERIMENT STUDY
E. RUNTIME ANALYSIS
Figure 6 shows the time consumption of al- gorithms relative to SAC. As can be seen, the distributional value estimation used in DSAC, DOAC and our methods introduces ex- tra time consumption distinctly, Nevertheless, the relative time consumption of OVDE G and OVDE Q SAC 1.21 and 1.17, respectively, which means that OVD-Explorer spends about 20% more time than SAC to achieve up to nearly 100% performance gain shown in Fig- ure 6(b). This demonstrates the extra time con- sumption is well worth it. Besides, the time consumption of OVDE Q close to that of DSAC, only with larger variance, which in- dicates that the additional time consumption of OVDE.Q Q is minimal while performing better exploration.
1.2
1.0
SAC
DSAC
DOAC OVDE_S OVDE_O
Figure 6: Runtime analysis. The data is from trial of each algorithm on Noisy Ant-v2 task, and the errorbar represents half of the standard devia- tion.
E.2 ANALYSIS ABOUT OVD-EXPLORER'S ADVANTAGE IN THE CASE OF GRIDCHAOS
With the heatmap of the visiting frequency of agent during exploration, and the heatmap about the uncertainty estimation, we can visually analyze the patterns and advantages of OVD- DVD-xxplore.
(a)
(b)
(c)
(d)
(c)
ff
Figure 7: State visiting frequency heatmap from 1.0 10s 2.5 x 10s steps of one trial for (a)
OVD-Explorer (b) DSAC and (c) DOAC. (d) Estimated aleatoric uncertainty of -xpplorrr,,, (e)
Epistemic-aleatoric ratio of Oxp-Explorer (f) Estimated uncertainty for exploration in DOAC.
"htpp:///itthub... http:///ihuuu..oo////aaaaaaaa hhapss//ggthhaa.oommmillllcceees
18

---

Under review as a conference paper at ICLR 2022
The distinctly different exploration patterns can be easily found. Figure ((a) 7(b) and 7 present the state visiting frequency of OVD-ExS DyD-xxplorer DSAC and DOAC, respectively. We can see that OVD-Explorer explores directly to the right half, where the environmental randomness is lower, whereas DSAC and DOAC are both stuck in the left half with higher environmental randomness.
Furthermore, show how OVD-Explorer could explore directly without being trapped by the ran- dommess through the estimated uncertainty. In specific, Figure [7dd] shows that the aleatoric uncer- tainty estimated by OVD-Explorer is consistent with environment settings, where the environment noise is higher on the left half. Figure [ shows the ratio of estimated epistemic uncertainty and aleatoric uncertainty (i.e., epistemic-aleatonn
ratio) OVD-Explore, and higher ratio means higher epistemic uncertainty or lower alcatoric uncertainty, which is exactly the direction DVD-Explow explores. The ratio is larger on the right half, which means that OVD-Explorer can avoid being stuck in the left half. Meanwhile, Figure 7( presents the estimated uncertainty in DOAC, which is larger on the left half. DOAC encourages exploring area with relatively large estimated uncertainty, explains why DOAC is stuck in the left half.
E.3 EVALUATION ON SEVERAL OTHER NOISE SCALE IN GRIDCHAOS
OVD- DVD-Explorer can explore efficiently in heteroscedastic stochastic environment by considering dif- ferently about epistemic and alcatoric uncertainty for exploration shown in Section To further empirically prove its strength, we test OVD-Explorer in GridChaos with different noise scales in four quadrants, and the result is shown as Figure and Table[5] We find that OVD-Explorer can perform well in all those different noise injection of environment.
SOM
COMC
DMAG
10: 1800 banipe maning
Nemker arraiin Baid
bur de ottranng lavo
Sumaer of Tarnarylo
Nurfaar of Taaii laech
(a)
(b)
(d)
(e)
Figure 8: Training curves on GridChaos with noise of different scale. The x-axis indicates number of training epoch (100 environment steps for each training epoch), while the y-axis is the evaluation result represented by average episode return. The shaded region denotes half standard deviation of average evaluation over 5 seeds. Curves are smoothed uniformly for visual clarity. These results are corresponding to row row Table
Table 5: The results for GridChaos. Noise setup shows the different setup for environmental het- erogeneous Gaussian noise scale, and the corresponding four columns represent the noise settings in the four quadrants of the Cartesian coordinate system, as shown in Figure 5 in which the goal locates in the first quadrant, and the triangle is initialised in the second or third quadrants. Average return shows the average episodic undiscounted return with half standard derivation in the last 100 epoch before totally 1250 epochs. FRG epoch means the minimum training epoches in the trials used to Firstly Reach the Goal before totally 1250 epochs. The row S is the standard GridChaos as shown in Figure 3(b)
the others are shown Figure
NOISE SETUP
AVERAGE RETURN
FRG EPOCH OVDE
DSAC
OVDE
DOAC
DSAC
DOAC
8
0.1
0.5
0.5
0.1
0.00+1.00
59.30 42
3.02:16.03
1250+
229
1222
18.94:137.87
58.99148.11
38.42:147.08
662
8
0.0
0.05
0.01
0.01
39.78:244872
79.52+33977
18.71437.41
694
144
846
0.05
0.1
0.05
0.05=+0.10
39.64=48.55
20.59:139.6
1250+
180
309
0.001
0.005
0.005
0.001
10.00-1-40.00
40.46-1-48.6
29.99:1-8..97
284
276
321
E
0.0
0.0
0.0
0.0
0.00
20.20 100.201.39.000 1.39.90
14.60.129.20
1250+
185
1118
Concretely, from the results, the following observations deserve to be noticed. First, OVD OVD-Explorer significantly achieve better average return in all those settings, especially when the noise is set high, and can learn to reach the goal faster (see column about FRG). It shows the ability of OVD- Explorer to guide agent explore against higher aleatoric uncertainty the left side (the second
19

---

Under review as a conference paper at ICLR 2022
and third quadrants). Second, for the task without noise shown in row E. which means the state transition is deterministic, OVD-Explorer still learns quickly. The results row E show the inherently high difficulty of this task, not only because of the very sparse reward, but also the gate leading to the goal is set very small (the width the gate is only 30% of the length of agent, i.e., the base of the isosceles triangle, which means that at the doorway the agent only move very small distance horizontally, otherwise it would be adsorbed to the wall and immobile).
E.4 EVALUATION IN GRIDCIIAOS WHEN THE ALEATORIC UNCERTAINTY IS HIGH AROUND GOAL
In the previous experiments in GridChaos, the noise (i.e., aleatoric uncertainty) near the goal is set lower. In such situation, OVD-Explorer, which follows the principle of OFU and further avoids exploring areas with higher alcatoric uncertainty, could bring significant advantage. Such setup of heterogeneous noise is reasonable, because in real life, the goal or optimal policy is always not expected to be highly stochastic.
Nevertheless, the evaluation about tasks with the existence of high randomness in the target region valuable, we conducted the following experiment in GridChaos, where the environment ran- domness in the right half (first and fourth quadrants), where the target located, was set larger. The results are shown Table Note that we use OVDE(P) denote the usual implementation that pes- simistically estimates the value distribution (i.c., using Equation 13). Besides, OVDE(M) denotes the implementation that does not pessimistically estimate the value distribution (i.e., we modify the mean of Gaussian distribution 2" in Equation 13 from the lower bound to expected value of the Q estimation p(s, a) as in Equation
Table 6: The results for GridChaos (additional). This shows row which are the cases where the optimal policy would face higher aleatoric uncertainty.
NOISE SETUP
AVERAGE RETURN
FRG EPOCH
1&44
2&:3
DSAC
OVDE(P)
OVDE(M)
DOAC
DSAC
(P)
(M)
DOAC
0.5
50.10=240.96
16.61+33.22
33.01=33..8
50.52=41.43
479
1071
583
321
G
0.1
0.05
0.00:10.00
19.84-139.66 68
39.96:148.95
30.14.139.99
-1
188
226
233
0.05
0.005
0.00+00.0
20.694.39.61
20.07-39.94
0.00=20.00
247
308
-1
0.01
0.005
0.0:00.0
40.00-t-8.99
60.00:148.99
39.99=48.98
200
236
301
0.005
0.001
20.00:1:40.00
39.981140.99
20.00:140.00
20.00:140.00
236
312
200
296
Our experimental findings are mainly the following three aspects.
Firstly, when facing extremely high aleatoric uncertainty around the goal (see row F), which causes the interaction around goal to be very unstable, chaotic and disorder, OVD-Explorer would strongly discourage exploring such area, and thus performance would be damaged. In contrast, DSAC and DOAC have no restriction on alcatoric uncertainty, and high randomness may instead increase the probability of achieving the goal.
Second, in most cases (see G, H. I, J), OVD-Explorer always can guide better exploration and achieve better performance than DSAC and DOAC, especially when the noise is negligible (see row J). This reflects the fact that our exploration objective (the mutual information shown in Equation makes great sense, achieving an appropriate trade-off between avoiding high aleatoric uncertainty and being optimistic about high epistemic uncertainty.
Third, an interesting finding is that OVD-Explorer may perform better by turning off the pessimistic estimation facing higher aleatoric uncertainty around the goal (see column OVDE(M)). This sug- gests that excessive pessimism is unnecessary there need to explore areas with high aleatoric uncertainty.
Overall, from the results in Table and Table 6 OVD-Explorer able to tackle most the cases quite well. When there is high randomness around the goal. OVD-Explorer has shortcoming that it will inevitably slow down the efficiency of reaching the goal, because it limit the exploration towards such area. Fortunately, this shortcoming can be mitigated by turning off the pessimistic estimation.
20

---

Under review as a conference paper at ICLR 2022
E.5
EVALUATION OF STATISTICAL SENSE
Table 7: Comparisons of related algorithms on Ant-v2. We report the averaged performance and standard deviation.
TAsk
EPOCH
SEED
DSAC
DOAC
OVD-EXPLORER G
OVD-EXPLORER Q
ANT-V2
2500
0. 3.
6206. 12022
1526.
7160.6:763.2
7590,3+154.9
ANT-V2
2500
9.6.7.8.9
1564.0-+1443.0
6664122:255.5
7190.1=813.8
7174.3+:50.0
ANT-22
2500
0-9
6.85.911287..2
6625.411446.8
7175.31189.00
7366.3.1666.6
To counteract the randomness from statistical perspective, conduct all experiments for 5 trails with different seeds (lypically 0-5), and report the average results with standard deviation. Next, verify that the 5 trails are sufficient to mitigate the statistical randomness, we run other 5 runs (seeds are set as 5, 6, 7, 8, 9, respectively) for those algorithms on Ant-v2, and show the results in the following. Note that the first row of results from our previously reported results, which the same as Table and the second row show the results new. The experimental results in Table Ssow that the results of 5 trials are sufficiently representative of the overall level, while the performance of OVD- D-Explorrr undoubtedly stays ahead.
E.6 STUDY ON EPISODIC HORIZON
Section 5.3 has shown great advantage of DVD-Explorer over DSAC and DOAC in stochastic Mu- joco tasks, which limits the length for an episode to 100 steps. To further empirically verify the efficiency of OVD--Explorer we test Noisy Ant-v2 task with different maximum episodic length setup. Our results in Figure 9 show that OVD-Explorer can significantly perform better than base- lines in different maximum episodic length (i.e., 250, 500, 750 and 1000). Noting that longer maxi- mum episodic length renders higher difficulty of solving tasks, especially for the high-dimensional tasks demanding exploration. specific, we have the following two conclusions.
500
1500
3000
400
1250
Aesrage Returt
2500
300
1000
2000
200
SAC
Averoge
750
DSAC
1500
DSAC
100
QVDE_G G
DOAC
500
DVDF_G
DOAC
1000
DVDF_G
DOAC
OVDE_0
250
OVDE_0
500
DVDE_g
500
2000 1500 2000
2500
2500
500 750
2000
1250
250
750
1000
1250
Number of Traininn Epoch
Number Training Epech
Number of Training Epoch
(a) 100 steps
(b) 250 steps
(c) 500 steps
5000
3000
Average Retum
2500
Average Beturn
4000
serrage Rctu
5000
2000
3000
1500-
2000
3000
DSAC
1000
DOAC
DOAC
2000
DOAC
OVDE
0-0.005
1000
OVDE
a=0.005
OVDE G a-0.001
a-0.005
a-0.005
1000
a-0.001
0
250
500
750
1000
1250
250
500
750
1000
1250
250
500
750
1000
1250
Number of Training Epoch
Mumber of Training Spoch
Number of Training Epoch
(d) 500 steps
(e) 750 steps
(f) 1000 steps
Figure 9: Training curves on Noisy Ant-v2 tasks with different maximum episodic length setup. The x-axis indicates number of training epoch (the number of environment steps for each training epoch is the same as the episodic horizon), while the y-axis is the evaluation result represented by average episode return. The shaded region denotes half standard deviation of average evaluation over seeds. Curves are smoothed uniformly for visual clarity. The sub-title of each figure represents the episodic horizon, also known as the maximum episode length.
21

---

Under review as a conference paper at ICLR 2022
Firstly, the exploration should be more conservative in the harder tasks, where we should set smaller a in OVD-Explorer. In Figure aa), (b) and (c), a is set to 0.05 by default, while we can find G decreases gradually with the increasing of the difficulty of
that the advantage of OVD--xxlore tasks. Further, if a is set to 0.005, then there a substantial performance improvement as shown in Figure(Dd.. Also, shown in Figure[e)) both OVD-Exploree methods perform well when the task episodic horizon is 750 with a set to 0.005. On the hardest task we tested, i.c., the Noisy Ant-v2 with horizon 1000 as shown in Figure (), OVD-Explorer gain remarkable performance, while set smaller as 0.001.
Secondly, DVD-Explore Q is more stable than OVD-Explorerco G. which is consistent with the con- clusion in Section 5.3 We can find from Figure 9a). (b) and (c) that DVD-Explore Q performs stably better while OVD-Explorer G degrades. OVD-Explorer 100, which is due to the Gaussian prior of noise. But when the task becomes harder, the prior helps less, and OVD-Explorer shows the advantage of more flexibly modeling aleatoric uncertainty and thus the performance more stable.
is better in easier task with horizon
E.7 STUDY ON HYPER-PARES PARAMETERS
OVD-Explorer is sensitive to a, as shown in Section 5.4 There is another hyper-parameter 3, which controls the scale of uncertainty quantification as shown in Equation [1] and Equation [3] further having an impact on 2" and Z". To evaluate its sensitivity about 8 we conduct the experiment on Noisy Ant-v2 task using OVD-Explorer G, and the result is shown in Figure [O] The results demonstrate that there is broad range of settings for B, which can lead to well performance.
Return
2
valuu
Figure 10: Sensitivity to B. The x-axis indicates different settings, while the y-axis is the evalua- tion result represented by average episode return in the last epoch before totally 1250 epoch. Error bars indicate half standard deviation of average evaluation over seeds. The 11 different values are 0.05, 0.1, 0.6, 1.2, 1.8, 2.4, 3.2, 3.6, 4.8, 6.0, 8.0.
E.8 ABLATION STUDY ON VALUE DISTRIBUTION ESTIMATION
As mentioned in Section 3.2 we estimate 2 pessimistically to alleviate over-estimation. Also, mentioned in Appendix F.4 the pessimism is unnecessary if there is need explore areas with high aleatoric uncertainty. In the following, to investigate the benefit of pessimistic estimation general case, we compare the performance of OVD-Explorer the modified versions that use normally estimated nn, Our results show that pessimistic estimation can mostly be better than that using normal estimation.
For OVDE_G (mean), we modified the mean of Gaussian distribution 2n in Equation 13 from the lower bound average value of the estimation μtss a) as in Equation 11 For OVDE.Q (mean), we modified the 2," (s, a) in Equation 14 from the minimum value to the average value, i.e., 2/ (s, a; 2, (x).
As shown in Figure II both OVDE_G (mean) and OVDE.Q (mean) perform worse than the pes- simistic version. To draw conclusion, pessimistic estimate is indeed required in general cases. Only when there is a need to explore areas with high aleatoric uncertainty, is such pessimistic esti- mation worth being turned off.
22

---

Under review as a conference paper at ICLR 2022
500
500-
Average Return
400
Average Return
400
300
200
200
OVDE _ (pessimistic)
OVDE Q (pessimistic)
100
(mean)
(mean)
500
500
Number Training Epach
Number Training Epoch
(a)
(b)
Figure 11: Training curves on Noisy Ant-v2 with different estimation of Z The x-axis indicates
number of training epoch (100 environment steps for each training epoch), while the y-axis is the evaluation result represented by average episode return. The shaded region denotes half standard deviation of average evaluation over seeds. Curves are smoothed uniformly for visual clarity.
E.9 THE PERFORMANCE COMPARED WITH RND
RND also follows OFU principle, modeling uncertainty based on network distillation and using it as an intrinsic motivation signal to facilitate agent exploration. For the sake of fairness, we implement RND based on DSAC, denoted by DSAC+RND the following, and evaluate it on standard Mujoco tasks and 3 noisy Mujoco tasks. We show the results in the following.
Table 8: Comparisons with RND. We report the averaged performance and standard deviation of runs. Each trail uses the mean undiscounted return over the last 100 epoch. The maximum value of each row is shown in bold.
EPOCH
DSAC+RND
B.D-EXPPORER.G
OYD-EXPLORER.Q
ANT-V2
2500
1206.9
4366.1
7160.64763.2
7590.3=154.9
HALFCHEETAR-N2
2500
13890.013424.4
12198.1=23388.
14594,511579.8
14792.4=997.4
HOPPER-
1250
2199.711602.7
1077.91144.1
2239.51122..2
2619.3
N-HALFCHEETAR--2
1250
431.394:35668 68
409.48-1155.88
445.28-137.52
429.63=34.45
N-HOPPER-V2
1250
244.53=44.71
231.46:99.94
252.09:7.82
237. 237.68=13.11
N-ANT-V2 (250)
1250
1275.87+172.64
1306.05-1223.18
1384.43--84.99
For the complex task Ant-v2, RND brings much greater improvement by facilitating exploration based on the DSAC. For the other simpler tasks, RND does not bring significant performance im- provement. This experiment demonstrates some extent the effectiveness of RND exploration on several Mujoco tasks, but at the same time, our algorithm OVD-Explorer is still better.
E.10 ANALYSIS ABOUT EXPLORATION PROCESS OF OVD-EXPLORED
In the following, we further verify from statistical analysis that OVD-Explorer is in compliance with our raised exploration principle, i.e., OVD-Explorer can achieve better trade-off between opti- mistic exploration and effectively avoiding exploring the areas with high alcatoric uncertainty. We show the values of uncertainty estimations and our exploration objective (mutual information) at different stages during the training processes of two trials with different noise settings in figures.
In Figure 12 the environment noise lower around the goal, noting that the darker background color in the map represents higher alcatoric uncertainty, and the red dot represents the coordinate of the current state. The performance under this trail given and the agent hardly ever reaches the goal before the 1000th epoch. Therefore, in the early stage, the aleatoric uncertainty is inaccurate and remains very low, as the value distribution shows little divergence. The figure also shows that our exploration objective (in green) is high when the epistemic uncertainty is high. So before the 1000th epoch, the exploration is guided by epistemic uncertainty, which follows the OFU principle. Later, once the goal has been explored, the aleatoric uncertainty is properly modelled, i.c., the aleatoric uncertainty towards left is larger than right at current state (see epoch 1240). Then the mutual information value towards left is lower than right, although the epistemic towards left is higher. It
23

---

Under review as a conference paper at ICLR 2022
indicates that OVD-Explorer can property balance epistemic uncertainty and aleatoric uncertainty, and effectively avoid to explore the areas with higher aleatoric uncertainty.
In Figure 13 the environment noise is set higher around the goal. At the stage before the 1000th epoch, the exploration guidance is similar to Figure The agent would hardly estimate the accurate aleatoric uncertainty without obtaining any reward. In the later stage, at the 1240th epoch, OVD- Explorer suggests exploring to the right, even though it has been recognized that the environmental uncertainty on the right is high. This is because the epistemic unertainty dominates under the mutual information. In contrast, at the 1249th epoch, when the action towards right has been explored much, the significant higher aleatoric uncertainty towards right dominates. Therefore, following the mutual information, the action towards left is preferred. This demonstrates the trade off that OVD-Ey DVD-xxllorer make, which satisfies our raised exploration principle.
24

---

Under review as a conference paper at ICLR 2022
Training epoch 20
Training epoch 400
Training epoch 800
1.0
Type
Epistemic uncertainty
0.5
Aleatoric uncertainty
Mutual information
0.0
up down left
right
up down
left right
up down
left
right
Training epoch = 1000 Training epoch 1240 Training epoch 1249
1.0
Type
Epistemic uncertainty
0.5
Aleatoric uncertainty
Mutual information
0.0
down left right
down left right
down
left
right
Average Retum
OVDE
50
0
250
500
750
1000
1250
Number of Training Epoch
Figure 12: The statistical analysis for the training process, with the aleatoric uncertainty around the goal is lower.
Training epoch = 20
Training epoch 400
Training epoch = 800
1.0
Type Epistemic uncertainty
0.5
Aleatoric uncertainty
Mutual information
0.0
up down left
right
down
left
right
down
left
right
Training epoch 1000 Training epach 1240 Training epoch 1249
1.0
Type Epistemic uncertainty
0.5
Aleatoric uncertainty
Mutual information
0.0
up down left right
up
down
left
right
down
left
right
Average Return
OVDE
60
40
20
250
500
750
1000
1250
Number Training Epoch
Figure 13: The statistical analysis for the training process, with the aleatoric uncertainty around the goal is set higher.
25


Reviews:
[{"review_id": "wn-WikliXza", "reviewer": "Reviewer_Sfg5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\nOverall, I appreciate the idea of maximizing mutual information between policy and values and the framework it builds to distinguish epistemic and aleatoric uncertainty, which I consider to be novel. The extensive experiments also show that OVD-Explorer can avoid aleatoric uncertainty while exploring epistemic uncertainty, and has good performance on exploration-demanding environments.\n\n### Weaknesses\nI apologize if I made any mistakes here. My major concern is about the technical soundness in paper's theoretical deriviation. In particular, in proof of Lemma 1, it is not clear whether $a$ and $a'$ are fixed or random. In the derivation of $\\mathbf{F}^\\pi$, it seems $a$ is randomly sampled from $\\pi(s)$ while $a'$ is another fixed action, which looks quite wired to me. From my pespective, $a$ and $a'$ should be in a symmetric situation. That is, they should be either both fixed or both random.\n\nMeanwhile, since $p(a\\mid \\bar{z}(s, a), s)$ is approximated instead of exactly evaluated, it doesn't look appropriate to claim that $p(a\\mid \\bar{z}(s, a), s)=\\frac{1}{C}\\pi(a\\mid s)\\mathbb{E}\\_{z^\\pi(s, a)\\sim Z^\\pi(s, a)}\\left[\\mathbf{1}_{z^\\pi(s, a)\\leq\\bar{z}(s, a)}\\right]$.\n\nFurthermore, the writing of this paper may also need some improvements. The detailed suggestions are put in later section.\n\n### Questions\n- Connection between law of total variance and two kinds of variances?\n- Do $a_0, \\dots, a_k$ defined in Eq. (8) enumerate all possible actions in $\\mathbf{A}$ or just some subset of $\\mathbf{A}$?\n- Can OVD-Explorer be applied to value-based algorithm or the case when the action set is discrete and finite? It is not very clear to me how $\\frac{\\partial \\bar{z}(s, a)}{\\partial a}$ should be computed under these two scenarios.\n\n### Suggestions on Writing\n- Based on the standard notation in information theory, it's probably better to use semicolon \";\" to separate $\\bar{Z}^\\pi(s, a_k)$ and $\\pi(s)$ when talking about mutual information.\n- Since $\\mu_{Z^\\pi}$ depends on $\\sigma_{\\mathrm{epistemic}}$, in Fig. 2(a), assuming means of $Z^\\pi$ at $a_1$ and $a_2$ are the same violates Eq. (13). That is, although the conclusion is not affected, PDF curves cannot overlap in Fig. 2(a). It's better to move the curve of $Z^\\pi(s, a_1)$ to the left a little bit.\n- The statement of Proposition 1 is confusing. Maybe you can simply state that \"the gradient of $\\hat{\\mathbf{F}}^\\pi$ with respect to action $a$ at $a=\\mu_T$ is ...\" in Proposition 1 and then give the formula for $\\mu_E$ outside the proposition, which clarifies how you obtain $\\mu_E$ by one-step gradient ascent.\n- \"Target policy\" in Algorithm 1 may be confused with target policy network $\\bar{\\phi}$. Furthermore, it may be better to put Algorithm 2 also into the main context if extra space is allowed later, which can help understand Algorithm 1 a lot.\n- It may be better to mention OVDE_G and OVDE_Q in Section 4 and explain that they differ mainly in using Eq. 13 or 14 in line 3 of Algorithm 1.", "summary_of_the_paper": "This paper proposes a new information-theoretic exploration approach, called OVD-Explorer, for deep reinforcement learning with continuous action space. The OVD-Explorer is capable of exploring epistemic uncertainty while avoiding aleatoric uncertainty. The paper also contains extensive experiments to evaluate the performance of OVD-Explorer.", "summary_of_the_review": "This paper proposes a novel explorer that can distinguish epistemic and aleatoric uncertainty. Although technically the theoretical motivation does not look very sound to me, the experiment results seem to be quite promising.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "t3AnTVGKtzD", "reviewer": "Reviewer_xZHr", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper considers the important problem of over-exploration, which is perhaps one of the main reasons for the inefficiency of existing online RL algorithms. Good empirical performance is the strong point of this paper.\n\nThe approach in this paper is based on the following intuition: \"avoid areas with high aleatoric uncertainty\". However, I am not sure this intuition is correct.\n\n**Example**\nConsider an MDP with actions 0 and 1 from the initial state. Upon taking action 0, the agent moves to a terminal state with a reward of 1. Upon taking action 1, the agent moves to any of the terminal states $s_1$, ..., $s_{10}$ all with probability 1/10. State $s_1$ gives a reward of 100 but states $s_2, ..., s_{10}$  all give zero rewards. In this example, the optimal action is 1 which receives a reward of 10 on average. In this example, the agent must explore the area with high aleatoric uncertainty *even more* to figure out that action 1 can lead to a large reward.\n\nIn the paper, the authors write\n\n> This issue, to explore overly the state-action pairs visited frequently but with high aleatoric uncertainty, is referred to as the over exploration issue.\n\nOFU principle constructs confidence bounds based on the samples and high aleatoric uncertainty does not misguide the confidence bounds and result in over exploration. For example, in tabular MDPs, Hoeffding-based exploration, adds a bonus of $\\frac{c}{\\sqrt{N(s,a)}}$ to the rewards, and the exploration bonus only depends on the number of times $(s,a)$ is visited and it does not depend on aleatoric uncertainty (which is e.g. related to the entropy of $P(s'|s,a)$).\n\nFurthermore, Bernstein-based exploration, adds a bonus of $c \\sqrt{\\frac{\\mathbb{V}_{s'|s,a}(V(s'))}{N(s,a)}}$, which is proved to be information-theoretically optimal up to log factors (see [1]). Notice from this bonus that when the variance of $V(s')$ is larger (which can be attributed to large aleatoric uncertainty), *more exploration is proved to be required*. [2] relates exploration to environmental attributes such as degree of stochasticity.\n\nIn OFU, overexploration happens when the constructed upper confidence bound is *loose*, which results in revisiting regions of the MDP that are already explored. This can happen e.g. when coefficient $c$ in the above bonuses is too large.\n\n**Questions/Comments**\n- Why is maximizing the multi-variable mutual information in (8) a good idea? \n- The exploration strategy has many components such as optimistic estimation of $\\mu_{\\bar{Z}}$ in (11), pessimistic estimation of $Z^\\pi$ in (13), etc. It is unclear which of these components are helping the empirical results.\n\n**References**\n\n[1] Zhang, Zihan, Yuan Zhou, and Xiangyang Ji. \"Almost Optimal Model-Free Reinforcement Learning via Reference-Advantage Decomposition.\" Advances in Neural Information Processing Systems 33 (2020).\n\n[2] Zanette, Andrea, and Emma Brunskill. \"Tighter problem-dependent regret bounds in reinforcement learning without domain knowledge using value function bounds.\" International Conference on Machine Learning. PMLR, 2019.", "summary_of_the_paper": "The paper is concerned with the problem of over-exploration in RL. The authors propose to capture the aleatoric uncertainty during exploration and propose a new exploration method called Optimistic Value Distribution Explorer (OVD-Explorer). OVD-Explorer is designed to explore optimistically while avoiding the areas with high aleatoric uncertainty. The authors propose a new measure for policy's exploration ability which aims at maximizing the mutual information between policy and policy return upper bounds. OVD-Explorer achieves good empirical performance, outperforming other methods.", "summary_of_the_review": "The paper presents a new exploration method and shows improved performance compared to state-of-the-art methods. However, the intuition behind the exploration strategy does not seem to be correct.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "LzkN-D9gxhj", "reviewer": "Reviewer_nxh5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper studies the over-exploration issue in RL, which is important to designing good exploration algorithms. The paper proposes an information-driven exploration algorithm OVD-Explorer that has potential to deal with the over-exploration issue. Here are some of my questions and comments.\n1. The RHS of Equation (7) depends on state $s$, so I think the LHS should also depend on $s$. Equation (7) is the key optimization problem that OVD-Explorer needs to solve, and this maximization is over the policy space which could be much more complicated than the action space. Can the authors comment on the computational complexity of OVD-Explorer? In addition, comparing to information-directed sampling (IDS) that optimizes over the action space, does OVD-Explorer have advantages in terms of computational complexity?\n2. Besides the version of IDS proposed [1], [2] also proposes the so-called value-IDS for RL. I think it is necessary to compare OVD-Explorer with other information-theoretic algorithms, especially through comprehensive numerical experiments. Indeed, both OVD-Explorer and value-IDS use Gaussian distributions to to approximate the true distributions of state-action values.\n3. Honestly speaking, I had a hard time verifying the properties of OVD-Explorer (e.g., Theorem 1, Lemma 1). In Theorem 1, when $k$ is finite, it is fine, but when $k$ is infinite, things need to be written more carefully and rigorously. Also, please elaborate more the constant $C$ and how it is related to other elements. In Lemma 1, does $\\sum_{a\\sim\\pi(a|s)}$ implies this result only holds for finite-action case? Btw, the notation $a\\sim\\pi(a|s)$ is a bit weird since both sides has $a$ there. In the proof for two-action case (i.e., $k=2$), there are actions $a$ and $a'$, but why the summation is over $a\\sim\\pi(s)$? Btw, what is the difference between $\\pi(s)$ and $\\pi(a|s)$? I am confused with the proof for two-action case and how it could be extended to infinite-action case. I hope these proofs could be more readable and verifiable.\n4. I think it is necessary to elaborate more on why OVD-Explorer has potential to deal with the over-exploration issue. I could not get from the paper the intuitions why maximizing the mutual information between policy and the upper bounds of policy's returns can make OVD-Explorer avoid exploring the areas with high aleatoric uncertainty. \n5. It would be great that some theoretical guarantee (e.g. regret bound, sample complexity) can be showed for OVD-Explorer\n6. There is no need to introduce the abbreviation OVD several times in the main text.\n\n\n[1] Nikolay Nikolov, Johannes Kirschner, Felix Berkenkamp, Andreas Krause, Information-Directed Exploration for Deep Reinforcement Learning, ICLR 2019.\n[2] Xiuyuan Lu, Benjamin Van Roy, Vikranth Dwaracherla, Morteza Ibrahimi, Ian Osband, Zheng Wen, Reinforcement Learning, Bit by Bit, 2021.", "summary_of_the_paper": "The paper proposes the so-called Optimistic Value Distribution Explorer (OVD-Explorer) to deal with over-exploration issue in reinforcement learning (RL). It is claimed that the proposed algorithm is tractable for continuous action space and can avoid exploring  the areas with high aleatoric uncertainty. The paper tests the performances of OVD-Explorer and some of its competitors on some common environments.", "summary_of_the_review": "This is an interesting paper, but it seems that some improvements are needed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "43hI7nAnZI3", "reviewer": "Reviewer_YS7S", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Pros**\n\n* The proposed OVD-Explorer exploration algorithm is able to force the agent to visit state-action pairs that are not visited frequently, while at the same time avoiding areas with high aleatoric uncertainty. In general the idea of using mutual information is quite novel and interesting. \n* A theoretical analysis is also provided verifying that the over-exploration issue can be tackled through the maximisation of such mutual information.\n* The empirical results on the noisy variants of the five MuJoCo environments show the efficiency of the OVD-Explorer compared to the other baselines. \n* In general the paper is well written. The proposed algorithm and the empirical results are presented in a clear way.\n\n**Cons - Comments**\n\n* The OVD-Explorer exploration algorithm has been tested only on five domains. The evaluation of  OVD-Explorer in more domains is necessary, e.g., DeepMind control suite.\n* The authors should make more clear the problem of aleatoric exploration. When do we encounter this kind of problem in the real world? Some real world use cases should be presented making clear the necessity of this kind of exploration algorithmic schemes. \n* How easy is for OVD-Explorer to be integrated with any other RL algorithm apart from SAC?\n* The reasoning behind the usage of mutual information for alleviating the aleatoric uncertainty is not totally clear.\n\n", "summary_of_the_paper": "This paper introduces OVD-Explorer, an efficient exploration algorithm that is able to detect and avoid areas with high aleatoric uncertainty. Actually, the proposed algorithm explores state-action pairs that have not been visited frequently (high epistemic uncertainty) and at the same time avoids areas with high aleatoric uncertainty. In order to achieve this behavior, OVD-Explorer maximises the mutual information between policy and corresponding upper bounds. Theoretical results show that the over-exploration issue can be tackled through the maximisation of such mutual information. Empirical analysis has been conducted on a toy task and at five MuJoCo environments including their stochastic variants.\n", "summary_of_the_review": "In general the contributions of this work are significant and novel. The proposed OVD-Explorer exploration algorithm seems to be efficient on the noisy variants of the five MuJoCo environments, showing that it is able to avoid areas with high aleatoric uncertainty. Apart from that a theoretical analysis is also provided. The main weakness of this work is the fact that the OVD-Explorer has been tested only on five MuJoCo environments. I would have expected the empirical analysis to have been conducted on more environments, e.g., DeepMind control suite. Last but not least, authors should make more clear the problem of aleatoric exploration by providing some real-world cases where we encounter this type of problem. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers found the work interesting but have concerns about the correctness of some of the claims in the paper. Also some reviewers would like to see more experiments and some have concerns about the theoretical results. Overall, I see the work promising but it requires a major revision and some improvements to pass the bar. I would recommend the authors to use the reviewers' comments and prepare the paper for future venues.
Decision: Reject