Year: 2020

Abstract:
The principle of optimism in the face of (aleatoric and epistemic) uncertainty has been utilized to design efficient exploration strategies for Reinforcement Learning (RL). Different from most prior work targeting at discrete action space, we propose a generally information-theoretic exploration principle called Max-Q Entropy Search (MQES) for continuous RL algorithms. MQES formulates the exploration policy to maximize the information about the globally optimal distribution of Q function, which could explore optimistically and avoid over-exploration by recognizing the epistemic and aleatoric uncertainty, respectively. To make MQES practically tractable, we firstly incorporate distributional and ensemble Q function approximations to MQES, which could formulate the epistemic and aleatoric uncertainty accordingly. Then, we introduce a constraint to stabilize the training, and solve the constrained MQES problem to derive the exploration policy in closed form. Empirical evaluations show that MQES outperforms state-of-the-art algorithms on Mujoco environments.

heading: 1 INTRODUCTION
text: In Reinforcement Learning (RL), one of the fundamental problems is exploration-exploitation dilemma, i.e., the agents explore the states with imperfect knowledge to improve future reward or instead maximize the intermediate reward at the perfectly understood states. The main obstacle of designing efficient exploration strategies is how the agents decide whether the unexplored states leading high cumulative reward or not.
Popular exploration strategies, like -greedy (Sutton & Barto, 1998) and sampling from stochastic policy (Haarnoja et al., 2018), lead to undirected exploration through additional random permutations. Recently, uncertainty of systems are introduced to guide the exploration (Kirschner & Krause, 2018; Mavrin et al., 2019; Clements et al., 2019; Ciosek et al., 2019). Basically, as Moerland et al. (2017) points out, two source of uncertainty exists in the RL system, i.e., epistemic and aleatoric uncertainty. Epistemic uncertainty is also called parametric uncertainty, which is the ambiguity of models arisen from the imperfect knowledge to the environment, and can be reduced with more data. Aleatoric uncertainty is an intrinsic variation associated with the environment, which is caused by the randomness of environment, and is not affected by the model. In the RL system, if the states are seldom visited, the epistemic uncertainty at these states are relatively large. Hence, the exploration methods should encourage exploration when epistemic uncertainty is large. Moreover, heteroscedastic aleatoric uncertainty means that different states may have difference randomness, which renders different aleatoric uncertainty. If we do not distinguish these two uncertainties and formulate them separately, we may explore the states visited frequently but with high randomness, i.e., low epistemic uncertainty and high aleatoric uncertainty, which is undesirable.
By introducing uncertainty, the exploration objectives like Thompson Sampling (TS) (Thompson, 1933; Osband et al., 2016) and Upper Confidence Bound (UCB) (Auer, 2002; Mavrin et al., 2019; Chen et al., 2017) are utilized to guide the exploration in RL. However, since the aleatoric uncertainty in the RL systems are heteroscedastic, i.e., the aleatoric uncertainty depends on states and actions and can be different, the above methods are not efficient. Hence, Nikolov et al. (2019) proposes novel exploration objective called Information-Directed Sampling (IDS) accounting for
epistemic uncertainty and heteroscedastic aleatoric uncertainty. However, these methods (Nikolov et al., 2019; Mavrin et al., 2019; Chen et al., 2017; Osband et al., 2016) can only be applied in the environment with discrete action space.
In this paper, we propose a generally information-theoretic principle called Max-Q Entropy Search (MQES) for off-policy continuous RL algorithms. Further, as an application example of MQES, we combine distributional RL with soft actor-critic method, where the epistemic and aleatoric uncertainty are formulated accordingly. Then, we incorporate MQES to Distributional Soft Actor-Critic (DSAC) (Ma et al., 2020) method, and show how MQES utilizes both uncertainty to explore. Finally, our results on Mujoco environments show that our method can substantially outperform alternative state-of-the-art algorithms.

heading: 2 RELATED WORK
text: Efficient exploration can improve the efficiency and performance of RL algorithms. With the increasing emphasis on exploration efficiency, various exploration methods have been developed. One kind of methods use intrinsic motivation to stimulate agent to explore from different perspectives, such as count-based novelty (Martin et al., 2017; Ostrovski et al., 2017; Bellemare et al., 2016; Tang et al., 2017; Fox et al., 2018), prediction error (Pathak et al., 2017), reachability (Savinov et al., 2019) and information gain on environment dynamics (Houthooft et al., 2016). Some recently proposed methods in DRL, originating from tracking uncertainty, do efficient exploration under the principle of OFU (optimism in the face of uncertainty), such as Thompson Sampling (Thompson, 1933; Osband et al., 2016), IDS (Nikolov et al., 2019; Clements et al., 2019) and other customized methods (Moerland et al., 2017; Pathak et al., 2019).
Methods for tracking uncertainty. Bootstrapped DQN (Osband et al., 2016) combines Thompson sampling with value-based algorithms in RL. It is similar to PSRL (Strens, 2000; Osband et al., 2013), and leverages the uncertainty produced by the value estimations for deep exploration. Bootstrapped DQN has become the common baseline for lots of recent works, and also the well-used approach for capturing epistemic uncertainty (Kirschner & Krause, 2018; Ciosek et al., 2019). However, this takes only epistemic uncertainty into account.
Distributional RL approximates the return distribution directly, such as Categorical DQN (C51) (Bellemare et al., 2017), QR-DQN (Dabney et al., 2018b) and IQN (Dabney et al., 2018a). Return distribution can be used to approximate aleatoric uncertainty, but those methods do not take advantage of the return distribution for exploration.
Exploration with two types of uncertainty. Traditional OFU methods either focus only on the epistemic uncertainty, or consider the two kinds of uncertainty as a whole, which can easily lead the naive solution to favor actions with higher variances. To address that, Mavrin et al. (2019) studies how to take advantage of distributions learned by distributional RL methods for efficient exploration under both kinds of uncertainty, proposing Decaying Left Truncated Variance (DLTV).
Nikolov et al. (2019) and Clements et al. (2019) propose to use Information Direct Sampling (Kirschner & Krause, 2018) for efficient exploration in RL (IDS for RL), which estimate both kinds of uncertainty and use IDS to make decision for acting with environment. We refer to the practice of uncertainty estimation in (Clements et al., 2019) as shown in Sec. 4.2.1. IDS integrates both uncertainty and has made progress on the issue of exploration, but this is limited on discrete action space. We do focus on how best to exploit both uncertainty for efficient exploration in a continuous action space in our paper.
Optimistic Actor Critic. More closely related to our work is the paper of OAC (Ciosek et al., 2019), which uses epistemic uncertainty to build the upper bound of Q estimation QUB. OAC is based on Soft Actor-Critic (SAC) (Haarnoja et al., 2018), additionally proposing exploration bonus to facilitate exploration. Despite the advantages that OAC has achieved over SAC, it does not consider the potential impact of the aleatoric uncertainty, which may cause misleading for exploration.

heading: 3 PRELIMINARIES

heading: 3.1 DISTRIBUTIONAL RL
text: Distributional RL methods study distributions rather than point estimates, which introduce aleatoric uncertainty from distributional perspective. There are different approaches to represent distribution in RL. In our paper, we focus on quantile regression used in QR-DQN (Dabney et al., 2018b), where the randomness of state-action value is represented by the quantile random variable Z with value z. Z maps the state-action pair to a uniform probability distribution supported on zi, where zi indicates the value of the corresponding quantile estimates. If τi is defined as the quantile fraction, the cumulative probabilities of such quantile distribution is denoted by FZ(zi) = Pr(Z < zi) = τi = 1/N for i ∈ 1, ..., N . Similar to the Bellman operator in the traditional Q-Learning (Watkins & Dayan, 1992), the distributional Bellman operator T πD under policy π is given as:
T πDZ(st, at) D = R(st, at) + γZ(st+1, at+1), at+1 ∼ π(·|st+1). (1)
Notice that this operates on random variables, D= denotes that distributions on both sides have equal probability laws. Based on the distributional Bellman operator, Dabney et al. (2018b) proposes QR-DQN to train quantile estimations via the quantile regression loss, which is denoted as:
LQR(θ) = 1
N N∑ i=1 N∑ j=1 [ρτ̂i(δi,j)] (2)
where δi,j = R(st, at) + γzj(st+1, at+1; θ)− zi(st, at; θ), ρτ (u) = u ∗ (τ − 1u<0), and τ̂i means the quantile midpoints, which is defined as τ̂i = τi+1+τi 2 .

heading: 3.2 DISTRIBUTIONAL SOFT ACTOR-CRITIC METHODS
text: Following Ma et al. (2020), Distributional RL has been successfully integrated with soft ActorCritic (SAC) algorithm. Here, considering the maximum entropy RL, the distributional soft Bellman operator T πDS is defined as follows:
T πDSZ(st, at) D = R(st, at) + γ[Z(st+1, at+1)− αlogπ(at+1|st+1)] (3)
where at+1 ∼ π(·|st+1), st+1 ∼ P(·|st, at). The quantile regression loss in DSAC is different from original QR-DQN only on the δi,j by considering the maximum entropy RL framework. DSAC extends the clipped double Q-Learning proposed on TD3 (Fujimoto et al., 2018) to overcome the overestimation problem. Two Quantile Regression Deep Q Networks have the same structure that are parameterized by θk, k = 1, 2. Following the clipped double Q-Learning, the TD-error of DSAC is defined as:
yti = min k=1,2 zi(st+1, at+1; θ̄k) (4)
δki,j = R(st, at) + γ[y t i − α log π(at+1|st+1; φ̄)]− zj(st, at; θk) (5)
where θ̄ and φ̄ represents their target networks respectively. DSAC has the modified version of critic, while the update of actors is unaffected. It is worth noticing that the state-action value is the minimum value of the expectation on certain distributions, as
Q(st, at; θ) = min k=1,2 Q(st, at; θk)
= 1
N min k=1,2 N−1∑ i=0 zi(st, at; θk) (6)
Thus, in DSAC, the original problem aims to maximize the following objective function:
Jπ(φ) = Est∼D, ∼N [logπ(f(st, t;φ)|st)−Q(st, f(st, t;φ); θ)], (7)
where D is the replay buffer, f(st, t;φ) means sampling action with re-parameterized policy.

heading: 4 ALGORITHM
text: This paper proposes a new exploration principle for continuous RL algorithms, i.e., MQES, which leverages epistemic and aleatoric uncertainties to explore optimistically and avoid over-exploration. To make MQES practically tractable, distributional and ensembleQ function approximations are introduced to formulate the epistemic and aleatoric uncertainty accordingly. Nevertheless, a constraint is introduced in the MQES to stabilize the training, and the approximated exploration policy is derived in the closed form. All these mechanisms are detailed in the following sections accordingly.

heading: 4.1 EXPLORATION STRATEGY: MAX-Q ENTROPY SEARCH
text: To achieve a better exploration, MQES derives an exploration policy πE which aims at reducing the epistemic uncertainty and obtain more knowledge of the globally optimal Q function.
Firstly, we define exploration action random variable AE ∼ πE(a|s) with value aE ∈ A, where A is the action space. Z∗(s, a∗) is the random variable following the distribution describing the randomness of return obtained by globally optimal policy π∗, and the value of Z∗(s, a∗) is defined as z∗(s, a∗). Through maximizing the mutual information between random variables Z∗(s, a∗) and AE , we reduce the uncertainty of globally optimal Q function Q∗ to encourage exploration.
Specifically, at timestamp t, we find the exploration policy πE in the candidate distribution family Π that can maximize the information about the optimal action random variable A∗ as follows:
πE = arg max π∈Π
Fπ(st), (8)
where F(·) is the mutual information and can be written as follows:
Fπ(st) = MI(Z ∗(s, a∗), A|s = st)
= H [π(at|st)]−H [p(at|z∗(st, a∗), st)] . (9)
Here MI(·) and H(·) denote the mutual information and entropy of the random variable, respectively. To obtain exploration policy πE , we need to measure the posterior probability p(·) in the above equation. For simplicity, we omit the timestamp t in the following.
To measure the posterior probability p(a|a∗, s), we propose the following proposition. Proposition 1. Generally, the posterior probability is estimated as follows:
p(a|z∗(s, a∗), s) ∝ π(a|s)ΦZπ(s,a)(z∗(s, a∗)), (10)
where Φx is the cumulative distribution function (CDF) of x, Z∗ and ZπE are the random variables, whose distributions describing the randomness of the returns obtained by optimal policy π∗ and exploration policy πE , respectively, and z∗ is the value of random variable Z∗. (see proof in Appendix A).
To measure the intractable distribution of Z∗ during training, we use the Ẑ∗ for approximation (i.e., Ẑ∗ ≈ Z∗), which will be defined later. In general, Ẑ∗ is referred to as the optimistic approximator (Mavrin et al., 2019; Chen et al., 2017), and can be formulated using the uncertainty, which will be detailed in Sec. 4.2.1.
Therefore, the Fπ(s) in Eq. 9 can be estimated as follows:
Fπ(s) ≈ F̂π(s) = Eπ,Ẑ∗ [log π(a|s)(G(s, a)− 1) +G(s, a) logG(s, a)] , (11)
where G(s, a) = 1C ∗ΦZπ(s,a)(ẑ ∗(s, a)). Specifically, G(s, a) measures the difference between Zπ and Ẑ∗, i.e., large value of CDF means that ẑ∗ is much bigger than the mean of Zπ .
By introducing distributional value functions in Eq. 10 to estimate the posterior probability, we can use the uncertainty of value function to encourage the exploration, which will be discuss in Sec. 4.2.2.

heading: 4.2 MQES-BASED EXPLORATION FOR MODERN RL ALGORITHMS
text: In this section, we propose a scheme to incorporate exploration policy derived from MQES to existing policy-based algorithms, e.g., SAC and TD3, which renders the stable and well-performed algorithm with more efficient exploration.
First, to obtain exploration policy, we employ a constraint to ensure the difference between the exploration and target policies within a certain range (i.e., KL(π||πT ) ≤ α). The target policy πT here, we mean the policy learned by any existing policy-based algorithms. It is worth noting that MQES introduces distributional and ensemble critics to the existing framework (e.g., introducing distributional critic to SAC formulates DSAC). Moreover, we utilize the critic of target policy to formulate Ẑ∗ and ZπE , which will be stated later.
Intuitively, introducing the constraint in MQES ensures that the critic of target policy guides the exploration properly and stabilizing the training. Otherwise, the exploration could be ineffective, and the update of target policy can be dramatically bad. Specifically, if the difference between π and πT are with significant difference, ZπT could not criticize exploration policy properly, and it may explore with wrong guidance, where the experiences stored in the replay buffer are with poor quality and the update of target policy fails sequentially.
Second, after introducing the KL constraint, the MQES-based exploration for modern RL algorithms is given as follows:
πE(a|s) = arg max π F̂π(s),
s.t. KL(π||πT ) ≤ α, (12) where both the exploration πE = N (µE ,ΣE) and target policy πT = N (µT ,ΣT ) are Gaussian distributions. By expanding F̂π linearly, we solve the problem in Eq. 12 using the following proposition: Proposition 2. The MQES exploration policy πE = N (µE ,ΣE) derived from Eq. 12 is as follows:
µE = µT + √ 2α∥∥∥EẐ∗ [m ∂Ẑ∗(s,a)∂a |a=µT ]∥∥∥
ΣE
ΣEEẐ∗
[ m ∂Ẑ ∗(s, a)
∂a |a=µT
] ,ΣE = ΣT . (13)
In specific, the i-th element of vectorm ismi = log G(s,µT )√
(2π)Σii +1,G(s, µT ) =
ΦZπE (s,µT ) (Ẑ∗(s,µT ))
C ,
i ∈ {1, ..., n} and n is the action dimension (see proof in Appendix B).
It is worth noting that the expectation against EẐ∗ can be estimated by sampling, and the estimation of Eq. 13 is as follows:
µE = µT +
√ 2α
K K∑ i=1 1∥∥∥m ∂Ẑ∗i (s,a)∂a |a=µT ∥∥∥ ΣE ΣE
[ m ∂Ẑ ∗ i (s, a)
∂a |a=µT
] , (14)
4.2.1 FORMULATION OF Ẑ∗ AND ZπE
In this section, we formulate the epistemic and aleatoric uncertainty with the critic of target policy, thereby distributions of Ẑ∗ and ZπE can be estimated. The remaining parts describe how to achieve these two estimations, respectively.
Formulation of Ẑ∗. In order to formulate the distribution of estimated optimalQ value, i.e., Ẑ∗, we firstly estimate its upper confidential bound, denoted by QUB. Aligned with Clements et al. (2019), we adopt two independent distribution approximators Z(s, a; θ1) and Z(s, a; θ2) parameterized by θ1 and θ2, respectively. Then we measure the epistemic uncertainty first as follows:
σepistemic(s, a; θ) = 1
2 Ei∼U(1,N)|zi(s, a; θ1)− zi(s, a; θ2)|, (15)
where N is the number of quantiles, and zi(s, a; θ) is the value of the i-th quantile drawn from Z(s, a; θ).
Consequently, the upper confidential bound of Q-value is given leveraging the σepistemic as follows:
QUB(s, a; θ) = µZ(s, a; θ) + βσepistemic(s, a; θ), (16) where µZ(s, a; θ) = 1NΣ N i=1 1 2Σk=1,2zi(s, a; θk) is the mean estimation over quantile distributions, β determines the magnitude of uncertainty we use. QUB is commonly considered as a approximation of the optimal Q value in the existing work (Ciosek et al., 2019; Kirschner & Krause, 2018).
Moreover, as shown in (Dabney et al., 2018b), the aleatoric uncertaintly can be captured by return distribution, which can be derived in our method by considering those two quantile distributions as follows: σ2aleatoric(s, a; θ) = vari∼U(1,N)[Eθkzi(s, a; θk)]. (17) Inspired by (Wang & Jegelka, 2017), we adoptQUB and σ2aleatoric as mean and variation and formulate the Gaussian distribution Ẑ∗ as follows: Ẑ∗(s, a; θ) ∼ N (QUB(s, a; θ), σ2aleatoric(s, a; θ))1ẑ∗≥QUB , (18) where Ẑ∗ follows truncated Gaussian distribution ensuring the globally optimal constraint, i.e., E[Z∗] = Q∗ ≥ QUB . Nevertheless, since the distributions of Q functions describe the aleatoric uncertainty, we set the variance of Ẑ∗ as the aleatoric uncertainty obtained from Eq. 17.
Formulation of ZπE . Since the target for critic in the advanced algorithms, like SAC and TD3, is usually estimated pessimistically, we take the pessimistic estimation for ZπE to make MQES compatible with the existing modern RL algorithms. Here we present two modeling approaches: Gaussian and quantile distributions.
Intuitively, we assume ZπE to be the Gaussian distribution with pessimistic estimation as the mean: ZπE (s, a; θ) ∼ N (QLB(s, a; θ), σ2aleatoric(s, a; θ)), (19) where QLB(s, a; θ) = µZ(s, a; θ)− βσepistemic(s, a; θ), estimates its lower confidential bound. On the other hand, as the quantile distribution is a value distribution which naturally formulates the underlying aleatoric uncertainty, we can utilize the quantile functions to model the pessimistic quantile distribution directly, breaking the Gaussian assumption above. Specifically, we take the smaller estimates at each quantile, and then the distribution ZπE (s, a; θ) is a uniform distribution over each quantile value zπEi (s, a; θ), shch as
zπEi (s, a; θ) = min k=1,2 zi(s, a; θk). (20)
Different from the uni-modal Gaussian distribution, the quantile function is able to represent multimodal distributions, which is more flexible. As the quantile function represents the inverse function of CDF, meaning that we can easily get a general idea of the properties of this pessimistic quantile distribution. It is worth noting that we can utilize other methods to formulate ZπE , like mean estimation, i.e., E [ZπE ] = µZ(s, a; θ) and zπEi (s, a; θ) = Ek=1,2 [zi(s, a; θk)]. But, it only affects the choice of hyper-parameter β and do not affect the final performance.
Algorithm 1 Exploration policy derived from MQES Initialise: Current state st, current value distribution estimators θk, k = 1, 2, current policy network φ, target policy πT (·|st;φ) ∼ N (µT (st;φ), σT (st;φ)) Output: the MQES exploration policy πE 1: Calculate Z(st, at; θk), k = 1, 2 2: Calculate epistemic uncertainty σepistemic(st, at) according to Eq. 15 3: Calculate upper bound QUB(st, at) using σepistemic(st, at) according to Eq. 16 4: Calculate aleatoric uncertainty σ2aleatoric(st, at) according to Eq. 17 5: Construct Ẑ∗(st, at) using QUB(st, at) and σ2aleatoric(st, at) (see Eq. 18) 6: Construct ZπE (st, at) according to Eq. 19 / 20 7: Calculate µE using Zπ(st, at) and Ẑ∗(st, at) according to Eq. 14 8: return πE ∼ N (µE , σT (st;φ))
The above Alg. 1 summarizes the overall procedure of MQES, including the estimation of uncertainty (Line 2, 4) and the upper confidential bound QUB (Line 3), formulation of ZπE and Ẑ∗ (Line 5-6) and exploration policy generation using KL constraint (Line 7). The generated exploration policy can be adopted by any modern policy-based RL algorithms for an more effective exploration.

heading: 4.2.2 ANALYSIS OF MQES-BASED EXPLORATION
text: This section analytically explains how MQES encourages exploration accounting for the aleatoric and epistemic uncertainty. For simplicity, we assume that the sample number is K = 1, and Eq. 14 degrades to:
µE = µT + √ 2α∥∥∥m ∂ẑ∗(s,a)∂a |a=µT ∥∥∥
ΣE
ΣE
[ m ∂ẑ ∗(s, a)
∂a |a=µT
] , (21)
where mi = log G(s,µT )√ (2πE)Σii + 1.
Take Gaussian MQES for example, we have ZπE (s, a) ∼ N (QLB(s, a), σ2aleatoric(s, a)), and the bias term added to µE is decided by the epistemic and aleatoric uncertainty.
Specifically, since epistemic uncertainty is involved in Ẑ∗(s, a), the gradient ∂ẑ ∗(s,a) ∂a encourages the optimistic exploration. Epistemic uncertainty-based exploration, can avoid the pessimistic underexploration. The aleatoric uncertainty is introduced by CDF Φ. If we have two state-action pairs, i.e., (s1, a1) and (s2, a2), and ZπE (si, ai) ∼ N (QLB(si, ai), σ2aleatoric(si, ai)), i = 1, 2, and we assume that QLB(s1, a1) = QLB(s2, a2), σ21 > σ 2 2 , and ẑ ∗(s1, a1) = ẑ ∗(s2, a2). Obviously, ΦZπ(s1,a1)(ẑ ∗(s1, a1)) < ΦZπ(s2,a2)(ẑ
∗(s2, a2)), which means that larger aleatoric uncertainty leads to smaller action bias.
Therefore, the MQES encourages the exploration by selecting the action increasing the optimistic value function, and avoid the over-exploration by setting smaller action bias at the state, where the aleatoric uncertainty is high.

heading: 5 EXPERIMENTS
text: MQES is designed for efficient exploration in continuous action space problem in RL, allowing the agent to be aware of explore directions that may lead to higher optimistic value function with smaller aleatoric uncertainty. Comparisons between MQES and state-of-the-art algorithms are conducted to verify the MQES regarding the effectiveness and efficiency. Empirical evaluations show that MQES outperforms state-of-the-art algorithms on a series of continue control tasks.

heading: 5.1 IMPLEMENTATION DETAILS AND EXPERIMENT SETTINGS
text: We compare MQES against SAC (Haarnoja et al., 2018) and its distributional variant DSAC (Ma et al., 2020). Ma et al. (2020) also shows the performance of TD4, which is the distributional extension of TD3 (Fujimoto et al., 2018), and can also be used to capture epistemic and aleatoric uncertainty as is pointed out in Sec. 4.2. However, DSAC outperforms TD4 as shown in Ma et al. (2020), so we evaluate based only on SAC and DSAC, and further implement MQES based on DSAC in order to develop the exploration ability.
The training process of MQES is the same as in DSAC, except for the behavior policy used, while we enrich the experience replay with the data generated by πE . The pseudo code of the whole process can be found in Appendix C. In order to ensure a fair comparison, the hyper-parameters of DSAC and MQES are the same (see Appendix D). In addition, we have 3 hyper-parameters associated with MQES. The parameter √ 2α controls the exploration level, and β determines the magnitude of uncertainty we use, and C is the normalization factor.
We implement both approaches for building Zπ as illustrated in Sec. 4.2.1, and we use MQES G and MQES Q to indicate respectively to Gaussian distribution and quantile distribution. We test MQES on several tasks in Mujoco (Todorov et al., 2012), including standard version, as well as modified sparse version and stochastic version. We limit the maximum length of each episode to 100. We run 1250 or more epochs for each task where there are 100 training steps per epoch, with evaluating every epoch where each evaluation reports the average undiscounted return with no exploration noise.

heading: 5.2 STANDARD MUJOCO TASKS
text: We evaluate both MQES G and MQES Q in 5 tasks of standard Mujoco, and the result in Figure 1 and Appendix F show that our methods outperform SAC for all those tasks, and also reach better performance than DSAC.
Performance. Our results demonstrate that in complex tasks, such as Humanoid-v2 and Ant-v2, our MQES-based exploration policy performs better, while DSAC suffers from the inefficiency caused by deficient exploration. In Ant-v2, DSAC was overtaken in the early stages of training and then MQES stays ahead. Also in Humanoid-v2, the performance of our algorithm always maintains better than DSAC. In some relatively easier tasks, it seems that those tasks are not very demanding for exploration, but MQES performs still at a very advanced level. In Walker2d-v2, MQES and DSAC alternated lead until 1000 epochs, after which MQES had a significant improvement. The final results are shown more specifically in the Table 3.
Gaussian and Quantile Zπ . One can find that, expect for Humanoid-v2, there is no absolute superiority between the two modeling approaches. We hypothesis this is because environment of Humanoid-v2 is more complex than others, and more flexible quantile Zπ is needed, which could model the environment more accurately.

heading: 5.3 SPARSE MUJOCO TASKS
text: To further show the strength of our algorithm regarding exploration, we evaluate on the sparse version of Mujoco tasks. Specifically, the reward is +1 when the move-distance threshold 1 is reached, otherwise 0. Obviously, since we set the maximum length of episode to 100, the maximum average episode return in the sparse tasks is up to 100.
As shown in Figure 1 and Appendix F, SAC performs poorly in those tasks, which is to be expected, since solving the sparse reward problem requires not only more accurate estimates of critic, but also more efficient exploration. Even in sparse HumanoidStandup-v2 task, when our algorithm MQES
1We set the threshold according to the statistical analysis of untrained interaction behavior, using 99.9% quantile value, and the threshold of all environments are shown in Appendix E
reaches nearly maximum scores, SAC learns almost nothing, and DSAC performs obviously a bit inferior. As can be seen in combination with the Figure 1 and Table 3, both MQES algorithms perform the best, and both do achieve significantly better results faster than DSAC.
MQES performs extremely well in these sparse environments, which on the one hand demonstrates the importance of exploration when solving sparse reward problems, and on the other hand shows the advantages and capabilities of MQES for efficient exploration, presenting that incorporating uncertainty to exploration could render better performance.

heading: 5.4 ABLATION STUDY
text: In this section, we conduct two ablation experiments to show the performance gain by distinguishing aleatoric and epistemic uncertaity and the sensitivity to the hyper-parameters.

heading: 5.4.1 GAIN OF ALEATORIC UNCERTAINTY
text: The exploration with epistemic uncertainty is proved to be efficient (Ciosek et al., 2019) by avoiding the under-exploration. Hence, we present the gain brought by the aleatoric uncertainty by conducting ablation study in this section. Generally, MQES-based actor-critic algorithm degrades to Optimistic Actor-Critic (OAC) (Ciosek et al., 2019) if mi = 1 in Eq. 13. Consequentially, we could compare MQES with OAC to show the necessary of introducing the aleatoric uncertainty to exploration.
In order to show that MQES performs robustly in the face of aleatoric uncertainty, we modified the standard Ant-v2 task by adding heterogeneous noise to the observation to increase the aleatoric uncertainty of the environment. We also extend the OAC to the distributional form (DOAC, Distributional OAC), i.e., to estimate the values in the same way as MQES, in order to ensure a fair comparison. As shown in Figure 2, DOAC does not take into account the aleatoric uncertainty, resulting in its performance being close to that of DSAC. The experimental results confirm that MQES is effective in avoiding the interference of aleatoric uncertainty in the environment and ensuring exploration efficiency.

heading: 5.4.2 ABLATION STUDY ON HYPER-PARAMETERS
text: MQES is sensitive to √
2α, which controls the distance between the behavior policy πE and the target policy πT . If√
2α is quite small, then the MQES degenerates to DSAC and shows little exploration, and if √ 2α is larger, then the performance becomes worse because of the gap between behavior policy and target policy. We evaluate the sensitivity to hyperparameters on Ant-v2 task using MQES G, and the sensitivity to the constraint √ 2α is shown in Figure 3, and the sensitivity to β is shown in Appendix F, where the error bar indicates half standard deviation. Figure 3: Ablation study on √ 2α.

heading: 6 CONCLUSION
text: In this paper, we propose MQES, a generally exploration principle for continuous RL algorithms, which formulates the exploration policy to maximize the information about the globally optimal distribution of Q function. To make MQES practically tractable, we firstly incorporate distributional and ensemble Q function approximations to MQES, which could formulate the epistemic and aleatoric uncertainty accordingly. Secondly, we introduce a constraint to stabilize the training, and solve the constrained MQES problem to derive the exploration policy in closed form. Then, we analyze and show that it explores optimistically and avoid over-exploration by recognizing the epistemic and aleatoric uncertainty, respectively. Empirical evaluations show that MQES works better at the complex environments, where the exploration is needed.

heading: A PROOF OF PROPOSITION 1
text: Proof. The proof is similar to Wang & Jegelka (2017). Here, we utilize Zπ to criticize π, i.e.,
p(a|z∗(s, a∗), s) = π(a|s)Ezπ∼Zπ
[ 1zπ(s,a)≤z∗(s,a∗) ] C ,
where C is the normalization factor, and EZπ(s,a) [ 1zπ(s,a)≤z∗(s,a∗) ] = ΦZπ(s,a)(z ∗(s, a∗))

heading: B PROOF OF PROPOSITION 2
text: Proof. Inspired by the methods of multipliers, it is equivalent to maximize objective function F̂π(s, a) and minimize constraint KL(π||πT ) simultaneously. Firstly, we minimize the KL(π||πT ) = KL(N (µ,Σ)||N (µT ,ΣT )) against the variance matrix Σ, i.e.,
min Σ
KL(N (µ,Σ)||N (µT ,ΣT )). (22)
The optimal solution for problem (22) is
ΣE = ΣT . (23)
Then, F̂π(s, a) is expanded linearly around at = µT :
F̂π(s, a) ≈ aT∇aF̂π(s, a)|a=µT + const (24) = aT [ m̂ ∂ẑ ∗(s, a)
∂a |a=µT
] + const, (25)
where the element of vector m̂ = {m̂i}ni=1 is
m̂i = 1
C φZπ(s,µT )(ẑ
∗(s, µT ))(log ΦZπ(s,µT )(ẑ
∗(s, µT )) C √ (2π)Σii + 1), (26)
n is the dimension of action, and φ(x) is the probability distribution function (pdf). Then, problem (12) is reformulated as:
max µ
EẐ∗ { µT [ m ∂ẑ ∗(s, a)
∂a |a=µT ]} s.t. 1
2 (µ− µT )TΣ−1(µ− µT ) ≤ α, (27)
Then, the Lagrange function of problem (27) is given as: L(µ) = Eẑ∗ { µT [ m̂ ∂ẑ ∗(s, a)
∂a |a=µT
]} − λ [ 1
2 (µ− µT )TΣ−1(µ− µT )− α
] . (28)
According to the KKT condition, we derive the following equations: First, from the stationary, i.e., ∇µL(µ) = m ∂Ẑ ∗(s,a) ∂a |a=µT − λΣ −1(µ− µT ) = 0, we get
µ = µT + 1
λ ΣEẐ∗
[ m̂ ∂ẑ ∗(s, a)
∂a |a=µT
] , (29)
where λ > 0. Secondly, since λ > 0, the constraint is active, i.e., (µ − µT )TΣ−1(µ − µT ) = 2α. Together with Eq. 29, we get the following equation:
λ = ∥∥∥Eẑ∗ [m̂ ∂Ẑ∗(s,a)∂a |a=µT ]∥∥∥ Σ√
2α . (30)
Finally, (13) is obtained by plugging (30) to (29).

heading: C ALGORITHM 2: MQES FOR DSAC
text: In this section, we show the whole algorithm of our implementation of MQES based on DSAC in Alg. 2.
Algorithm 2 MQES for DSAC Initialise: Value networks θ1, θ2, policy network φ and their target networks θ̄1, θ̄2, φ̄, quantiles number N, target smoothing coefficient (τ ), discount (γ), an empty replay pool D 1: for each iteration do 2: for each environmental step do 3: at ∼ πE(at, st) according to Alg. 1 4: D ← D ∪ {(st, at, r(st, at), st+1)} 5: end for 6: for each training step do 7: for i = 1 to N do 8: for j = 1 to N do 9: calculate δki,j , k = 1, 2, following Eq. 5 10: end for 11: end for 12: Calculate LQR(θk), k = 1, 2 using δki,j following Eq. 2 13: Update θk with∇LQR(θk) 14: Calculate Jπ(φ), following Eq. 7 15: Update φ with∇Jπ(φ) 16: end for 17: Update target value network with θ̄k ← τθk + (1− τ)θ̄k, k = 1, 2 18: Update target policy network with φ̄← τφ+ (1− τ)φ̄ 19: end for

heading: D HYPER-PARAMETERS SETTING
text: The hyper-parameters in our experiment are guaranteed to be consistent, as shown in Tab. 1.

heading: E THRESHOLD SETTINGS FOR SPARSE TASKS
text: As illustrated in Sec. 5.3, We set the threshold according to the statistical analysis of untrained interaction behavior, using 99.9 quantile value, as shown in Tab. 2.

heading: F ADDITIONAL EXPERIMENT RESULTS
text: Limited by the length of the text, we present the results of the evaluation on some simple tasks in Fig. 5. All evaluation data can be seen in Tab. 3.
The sensitivity to β is shown in Fig. 4. β controls the uncertainty magnitude, the smaller the value, the smaller the degree of optimism or pessimism, and vice versa. We evaluate the effect of β in Antv2 task using MQES Q. What can be seen is that larger β will degrade performance. Although smaller β are more profitable in the Ant-v2 task, it is not necessarily in other task, so we set a uniform 1.6 in our experiments.

Review summaries:
This work introduces max-Q Entropy Search (MQES) exploration principle  for continuous RL algorithms. MQES addresses the exploration-exploitation dilemma that constitutes a fundamental RL problem. Actually, MQES defines an exploration policy able to explore optimistically and avoid over-exploration. One of the main advantages of MQES is its ability to recognise the epistemic and aleatoric uncertainty. Empirical analysis has been conducted on Mujoco, showing that the performance of MQES is comparable to  those of other state-of-the-art algorithms.

This paper proposes MQES, a Max-Q entropy search for policy optimization in continuous RL. The authors propose to combine advantages of the information-theoretic principle and distributional RL, in which epistemic and aleatoric uncertainty are estimated using similar entropy-search acquisition functions in the Bayesian Optimization (BO). As said, this is a new method to introduce a more efficient exploration strategy. As a result, policy improvement is formulated as a constraint optimization problem where a next exploration policy can be solved in a closed-form. The proposed method is evaluated on Mujoco tasks and compared against other off-policy approaches, SAC, and DSAC. The results show MQES outperforms other methods in domains where exploration is needed.