MQES: Max-Q Entropy Search for Efficient Exploration in Continuous Reinforcement Learning
Authors: Jinyi Liu
Zhi Wang
Jianye HAO
YAN ZHENG
Venue:  (2021)

Abstract: The principle of optimism in the face of (aleatoric and epistemic) uncertainty has been utilized to design efficient exploration strategies for Reinforcement Learning (RL). Different from most prior work targeting at discrete action space, we propose a generally information-theoretic exploration principle called Max-Q Entropy Search (MQES) for continuous RL algorithms.
MQES formulates the exploration policy to maximize the information about the globally optimal distribution of $Q$ function, which could explore optimistically and avoid over-exploration by recognizing the epistemic and aleatoric uncertainty, respectively. To make MQES practically tractable, we firstly incorporate distributional and ensemble $Q$ function approximations to MQES, which could formulate the epistemic and aleatoric uncertainty accordingly. Then, we introduce a constraint to stabilize the training and solve the constrained MQES problem to derive the exploration policy in closed form. Empirical evaluations show that MQES outperforms state-of-the-art algorithms on Mujoco environments.

Under review as a conference paper at ICLR 2021
MQES: MAX-Q ENTROPY SEARCH FOR EFFICIENT EXPLORATION IN CONTINUOUS REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
The principle of optimism in the face of (aleatoric and epistemic) uncertainty has been utilized to design efficient exploration strategies for Reinforcement Learn- ing (RL). Different from most prior work targeting at discrete action space, we propose generally informatioon-heorrete cxploration principle called Max-Q En- tropy Search (MQES) for continuous RL algorithms. MQES formulates the explo- ration policy maximize the information about the globally optimal distribution of Q function, which could explore optimistically and avoid over-exploration by recognizing the epistemic and aleatoric uncertainty, respectively. To make MQES practically tractable, we firstly incorporate distributional and ensemble Q func- tion approximations to MQES, which could formulate the epistemic and aleatoric uncertainty accordingly. Then, we introduce constraint to stabilize the train- ing, and solve the constrained MQES problem to derive the exploration policy closed form. Empirical evaluations show that MQES outperforms state-of-the-art algorithms on Mujoco environments.
INTRODUCTION
In Reinforcement Learning (RL), one of the fundamental problems is ppplorationexxplitattii dilemma, i.c., the agents explore the states with imperfect knowledge to improve future reward or instead maximize the intermediate reward at the perfectly understood states. The main obstacle of designing efficient exploration strategies how the agents decide whether the unexplored states leading high cumulative reward or not.
Popular exploration strategies, like c-greedy (Sutton Barto, 1998) and sampling from stochastic policy (Haarnoja et al., 2018), lead to undirected exploration through additional random permuta- tions. Recently, uncertainty of systems are introduced to guide the exploration (Kirschner Krause, 2018: Mavrin et al., 2019; Clements et al., 2019; Ciosek et al.. 2019). Basically, Moerland et al. (2017) points out, two source of uncertainty exists in the RL system, i.c., epistemic and alcatoric uncertainty. Epistemic uncertainty is also called parametric uncertainty, which is the ambiguity of models arisen from the imperfect knowledge to the environment, and can be reduced with more data. Aleatoric uncertainty an intrinsic variation associated with the environment, which is caused by the randomness of environment, and is not affected by the model. In the RL system, if the states are seldom visited, the epistemic uncertainty these states relatively large. Hence, the exploration methods should encourage exploration when epistemic uncertainty is large. Moreover, heteroscedastic aleatoric uncertainty means that different states may have difference randomness, which renders different aleatoric uncertainty. If we do not distinguish these two uncertainties and formulate them separately, we may explore the states visited frequently but with high randomness, ie., low epistemic uncertainty and high alcatoric uncertainty, which is undesirable.
By introducing uncertainty, the exploration objectives like Thompson Sampling (TS) (Thompson, 1933; Osband et al., 2016) and Upper Confidence Bound (UCB) (Auer, 2002; Mavrin et al., 2019; Chen et al., 2017) are utilized to guide the exploration in RL. However, since the aleatoric uncer- tainty in the RL systems are heteroscedastic, i.e., the aleatoric uncertainty depends states and actions and can be different, the above methods are not efficient. Hence, Nikolov et al. (2019) proposes novel exploration objective called Information-Directed Sampling (IDS) accounting for

---

Under review as a conference paper at ICLR 2021
epistemic uncertainty and heteroscedastic aleatoric uncertainty. However, these methods (Nikolov et al., 2019; Mavrin et al., 2019; Chen et al., 2017: Osband et al., 2016) can only be applied in the environment with discrete action space.
In this paper. we propose a generally information-theoreeti principle called Max-Q Q Entropy Search (MQES) for off-policy continuous RL. algorithms. Further, as application example of MQES, combine distributional RL with soft actor-critic method, where the epistemic and aleatoric uncer- tainty are formulated accordingly. Then, we incorporate MQES Distributional Soft Actor-Critic (DSAC) (Ma et al., 2020) method, and show how MQES utilizes both uncertainty to explore. Finally, our results on Mujoco environments show that our method can substantially outperform alternative state-of-the-an algorithms.
RELATED WORK
Efficient exploration improve the efficiency and performance of RL algorithms. With the in- creasing emphasis on exploration efficiency, various exploration methods have been developed. One kind of methods use intrinsic motivation stimulate agent to explore from different perspectives, such as count-based novelty (Martin et al., 2017; Ostrovski et al., 2017; Bellemare et al., 2016; Tang al.. 2017: Fox al., 2018), prediction error (Pathak et al., 2017), reachability (Savinov et al., 2019) and information gain on environment dynamics (Houthooft et al., 2016). Some recently pro- posed methods in DRL, originating from tracking uncertainty, do efficient exploration under the principle of OFU (optimism in the face of uncertainty), such as Thompson Sampling (Thompson, 1933; Osband et al., 2016), IDS (Nikolov et al., 2019; Clements al., 2019) and other customized methods (Moerland et al., 2017; Pathak et al., 2019).
Methods for tracking uncertainty. Bootstrapped DQN (Osband et al., 2016) combines Thompson sampling with value-based algorithms in RL. It is similar to PSRL (Strens, 2000; Osband et al., 2013), and leverages the uncertainty produced by the value estimations for deep exploration. Boot- strapped DQN has become the common baseline for lots of recent works, and also the well-used approach for capturing epistemic uncertainty (Kirschner & Krause, 2018; Ciosek et al., 2019). How- ever, this takes only epistemic uncertainty into account.
Distributional RL approximates the return distribution directly, such as Categorical DQN (C51) (Bellemare et al., 2017). QR-DQN (Dabney et al., 2018b) and IQN (Dabney et al., 2018a). Re- turn distribution can be used to approximate aleatoric uncertainty, but those methods do not take advantage of the return distribution for exploration.
Exploration with two types of uncertainty. Traditional OFU methods either focus only on the epistemic uncertainty, consider the two kinds of uncertainty whole, which can easily lead the naive solution to favor actions with higher variances. To address that, Mavrin et al. (2019) studies how to take advantage of distributions learned by distributional RL methods for efficient exploration under both kinds of uncertainty, proposing Decaying Left Truncated Variance (DLTV).
Nikolov al. (2019) and Clements al. (2019) propose use Information Direct Sampling (Kirschner & Krause, 2018) 1) for efficient exploration in RL (IDS for RL), which estimate both kinds of uncertainty and use IDS make decision for acting with environment. We refer to the practice of uncertainty estimation in (Clements et al., 2019) as shown in Sec. 4.2.1. IDS integrates both uncertainty and has made progress on the issue of exploration, but this limited on discrete action space. We do focus on how best to exploit both uncertainty for efficient exploration in continuous action space in our paper.
Optimistic Actor Critic. More closely related to our work is the paper of OAC (Ciosek et al., 2019), which uses epistemic uncertainty to build the upper bound of Q estimation QUR OAC is based on Soft Actor-Critic (SAC) (Haarnoja et al., 2018), additionally proposing exploration bonus to facilitate exploration. Despite the advantages that OAC has achieved over SAC, it does not consider the potential impact of the aleatoric uncertainty, which may cause misleading for exploration.
2

---

Under review as a conference paper at ICLR 2021
3 PRELIMINARIES
3.1 DISTRIBUTIONAL RL
Distributional RL methods study distributions rather than point estimates, which introduce aleatoric uncertainty from distributional perspective. There are different approaches to represent distribution in RL. In our paper, we focus on quantile regression used in QR-DQN (Dabney et al., 201 where the randomness of state-action value is represented by the quantile random variable Z with value 2. Z maps the aate-action pair to a uniform probability distribution supported on 24, where 2; indicates the value of the corresponding quantile estimates. If is defined as the quantile fraction, the cumulative probabilities of such quantile distribution is denoted by Fz(z;) = Pr(Z < 2;) 1/N for i 1,..,,N N.
Similar to the Bellman operator in the traditional Q-Learning (Watkins & Dayan, 1992), the distri- butional Bellman operator TD under policy æ is given as:
( a,) R(st, az) 7 (5411, at+1), a++1 ~ +(188++1)-
Notice that this operates on random variables, denotes that distributions on both sides have equal probability laws. Based the distributional Bellman operator, Dabney et al. (201 8b) proposes QR-DQN to train quantile estimations via the quantile regression loss, which is denoted as:
N
N
1 N
LQ
pe, (dt.))
1=1
where ōg, = R(ss,a)) + (2y a(+1: 2, (1t, ar; a,; (2) u ====(== lucu), and f means the quantile midpoints, which is defined as f; = 1u
2
3.2 DISTRIBUTIONAL SOFT ACTOR-CRITIE METHODS
Following Ma et al. (2020), Distributional RL has been successfully integrated with soft Actor- Critic (SAC) algorithm. Here, considering the maximum entropy RI,, the distributional soft Bellman operator Tōs is defined as follows:
TD (st, uz) 2 R(sr, ar) + 7 7Z 1(4+1 (
(3)
where ac+1 ~ T( (-|5))1), s1+1 P(-ls(,a;) The quantile regression loss in DSAC is different from original QR-DQN only on the ōi, by considering the maximum entropy RL framework. DSAC extends the clipped double Q-Learning proposed on TD3 (Fujimoto et al., 2018) to overcome the overestimation problem. Two Quantile Regression Deep Q Networks have the same structure that are parameterized by Oz, k = 1, Following the clipped double Q-Learning, the TD-error of DSAC defined as:
v min 0(41;00) k=1.2
(4)
8i = R(s a:) + rlvl a log ग (a141 x)] 22 (s; 0%)
(5)
where 0 and & represents their target networks respectively. DSAC has the modified version of critic, while the update of actors is unaffected. It is worth noticing that the state-action value is the minimum value of the expectation on certain distributions, as
Q(st. ac: a6:
min Q(si, ag: k=1.2
N
(6)
Σ
zi(Se, ar; 0z)
Thus, in DSAC, the original problem aims to maximize the following objective function:
Ix()) E.. eri ())s:) Q(s [ ex; 0);
where D is the replay buffer, f(st, ((; x) means sampling action with re-parameterized policy.
3

---

Under review as a conference paper at ICLR 2021
4 ALGORITHM
This paper proposes a new exploration principle for continuous RL algorithms, ie., MQES, which leverages epistemic and aleatoric uncertainties to explore optimistically and avoid ever-exploration. To make MQES practically tractable, distributional and ensemble Q function approximations are troduced to formulate the epistemic and aleatoric uncertainty accordingly. Nevertheless, constraint introduced in the MQES stabilize the training, and the approximated exploration policy is de- rived in the closed form. All these mechanisms are detailed in the following sections accordingly.
4.1 EXPLORATION STRATEGY: MAX-Q ENTROPY SEARCH
To achieve better exploration, MQES derives an exploration policy Te which aims reducing the epistemic uncertainty and obtain more knowledge of the globally optimal Q function.
Firstly, define exploration action random variable Ag Tg(aas) with value ag A, where A is the action space, (*,s, a²) is the random variable following the distribution describing the randomness of return obtained by globally optimal policy x and the value of Z (s, (s,aa) is defined as 2" (s, a²). Through maximizing the mutual information between random variables Z* a") and AF, we reduce the uncertainty of globally optimal Q function Q' to encourage exploration.
Specifically, at timestamp I, we find the exploration policy in the candidate distribution family II that can maximize the information about the optimal action random variable A" follows:
T arg max F (8:),
well
where F(-) is the mutual information and can be written as follows:
F (8t) = MI (s, aa),a)),))) ), Als = st)
H (a, (x)] H [p(arl=" (st, aa)),))]]
Here MI(-) and H(.) denote the mutual information and entropy of the random variable, respec- tively. To obtain exploration policy TE, we need to measure the posterior probability p(-) in the above equation. For simplicity, we omit the timestamp t in the following.
To measure the posterior probability p(ala", s). propose the following proposition.
Proposition 1. Generally, the posterior probability is estimated as follows:
p(al2" (s,"""(s,"";)) s)
(10)
where 42 is the cumulative distribution function (CDF) Z" and = are the random variables, whose distributions describing the randonness of the returns obtained by optimal policy n² and ex- ploration policy n., respectively, and z" is the value of random variable 7', (see proof in Appendix A).
To measure the intractable distribution of Z* during training, we use the 2 for approximation (i.e., 2" 77), which will be defined later. In general, 2* referred to as the optimistic approximator (Mavrin et al., 2019; Chen et al., 2017), and can be formulated using the uncertainty, which will be detailed Sec. 4.2.1.
Therefore, the F F*(8) in Eq. 9 can be estimated as follows:
(8) FdT (s)
n,2 [log (G))) 1)
G(s,)))
(11)
where G(s, a) c & (s.a) (2 (s, (2 a)). Specifically, G(s, a) measures the difference between Z and 2", i.e., large value of CDF means that 3* is much bigger than the mean of n
By introducing distributional value functions in Eq. 10 to estimate the posterior probability, can use the uncertainty of value function to encourage the exploration, which will be discuss Sec. 4.2.2.
4

---

Under review as a conference paper at ICLR 2021
4.2 MQES-BAR BASED EXPLORATION FOR MODERN RL ALGORITTMM
In this section, we propose a scheme to incorporate exploration policy derived from MQES to ex- isting policy-based algorithms, e.g., SAC and TD3, which renders the stable and well-performed algorithm with more efficient exploration.
First, to obtain exploration policy, we employ a constraint to ensure the difference between the exploration and target policies within certain range (i.e., (L(=][TT) S a). The target policy тт here, we mean the policy learned by any existing policy-based algorithms. It is worth noting that MQES introduces distributional and ensemble crities to the existing framework (e.g., introducing distributional critic SAC formulates DSAC). Moreover, we utilize the critic of target policy to formulate 7" and e which will stated later.
Intuitively, introducing the constraint in MQES ensures that the critic of target policy guides the exploration properly and stabilizing the training. Otherwise, the exploration could be ineffective, and the update of target policy can be dramatically bad. Specifically, if the difference between and Ty are with significant difference, z could not criticize exploration policy properly, and may explore with wrong guidance, where the experiences stored in the replay buffer are with poor quality and the update of target policy fails sequentially.
Second, after introducing the KL constraint, the IQES-based exploration for modern RL algorithms given as follows:
TE (als) = arg max (*)
s.t.
KL(T||T) S
(12)
where both the exploration TE NNE, Eg) and target policy गт Nur, Σr) are Gaussian distributions. By expanding F linearly, we solve the problem in Eq. 12 using the following propo- sition: Proposition The MQES exploration policy aE NNEE,, Ex) derived from Eq. 12 as follows:
2a
(8, a)
PE pT
Ezg
m
(s.a)
du
le=pT
EE
Eg 2.
aa
la=pr
Eg Ex. (13)
In specific, the i-th element of vector m is m, log G(S,ur) +1, G(s,
2 Bx (2 (2 (sspT))
v ((2T)Eu
i nI. and n is the action dimension (see proof in Appendix B).
is worth noting that the expectation against Ez. can be estimated by sampling, and the estimation of Eq. 13 is follows:
2a
K
1
aZ; a)
#E = PT
K
(s,4)
Σ
m O
Oa
1=
(14)
m71
Da
a==
Eg
4.2.1 FORMULATION OF &" AND Z
In this section, we formulate the epistemic and aleatoric uncertainty with the critic of target policy, thereby distributions of 2" and n can be estimated. The remaining parts describe how to achieve these two estimations, respectively.
Formulation of 2. In order to formulate the distribution of estimated optimal Q value, i.c., 2", firstly estimate its upper confidential bound, denoted by QUB Aligned with Clements et al. (2019), adopt two independent distribution approximators (ss, a; 01) and ((s, a; 62) parameterized by 01 and 62, respectively. Then we measure the epistemic uncertainty first as follows:
1 Jepistemic (s, 0) E 01) 2, 02)1.
(15)
2
where N is the number of quantiles, and 2; a; 8) is the value of the i-th quantile drawn from 7(8, az 0).
5

---

Under review as a conference paper at ICLR 2021
Consequently, the upper confidential bound of Q-value is given leveraging the dppistemiic follows: QU (s. a;;) 0) - (z 8) + 30 epistemic a; 8),
(16)
where (s, 0) = 1 - 1 Ex=1,22; a; 0k) is the mean estimation over quantile distributions, 3 determines the magnitude of uncertainty we use. Q is commonly considered as approximation of the optimal value the existing work (Ciosek et al., 2019; Kirschner & Krause, 2018).
Moreover, as shown in (Dabney et al., 2018b), the aleatoric uncertaintly can be captured by return distribution, which can be derived our method by considering those two quantile distributions as follows:
aemmoric (s, a; f) vati (En, 2, a; 0x)].
(17)
Inspired by (Wang & Jegelka, 2017), we adopt QUB and o aleatoric as mean and variation and formulate the Gaussian distribution 2* follows:
as,,,a;)) ~ (s)) o aeeatoric (8, a; 2-qqUB
(18)
where 2" follows truncated Gaussian distribution ensuring the globally optimal constraint, i.e., B[Z*] = Q 2 QUA Nevertheless, since the distributions of Q funetions describe the aleatoric uncertainty, we set the variance of 2 as the alcatoric uncertainty obtained from Eq. 17.
Formulation of z** Since the target for critic in the advanced algorithms, like SAC and TD3, usually estimated pessimistically, we take the pessimistic estimation for 2n to make MQES compatible with the existing modern RL algorithms. Here we present two modeling approaches: Gaussian and quantile distributions.
Intuitively, we assume 7n to be the Gaussian distribution with pessimistic estimation as the mean:
Z (s, a: 0) (NQLBB = (s, a; 0) epistemic (s, o; 0), estimates its lower confidential bound.
(QQgBB((s) a; f), Aetettorcc (s, a: 8)),
(19)
where
On the other hand, as the quantile distribution is a value distribution which naturally formulates the underlying aleatoric uncertainty, we can utilize the quantile functions to model the pessimistic quantile distribution directly, breaking the Gaussian assumption above. Specifically, we take the smaller estimates at each quantile, and then the distribution =m a; 0) uniform distribution over each quantile value 2; (s, a; 0), shch
2, TK
a; 0) min (; az 0z). k=1,2
(20)
Different from the uni-modal Gaussian distribution, the quantile function able to represent multi- modal distributions, which is more flexible. As the quantile function represents the inverse function of CDF, meaning that we can easily get general idea of the properties of this pessimistic quantile distribution. It is worth noting that we can utilize other methods to formulate ZT like mean estimation, i.e., E [ = (z(s a; 0) and x Th (s, a; 0) = E [z, a; 0z)]. But, only affects the choice of hyper- parameter / and do not affect the final performance.
Algorithm Exploration policy derived from MQES
Initialise: Current state 5t, current value distribution estimators 02, k = 1, 2, current policy net- work o, target policy πT(- Φ) (uT((s: Φ), ())
Output: the MQES exploration policy Te
1: Calculate (sx,, az; 0x), k = 1,2
2: Calculate epistemie uncertainty Jepistemic (St, at) according to Eq. 15
3: Calculate upper bound Q ( a.) using dppistemie (st, ag) according to Eq. 16
4: Calculate aleatoric uncertainty o 2. (st, 0e) according to Eq. 17
5: Construct 2 (s,, ax) using QUB (8t, (8r,aa, de) and diccatore (St; ar) (see Eq. 18)
6: Construct 7Td (st, (e) according to Eq. 19 20
7: Calculate Pg using 7 (8e,, at) and 2" (st,,, (z) according to Eq. 14
8: return Tg ~ N ( (T(;
The above Alg. summarizes the overall procedure of MQES, including the estimation of uncer- tainty (Line 4) and the upper confidential bound QUB (Line 3), formulation of = and 2" (Line 5-6) and exploration policy generation using KL constraint (Line 7). The generated exploration policy can be adopted by any modern based RI algorithms for more effective exploration.

---

Under review as a conference paper at ICLR 2021
4.2.2 ANALYSIS OF MQES BASED EXPLORATION
This section analytically explains how MQES encourages exploration accounting for the aleatoric and epistemic uncertainty. For simplicity, we assume that the sample number is K and Eq. 14 degrades to:
V 22
0z (8,
a) da
PE = pT +
(s.a)
F.
m
O
MT
(21)
m
da
a=pr
ΣE
where m, = log G(s.pr) (2
Take Gaussian MQES for example, have 7*# N(QLB (s, a), o Iuteaooic (8, a)), and the bias term added to hg is decided by the epistemic and aleatoric uncertainty.
Specifically, since epistemic uncertainty is involved in (s, a): the gradient (s.e) encourages the optimistic exploration. Epistemic uncertainty-base based exploration, can avoid the pessimistic un- derexploration. The aleatoric uncertainty is introduced by CDF Φ. If we have two state-action pairs, ie.. ($1, (1) and ($2, a2), and zT (5;, a.) assume that QL1((1,01) 01) = QLB (32, (s (8* ($1, (1)) < Zz (sa (2 ($2, ($2,02) )), which means that larger aleatoric uncertainty
((QLB(( NN (5;, a.), aeeutoic (3,, a,)), = 1,2, and we
(51,
(52.02) a2). of > o². and 3 ("1,, = 2 (82, 02). Obviously,
leads to smaller action bias.
Therefore, the MQES encourages the exploration by selecting the action increasing the optimistic value function, and avoid the over-exploration by setting smaller action bias at the state, where the aleatoric uncertainty is high.
EXPERIMENTS
MQES is designed for efficient exploration in continuous action space problem RL, allowing the agent to be aware of explore directions that may lead to higher optimistic value function with smaller aleatoric uncertainty. Comparisons between MQES and state-of-the-art algorithms are conducted to verify the MQES regarding the efffectiveness and efficiency. Empirical evaluations show that MQES outperforms slate-of-the-e algorithms on a series of continue control tasks.
5.1 IMPLEMENTATION DETAILS AND EXPERIMENT SETTINGS
We compare MQES against SAC (Haarnoja et al., 2018) and its distributional variant DSAC (Ma et al., 2020). Ma et al. (2020) also shows the performance of TD4, which is the distributional extension of TD3 (Fujimoto al., 2018), and can also be used capture epistemic and aleatoric uncertainty as is pointed out in Sec. 4.2. However, DSAC outperforms TD4 as shown in Ma et al. (2020), so we evaluate based only on SAC and DSAC, and further implement MQES based on DSAC in order to develop the exploration ability.
The training process MQES is the same as in DSAC, except for the behavior policy used, while we enrich the experience replay with the data generated by Tg. The pseudo code of the whole process can be found Appendix C. order to ensure a fair comparison, the aapee-parameters of DSAC and MQES are the same (see Appendix D). In addition, we have hyper-parameters associated with MQES. The parameter 22x controls the exploration level, and ,3 determines the magnitude of uncertainty we use, and C is the normalization factor.
We implement both approaches for building n as illustrated in Sec. 4.2.1, and we use MQES _G and MQES_Q Q to indicate respectively to Gaussian distribution and quantile distribution. We test MQES on several tasks in Mujoco (Todorov al., 2012), including standard version, well modified sparse version and stochastic version. We limit the maximum length of each episode to 100. We 1250 or more epochs for each task where there are 100 training steps per epoch, with evaluating every epoch where each evaluation reports the average undiscounted return with no exploration noise.
7

---

Under review as a conference paper at ICLR 2021
250
300
600
Antrege fleurn
Average Return
400
300
200
Average
200
SAC
MOES
100
MQES_ 0
MOES,0 G
MOES_0
MOCS_0
450
MOE5_0
0
250
500
750
1000
1250
500
1000
1500
2000
2500
1000
2000
3000
Number Training #poch
Number of Treining Epuch
Number of Theining Epoch
(a) Walker2d-v2
(b) Ant-v2
(c) Humanoid-v2
50
fowrage Return
401
Averoge acturn
30-
20-
SAC
DSAC
20
DSAC
DSAC
MOES_O
9
MOES Q
MQFS_9
250
1500 Training Epoch
Training
Training
(d) Sparse Walker2d-v2
(e) Sparse Ant-v2
(I) Sparse HumanoidStandup-yy
Figure 1: Training curves on continuous control benchmarks in Mujoco. The x-axis indicates num- ber of training epoch (100 environment steps for each training epoch), while the y-axis the evalua- tion result represented by average episode return. The shaded region denotes one standard deviation of average evaluation over 5 seeds. Curves are smoothed uniformly for visual clarity.
5.2 STANDARD MUJOCO TASKS
We evaluate both MQES. G and MQES_Q in 5 tasks of standard Mujoco, and the result in Figure and Appendix F show that our methods outperform SAC for those tasks, and also reach better performance than DSAC.
Performance. Our results demonstrate that in complex tasks, such Humanoid-v2 and Ant-v2, our MQES- exploration policy performs better, while DSAC suffers from the inefficiency caused by deficient exploration. In Ant-v2, DSAC was overtaken in the early stages of training and then MQES stays ahead. Also in Humanoid-v2, the performance of our algorithm always maintains better than DSAC. some relatively easier tasks, seems that those tasks are not very demanding for exploration, but MQES performs still at very advanced level. In Walker2d-v2, MQES and ASAC alternated lead until 1000 epochs, after which MQES had a significant improvement. The final results are shown
specifically in the Table
Gaussian and Quantile Z". One can find that, expect for Humanoid-v2, there is no absolute superiority between the modeling approaches. We hypothesis this is because environment of Humanoid-v2 is more complex than others, and more flexible quantile 2 is needed, which could model the environment more accurately.
5.3 SPARSE MUJOCO TASKS
To further show the strength of our algorithm regarding exploration, we evaluate on the sparse ver- sion of Mujoco tasks. Specifically, the reward +1 when the move-distance threshold is reached, otherwise O. Obviously, since we set the maximum length of episode to 100, the maximum average episode return in the sparse tasks is up to 100.
As shown in Figure 1 and Appendix F, SAC performs poorly in those tasks, which is to be expected, since solving the sparse reward problem requires not only more accurate estimates of critic, but also more efficient exploration. Even in sparse HumanoidStanduppyy task, when our algorithm MQES
We set the threshold according to the statistical analysis of untrained interaction behavior, using 99.9% quantile value, and the threshold of all environments are shown Appendix
8

---

Under review as a conference paper at ICLR 2021
reaches nearly maximum scores, SAC learns almost nothing, and DSAC performs obviously bit inferior. As can be seen in combination with the Figure and Table 3. both MQES algorithms perform the best, and both do achieve significantly better results faster than DSAC.
MQES performs extremely well in these sparse environments, which on the one hand demonstrates the importance of exploration when solving sparse reward problems, and the other hand shows the advantages and capabilities of MQES for efficient exploration, presenting that incorporating uncertainty to exploration could render better performance.
5.4
ABLATION STUDY
In this section, we conduct two ablation experiments to show the performance gain by distinguishing aleatoric and epistemic uncertaity and the sensitivity to the hyper-parameters.
5.4.1 GAIN OF ALEATTRIC UNCERTAINTY
The exploration with epistemic uncertainty is proved to be efficient (Ciosek et al., 2019) by avoiding the ender-exploration. Hence, we present the gain brought by the aleatoric uncertainty by conducting ablation study in this section. Generally, MQES- besee actor-critic algorithm degrades to Optimistic Actor-Critic (OAC) (Ciosek et al., 2019) if m, in Eq. 13. Consequentially. we could compare MQES with OAC to show the necessary of introducing the alcatoric uncertainty to exploration.
In order to show that MQES performs robustly in the face of aleatoric uncertainty, we modified the standard Ant-v2 task by adding heterogeneous noise the observation increase the aleatoric uncertainty of the environment. We also extend the OAC to the distributional form (DOAC, Distributional OAC), iee, to estimate the values in the same way as MQES, in order to ensure fair comparison. As shown in Figure 2, DOAC does not take into account the aleatoric uncertainty. resulting in performance being close that of DSAC. The experi- mental results confirm that MQES is effective in avoiding the interference aleatoric uncertainty in the environment and ensuring exploration efficiency.
500
DSAC
MOCS_9
Aneraae agee Ruturra
DOAC
600
300
1000
1500
2000
2500
Number of Training Epoch
Figure 2: Gain of aleatoric uncer- tainty, keeping the same plotting settings Figure 1.
5.4.2 ABLATION STUDY ON
MQES is sensitive / /2, which controls the distance be- tween the behavior policy gg and the target policy TT- If /2a is quite small, then the MQES degenerates to DSAC and shows little exploration, and if /2a is larger, then the perfor- mance becomes worse because of the gap between behavior policy and target policy. We evaluate the sensitivity to hyper- parameters Ant-v2 task using MQES_G. and the sensitivity to the constraint /2a is shown in Figure and the sensitivity to 3 is shown in Appendix F, where the error bar indicates half standard deviation.
500
Averoge Retur
800
250
0
Value of typer parameter
Figure Ablation study /2a.
6 CONCLUSION
In this paper, we propose MQES, a generally exploration principle for continuous RL algorithms, which formulates the exploration policy to maximize the information about the globally optimal distribution of Q function. To make MQES practically tractable, we firstly incorporate distribu- tional and ensemble Q function approximations to MQES, which could formulate the epistemic and aleatoric uncertainty accordingly. Secondly, we introduce a constraint to stabilize the training, and solve the constrained MQES problem to derive the exploration policy in closed form. Then, we analyze and show that explores optimistically and avoid overeexploration by recognizing the epis- temic and aleatoric uncertainty, respectively. Empirical evaluations show that MQES works better the complex environments, where the exploration is needed.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Peter Auer. Using confidence bounds for exploitation-exploration trade-offs, J. Mach. Learn. Res., 3:397-422, 2002.
Mare G. Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Rémi Munos. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Infor- mation Processing Systems, pp. 1471-1479, 2016.
Mare G. Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. In Doina Precup and Yee Whye Teh (eds.). Proceedings of the 34th International Con- ference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 449-458. PMLR, 2017.
Richard Y Chen, Szymon Sidor, Pieter Abbeel, and John Schulman. ensembles. arXiv preprint arXiv:1706.00 01 502, 2017.
Ucb exploration via q-
Kamil Ciosek, Quan Vuong, Robert Loftin, and Katja Hofmann. Better exploration with optimistic actor critic. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Bus Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neurul Information Processing Systems 2019, NeurIPS 2019, 8-14 December 2019, Vancouver, BC, Canada, pp. 1785-17999 2019.
William R. Clements, Benoît-Marie Robaglia, Bastien Van Delft, Reda Bahi Slaoui, and Sébastien Toth. Estimating risk and uncertainty in deep reinforcement learning. CoRR, htss/1905.0990 2019.
Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for dis- tributional reinforcement learning. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning. ICML 2018, Stockholmsmüssan, Stock- holm, Sweden, July 10-15. 2018, volume 80 of Proceedings of Machine Learning Research, pp- 1104-1113. PMLR, 201 aa.
Will Dabney, Mark Rowland, Marc G. Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In Sheila A. Mcllraith and Kilian Q. Weinberger (eds.), Pro- ceedings of the Sheood AAAI Conference on Artificial Intellgence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Ed- ucational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pp. 2891. AAAI Press, 2018b.
Lior Fox, Leshem Choshen, and Yonatan Loewenstein. Dora the explorer: Directed outreaching reinforcement action-selection. In International Conference on Learning Representations, 2018.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th Inter- national Conference Machine Leurning, ICML 2018, Stockholmsmüssan. Stockholm, Sweden, July 10-15, 2018 volume 80 of Proceedings of Machine Learning Research, pp. 1582-1591. PMLR, 2018.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- maximum entropy deep reinforcement learning with a stochastic actor. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings the 35th International Conference on Machine Learning, 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 1856-1865. PMLR, 2018.
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. In Advances in Neural Information Processing Systems, pp. 1100-1117, 2016.
Johannes Kirschner and Andreas Krause. Information directed sampling and bandits with het- croscedastic noise. In Sébastien Bubeck, Vianney Perchet, and Philippe Rigollet (eds.), Con- ference On Learning Theory, COLT 2018, Stockholm, Sweden, 6-9 July 2018, volume 75 Pro- ceedings of Machine Learning Research, pp. 3844 PMLR, 2018
10

---

Under review as a conference paper at ICLR 2021
Xiaoteng Ma, Qiyuan Zhang, Li Xia, Zhengyuan Zhou, Jun Yang, and Qianchuan Zhao. Distribu- tional soft actor critic for risk sensitive learning. CoRR, abs/2004. 14547, 2020.
Jarryd Martin, Suraj Narayanan Sasikumar, Tom Everitt, and Marcus Hutter. Count-based explo- ration feature space for reinforcement learning. In Proceedings of the Twenty-Sixth Interna- lional Joint Conference on Artificial Intelligence, pp. 2471-2478, 2017.
Borislav Mavrin, Hengshuai Yao, Linglong Kong, Kaiwen Wu, and Yaoliang Yu. Distributional reinforcement learning for efficient exploration. In Kamalika Chaudhuri and Ruslan Salakhutdi- nov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML. 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 4424 4434. PMLR, 2019.
Thomas M. Moerland, Joost Brockens, and Catholijn M. Jonker. Efficient exploration with double uncertain value networks. CoRR, abs//7 1.0799 2017.
Nikolay Nikolov, Johannes Kirschner, Felix Berkenkamp, and Andreas Krause. Information- directed exploration for deep reinforcement learning. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
Ian Osband, Daniel Russo, and Benjamin Van Roy. (more) efficient reinforcement learning via pos- terior sampling. Christopher J. C. Burges, Léon Bottou, Zoubin Ghahramani, and Kilian Q. Weinherger (eds.), Advances in Neural Information Processing Systems 26: 27th Annual Confer- ence on Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States, pp. 3003-3011, 2013.
lan Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via boot- strapped DQN. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 29: Annual Confer- ence on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 4026-4034, 2016.
Georg Ostrovski, Marc G. Bellemare, Aäron van den Oord, and Rémi Munos. Count-based ex- ploration with neural density models. In Proceedings of the 34th International Conference on Machine Learning, pp. 2721-2730, 2017.
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In Proceedings of the 34th International Conference Machine Learning, pp. 2778-2787, 2017.
Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via disagreement. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 Proceedings of Machine Learning Research, pp. 5062-5071. PMLR, 2019.
Nikolay Savinov, Anton Raichuk, Damien Vincent, Raphaël Marinier, Mare Pollefeys, Timothy P. Lillicrap, and Sylvain Gelly. Episodic curiosity through reachability. 7th International Con- ference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, Open- Review.net, 2019.
Malcolm J. A. Strens. A bayesian framework for reinforcement learning. In Pat Langley (ed.), Proceedings of the Seventeenth International Conference on Machine Learning (ICML 2000), Stanford University, Stanford, CA, USA, June 29 July 2, 2000, pp. 943-950. Morgan Kaufmann, 2000.
R. S. Sutton and A. G. Barto. Reinforcement learning an introduction. Adaptive computation and machine learning. MIT Press, 1998.
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. #exploration: A study of count-based exploration for deep reinforcement learning. In Advances in Neural Information Processing Systems, pp. 2753-2762. 2017.
11

---

Under review as a conference paper at ICLR 2021
William R Thompson. On the likelihood that unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3/4):285-25 -244, 1933.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based con- trol. In 2012 IEEE/RSJ International Conference Intelligent Robots and Systems, IROS 2012, Vilamoura, Algarve, Portugal, October 7-12, 2012, pp. 5026-5033. IEEE, 2012.
Zi Wang and Stefanie Jegelka. Max-value entropy search for efficient Bayesian optimization. vol- ume 70 of Proceedings of Machine Learning Research, pp. 3627-3635, International Convention Centre, Sydney, Australia, 06-11 Aug 2017. PMLR.
Christopher J. C. H. Watkins and Peter Dayan. Technical note q-learning. Mach. Learn., 8:279-292, 1992.
12

---

Under review as a conference paper at ICLR 2021
A
PROOF OF PROPOSITION 1
Proof. The proof is similar to Wang & Jegelka (2017). Here, we utilize nd criticize æ,, i.c.
(s.)="(s,a"),,)) T(a [1 (exo))))r))))))
where C' is the normalization factor, and (z,(),) [M((s)))= (2<(xa((=(ssaa")
B PROOF OF PROPOSITION 2
Proof. Inspired by the methods of multipliers. it equívalent to maximize objective func- tion (s and minimize constraint ([(T||T)) simultaneously. Firstly, we minimize the KL KL(N(H, W ET)) against the variance matrix 2, i.c.
min KL(N(M, ( Er)). Σ
(22)
The optimal solution for problem (22) is
Eg = Er-
(23)
Then, (s,s,a) a) is expanded linearly around 0z = prr
ieT (s, a) Re a a for (s, a))a= + const
(24)
as,a)
=a m ④
da a= const,
(25)
where the element of vector = {
Dz (E 1),
m is the dimension of action, and (xz) is the probability distribution function (pdf). Then, problem (12) is reformulated as:
C .z 2z (2 (log
(26)
C2(22))E
(s,
max Eg- Fst m ②
da a=
s.t. 2 ((- (T)) - (1 -rr) S a.
1
(27)
Then, the Lagrange function of problem (27) is given as:
(5 a) la=pr
L(p) = Eg
m
[(0 (1 -T) (n (()n-))---)))m -
(28)
da
According to the KKT condition, we derive the following equations: First, from the stationary, i.e.,
(pL(H) m
a (s.a)
l=pp -E- (p pT) we get
Du
0 (s, a) m
PT
ME 2.
aa a=
(29)
where À > O. Secondly, since > O, the constraint active, i.e. (μ - --r) T (p - pT) 2a. Together with Eq. 29, we get the following equation:
0
E.
80
5.
(30)
2a
Finally, (13) is obtained by plugging (30) to (29).
C ALGORITHM 2: MQES FOR DSAC
In this section, we show the whole algorithm of our implementation of MQES based on DSAC in
Alg. 2.
13

---

Under review as a conference paper at ICLR 2021
Algorithm 2 MQES for DSAC
Initialise: Value networks 0,, 02. policy network & and their target networks 0,, 02, o, quantiles number target smoothing coefficient (r), discount (7), an empty replay pool D
for each iteration do
2: for each environmental step do
3: Gt ~ (a, st) according to Alg. 1
4:
D + D U {(st, az, 75t, dz), 8t+1)}
5:
end for
6:
for each training step do
7:
for i= to do for to N do
8:
9:
calculate 6t k = 1, following Eq.
end for 11: end for
Calculate Lon (f&), 1,2 using , following Eq.
Update 0% with (()n((0))
14:
Calculate J= (d), following Eq. 7
Update with (+)
16:
end for
17:
Update target value network with 0g +- 70p + (1 - - 1,2
18:
Update target policy network with < TØ + (1 - =)Φ
19: end for
Table 1: MQES parameters
Parameter
Value
Training
discount
0.99
Target smoothing coefficient
5e-3
Learning rate
3c-4
Optimizer
Adam
Replay buffer size
10e6
Batch size
256
Quantiles number
20
Action range
[-1,
Environment steps per epoch
100
Exploration
Exploration ratio
/2a
0.8
Uncertainty ratio
1.6
Normalization factor
C
0.5
D HYPER-PARAMETERS SETTING
The hyper-parameters in experiment guaranteed to be consistent, shown in Tab. 1.
E THRESHOLD SETTINGS FOR SPARSE TASKS
As illustrated Sec. 5.3, We set the threshold according to the statistical analysis of untrained interaction behavior, using 99.9 quantile value, shown in Tab. 2.
14

---

Under review as a conference paper at ICLR 2021
Table 2: Threshold settings for sparse reward tasks
Sparse tasks
Threshold
Sparse Ant-v2
0.13
Sparse Hopper-v2
0.605
Sparse HalfCheetah-v2
3.5
Sparse Walker2d-v2
0.525
Sparse Humanoid-v2
0.17
Sparse HumanoidStanduup
0.18
500
Iverage Return
460
440-
420-
460
Value hyper
Figure Ablation study on 3
F ADDITIONAL EXPERIMENT RESULTS
Limited by the length of the text, we present the results the evaluation on some simple tasks in Fig. 5. All evaluation data can be seen in Tab. 3.
The sensitivity to 3 shown in Fig. 4. 3 controls the uncertainty magnitude, the smaller the value, the smaller the degree of optimism or pessimism, and vice v2 task using M MQES E.SQ. What can be seen is that larger 3 will degrade performance. Although smaller B are more profitable in the Ant-v2 task, it is not necessarily in other task, we set uniform 1.6 in our experiments.
We evaluate the effect of in Ant-
15

---

Under review as a conference paper at ICLR 2021
Avorage Recurn
Average Btaturm
200
-60
SAC
-00
MUL-1GG
100
MiJES,
MOES_O
500
1000
1500
2000
2500
250
500
750
1000
1250
Number of Theining Epoch
Number of Training Epoch
(a) Pusher-v2
(b) Hopper-v2
80
25
Average Retur
Average Return
15
40
10
SAC
DSAC
36
DSAC
SAC
20
SAC
MOES
MOtS_G
MGES_G
MOFS_0
MOES_Q
MOFS_9
250 500
750
1000
1250
250
500
750
1000
1250
1000
2000
3000
Number of Training Epoch
Number Training Epoch
Number of Training Epoch
(c) SparseHopper-v2
(d) Sparse Halfcheetah-v2
(c) Sparse Humanoid-v2
Figure 5: Training curves on continuous control benchmarks in Mujoco. The x-axis indicates num- ber of training epoch (100 environment steps for each training epoch). while the y-axis is the evalua- tion result represented by average episode return. The shaded region denotes one standard deviation of average evaluation over 5 seeds. Curves smoothed uniformly for visual clarity.
Table Average return over seeds with one standard deviation at corresponding training step, i.e., 1.25 10 million training step for Hopper-v2. The maximum value of each row is shown in bold.
Task
le5
SAC
DSAC
MQES
MQES Q
Walker-v2
1.25
164.4 20.7
204.2 + 15.3
226.6 +18.7
212.1 # 18.7
Ant-v2
2.5
276.0 + 24.5
456.5 52.2
507.5 +12.2
456.5 : 48.4
Humanoid-v2
3.0
487.5 + 6.0
597.7
612.3 110.5
607.3 13.6
Sparse Walker2d-v2
1.25
35.6 + 8.8
3.9
50.6 13.1
47.9 1 3.7
Sparse Ant-v2
2.5
6.0
6.4
6.44
69.3 +15.3
Sparse HumanoidStandup
0.25
5.3
82.9
95.0
94.9 1.4
Pusher-v2
2.5
-55.7 26.9
0.6.1
-210
-20.8 10.3
Hopper-v2
1.25
203.3
19.9
248.944.6
233.617.6
234.2 11.6
Sparse H2pper-v2
1.25
25.3 2.5
2..7
2..3
26.42..3
Sparse HalfCheetah-v2
1.25
53.5 4: 12.7
70.2 70.2+111
70.3 +0.5
68.3 4 2.4
Sparse Humanoid-v2
3.0
24.9
4.66
16.00
51.0 10.3
16


Reviews:
[{"review_id": "umHwKNCpb_1", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an exploration scheme for RL in continuous action spaces using the principle of information maximization for globally optimal Q distribution.\n\n1. I felt that the paper isn't well written and discusses a lot of different concepts in a haphazard manner.  There are a lot of equations and symbols in the text without proper explanation and context which make it difficult to gather the main contribution. The language used gets vague in many statements made in the paper. For ex. \"Proposition 1. Generally, the posterior probability is as follows\".\n\n2. A lot of algorithm adaptations are proposed without actually carrying out ablations which make it difficult to discern if the proposed MI maximization is indeed responsible for performance. For ex. \"Since the target for critic in the advanced algorithms, like SAC and TD3, is usually estimated pessimistically..\". The authors should actually present ablations to support if a pessimistic estimate is indeed required for their adaptation for these methods. \n\n3. Why haven't the authors included OAC as a baseline given that it outperforms SAC in several tasks? Further the results show little difference in performance in comparison with DSAC on the mujoco tasks, given that only 5 seeds were used in evaluation, it brings the significance of the results under question. The authors should provide appropriate measures like P-values to support the experiments.", "extra_scores": {}, "extra_text": {}}, {"review_id": "r-FE_4YDv7n", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an information-theoretic approach to exploration in model-free RL, by encouraging an exploration policy that is maximally informative about the optimal (distributional) value function. The authors discuss tractable approximation to this objective which can be implemented in continuous MDPs. The method is evaluated on benchmark continuous control tasks (Mujoco).\n\nThe basic underlying idea is interesting and, to the best of my knowledge, novel. However there are some major issues and/or limitations which should be addressed prior to publication.\n\n**Clarity**\nThe paper is hard to follow and understand. \n* There is a use of terminology which might not be clear or known for the general RL/exploration audience (\"acquisition functions\", \"heteroscedastic aleatoric uncertainty\"). The entire discussion of the two types of uncertainty seems somewhat disconnected from the method itself. A short explanation of why and how the two types of uncertainty are important *for the exploration problem* would be very helpful.\n* There are some notation obscurities or inaccuracies. This is most notable in Section 4.1, which is unfortunate since this is where the key ideas of the approach are discussed.\n    * Equation 8, which is central to what follows, is rather confusing. The text mentions that \"$\\pi_E$ selects action $a_t$ that (...)\", but the equations then seems to define an entire policy. And the optimization problem (in the same Eq.) is in itself dependent on $a_t$, so it's not even clear one gets a valid policy/distribution from something like $\\pi_E = \\arg\\max_\\pi F^\\pi(s_t,a_t)$. \n    * Following the previous point, Eq. 9 is also confusing. It's not clear to me what the authors mean by measuring MI between $Z^*$ and *the pair* $(Z^\\pi, \\pi)$? It's also not clear what is the meaning of the \"posterior distribution\" denoted by $p$, and why the mutual information is measured for the Z parameters (return probs) but then re-expressed as the difference in entropies for the policies (action probs).\n\n**Quality**\nThe paper has a good balance of a theoretically motivated algorithm, a practical implementation of it, and some basic empirical evaluation. Other than clarity issues discussed before, I have some concerns regarding the evaluation, and one more conceptual concern regarding the general idea:\n* Since exploration here is encouraged by choosing informative actions about Q*, it's not clear that this method will be helpful in very sparse-reward settings (which are a central motivation for sophisticated exploration techniques). Put differently, relying on Q* to guide exploration ultimately couples exploration to the external reward, which seems rather undesirable to me. The method might be helpful from other perspectives (optimization, controlling the level of \"over-exploration\" etc), but it's not clear that it is helpful as an exploration method per se. \n* The empirical evaluation is done only against rather limited baselines (basically random exploration baselines). I would encourage the authors to compare their method to other forms of exploration. Particularly relevant to this paper is the work by Houthooft et al. 2016 (VIME) which also uses information-theoretic objective for exploration in continuous problems. The authors should at least cite this paper in their related work section.\n* Following the last two points, a great improvement could be if other than just demonstrating performance in terms of reward, the authors would evaluate the exploratory behavior itself of an agent trained with their method, in some simple environment (i.e in terms of novel states visited / distance traveled / etc.)\n\nThere are some more minor issues which should be addressed as well: \n* In the abstract: the use of \"optimism in the face of uncertainty\" is definitely not a \"recently\"\n* \"The above methods are not efficient\" (3rd paragraph, Introduction): This is not accurate. UCB (and other methods) **are** provably efficient for several problems/assumptions.\n* Some relevant literature is missing from the related work. Most notable is the VIME paper mentioned earlier (Houthooft et al. NeurIPS 2016) and the Fox et al. 2018 ICLR paper (DORA The explorer) which combines counter-based like exploration with an optimism principle for high-dim MDPs.\n* Section 3.2: $T^\\pi$ is **not** the \"Bellman optimiality operator\" but rather the bellman operator for policy $\\pi$.\n\n**Conclusions**\nThis work has some interesting idea which could be useful for training RL agents in continuous problems. However in the current form of the paper it's hard to evaluate and understand some of the key ideas of the work. Given this, and together with the more conceptual concerns regarding evaluation and the basic approach, I think the paper is not ready for publication.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "OVog7hbw0BY", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work introduces max-Q Entropy Search (MQES) exploration principle  for continuous RL algorithms. MQES addresses the exploration-exploitation dilemma that constitutes a fundamental RL problem. Actually, MQES defines an exploration policy able to explore optimistically and avoid over-exploration. One of the main advantages of MQES is its ability to recognise the epistemic and aleatoric uncertainty. Empirical analysis has been conducted on Mujoco, showing that the performance of MQES is comparable to  those of other state-of-the-art algorithms.\n\nIn general the paper is well written and can be easily followed by the reader. Nevertheless some parts of the MQES should be explained in more detail. For instance, authors should give more details about the target policy introduced at Section 4.3. Actually, the reader should check Algorithm 2 at Appendix in order to understand its purpose and how the target policy is updated. I think that it would be better Algorithm 2 to be moved in the main paper if it is possible. Another point that should be discussed more clearly is the impact of the 3 hyper-parameters (\\alpha, \\beta, and C) on the performance of MQES. To be more specific, why did you set the uncertainty ratio equal to 1.6? Finally, the empirical results are not discussed at all. It seems for example that the performance of MQES_G is more stable compared to that of MQES_Q. Moreover, the performance of DSAC is almost equal (or better) to that of MQES_Q. All these points should be explained or discussed by the authors.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "6XZfa778gXU", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "---\nSummary\n\nThis paper studies the problem of efficient exploration in continuous environment. It proposes a novel algorithm, Max-Q Entropy Search (MQES), and utilizes Distributional SAC (DSAC) to formulate the uncertainties. Experiments show that the proposed algorithm, MQES, outperforms the baselines (SAC, DSAC). \n\n---\nComments\n\nHowever, MQES doesn't show significant improvement over DSAC. For example, Except Sparse-HalfCheetah-v2, MQES_Q has almost same performance as DSAC. Except Sparse-HalfCheetah-v2 and Ant-v2, MQES_G has almost the same performance as DSAC. \n\nAnother question is: The horizon is cut to 100 while most papers and OpenAI Gym use 1000 by default. Why do the authors choose 100? How does MQES perform with longer horizon, like 1000? \n\nThe paper shows that exploring using both aleatoric and epistemic uncertainty can improve the performance. What if we consider only one of them, e.g., using only aleatoric uncertainty? I'd like to see this as ablation. \n\n\n---\nWriting Quality\n\nThe writing can also be improved. \n\nTable 1: Could you please highlight all algorithms within 1 std to the best? \n\nSparse-HalfCheetah-v2: Could you please provide more details about the environment? \n\nWhat does the mutual information between $(Z^*, \\pi^*)$ and $(Z^{\\pi_E}, \\pi_E)$ mean? Are $\\pi^*$ and $\\pi_E$ random variables? Moreover, in deterministic environments (as in Mujoco environments), $\\pi^*$ is also deterministic, so is $Z^*(s_t, a_t)$. \n\nEq 8: LHS is a scalar (probablity), while RHS is a policy. Please clarify notations to avoid confusion. \n\nEq 9: What is $p$? The first input of MI is simply a $Z^*$ while the second is a pair $(Z^\\pi, \\pi(a_t | s_t))$. Please elaborate. \n\n> To measure the intractable distribution of $Z^*$ during training, we use the $\\hat Z^*$ for approximation \n\nPlease rephrase it and say that $\\hat Z^*$ will be defined later. \n\nEq 20: This is not an unbiased estimation, as $\\mathbb{E}[X^{-1}] \\neq \\mathbb{E}[X]^{-1}$. Also please clarify K. \n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "2lWYm0iIR7U", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis paper proposes MQES, a Max-Q entropy search for policy optimization in continuous RL. The authors propose to combine advantages of the information-theoretic principle and distributional RL, in which epistemic and aleatoric uncertainty are estimated using similar entropy-search acquisition functions in the Bayesian Optimization (BO). As said, this is a new method to introduce a more efficient exploration strategy. As a result, policy improvement is formulated as a constraint optimization problem where a next exploration policy can be solved in a closed-form. The proposed method is evaluated on Mujoco tasks and compared against other off-policy approaches, SAC, and DSAC. The results show MQES outperforms other methods in domains where exploration is needed.\n\nThe main contribution of the paper is to introduce an approximation to distribution Q functions that are based on the epistemic and aleatoric uncertainty. The main objective is based on the mutual information maximization as described in 4.1. A practical implementation is proposed in 4.1. The idea makes sense however the presentation and its experiment results make it hard to understand some important details. Some of my major comments are as follows.\n\n1. Although the idea is interesting, it's not yet clear how the proposed method can be used as an additional module on top of other entropy-regularized off-policy approaches like SAC or TD3, etc. The paper can benefit more if it can be formulated in such a more general way.\n\n2. The practical implementation seems to make sense. However, it's still unclear to me how policies \\pi_E and \\pi_T are parameterized. Although a sketch of the main idea is described in Algorithm 1, it's not clear to me each term is parameterized and computed based on a particular parameterization. \n\n\n3. An updated policy as a solution of (18) gives an update on the mean but keeps the covariance unchanged. I was wondering then how this policy can adaptively change its exploration through the progress of learning?\n\n4. The experiment results are quite preliminary. There are no experiment settings. There are a number of hyperparameters of MQES that might affect overall performance of MQES, e.g. N, \\beta etc. but not discussed and ablated? The comparisons might also take into account other distributional policy search methods.\n\n5. And some minor comments\n\n\t- Eq. 4 and 5, 6: min and log instead of arg functions?\n\n\t- Eq. 9: What is the difference between posterior p(a|Z^*(s,\\pi^*) and \\pi^*? Is the posterior not the optimal policy since Z^* is the optimal distributional value function estimate? A detailed derivation for Eq.9 is expected.\n\n\t- Definitions for terms in 4.1:  p(a|Z^*(s,\\pi^*) vs  p(a|Z^*(s,\\pi) p(a|Z^*(s,\\pi^*) p(a|Z^*(s,\\pi) etc.\n\n\t- in Eq.14: Should the aleatoric uncertainty be the variance of the {min_{\\theta_k} z_i(\\theta_k)} instead of en expectation over \\theta_k. Because z_i is estimated as the min of two estimates, e.g. in Eq.4\n\n\t- Some theoretical steps are not clearly justified of why.\n\n\t- is $n$ used in Proposition 2?\n\n\t- the conventions of CDF in Proposition 2 and in Eq.21 should be made consistent with its first definition in Eq.11\n\n\t- definition of G() is not used after its first definition.\n\n\t- why the horizon is set to 100, instead of 1000 environment steps like in the SAC paper?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper contributes to the community by introducing an approximation to distribution Q functions, based on the epistemic and aleatoric uncertainty. The reviewers believe the ideas make sense. However the presentation and its experiment results make it hard for them to understand some important details. For example, the reviewers are confused about why the empirical results show the proposed methods are better.   

The majority of the reviewers are negative about the paper. After rebuttal, the reviewers are not convinced. Based on this, the meta-reviewer recommends rejection. Authors can strengthen paper by improving its presentation and addressing the concerns from the reviewers.   
Decision: Reject