Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: 
This paper proposes MQES, a Max-Q entropy search for policy optimization in continuous RL. The authors propose to combine advantages of the information-theoretic principle and distributional RL, in which epistemic and aleatoric uncertainty are estimated using similar entropy-search acquisition functions in the Bayesian Optimization (BO). As said, this is a new method to introduce a more efficient exploration strategy. As a result, policy improvement is formulated as a constraint optimization problem where a next exploration policy can be solved in a closed-form. The proposed method is evaluated on Mujoco tasks and compared against other off-policy approaches, SAC, and DSAC. The results show MQES outperforms other methods in domains where exploration is needed.

The main contribution of the paper is to introduce an approximation to distribution Q functions that are based on the epistemic and aleatoric uncertainty. The main objective is based on the mutual information maximization as described in 4.1. A practical implementation is proposed in 4.1. The idea makes sense however the presentation and its experiment results make it hard to understand some important details. Some of my major comments are as follows.

1. Although the idea is interesting, it's not yet clear how the proposed method can be used as an additional module on top of other entropy-regularized off-policy approaches like SAC or TD3, etc. The paper can benefit more if it can be formulated in such a more general way.

2. The practical implementation seems to make sense. However, it's still unclear to me how policies \pi_E and \pi_T are parameterized. Although a sketch of the main idea is described in Algorithm 1, it's not clear to me each term is parameterized and computed based on a particular parameterization. 


3. An updated policy as a solution of (18) gives an update on the mean but keeps the covariance unchanged. I was wondering then how this policy can adaptively change its exploration through the progress of learning?

4. The experiment results are quite preliminary. There are no experiment settings. There are a number of hyperparameters of MQES that might affect overall performance of MQES, e.g. N, \beta etc. but not discussed and ablated? The comparisons might also take into account other distributional policy search methods.

5. And some minor comments

	- Eq. 4 and 5, 6: min and log instead of arg functions?

	- Eq. 9: What is the difference between posterior p(a|Z^*(s,\pi^*) and \pi^*? Is the posterior not the optimal policy since Z^* is the optimal distributional value function estimate? A detailed derivation for Eq.9 is expected.

	- Definitions for terms in 4.1:  p(a|Z^*(s,\pi^*) vs  p(a|Z^*(s,\pi) p(a|Z^*(s,\pi^*) p(a|Z^*(s,\pi) etc.

	- in Eq.14: Should the aleatoric uncertainty be the variance of the {min_{\theta_k} z_i(\theta_k)} instead of en expectation over \theta_k. Because z_i is estimated as the min of two estimates, e.g. in Eq.4

	- Some theoretical steps are not clearly justified of why.

	- is $n$ used in Proposition 2?

	- the conventions of CDF in Proposition 2 and in Eq.21 should be made consistent with its first definition in Eq.11

	- definition of G() is not used after its first definition.

	- why the horizon is set to 100, instead of 1000 environment steps like in the SAC paper?
review_title: Interesting idea but unconvincing results.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This work introduces max-Q Entropy Search (MQES) exploration principle  for continuous RL algorithms. MQES addresses the exploration-exploitation dilemma that constitutes a fundamental RL problem. Actually, MQES defines an exploration policy able to explore optimistically and avoid over-exploration. One of the main advantages of MQES is its ability to recognise the epistemic and aleatoric uncertainty. Empirical analysis has been conducted on Mujoco, showing that the performance of MQES is comparable to  those of other state-of-the-art algorithms.

In general the paper is well written and can be easily followed by the reader. Nevertheless some parts of the MQES should be explained in more detail. For instance, authors should give more details about the target policy introduced at Section 4.3. Actually, the reader should check Algorithm 2 at Appendix in order to understand its purpose and how the target policy is updated. I think that it would be better Algorithm 2 to be moved in the main paper if it is possible. Another point that should be discussed more clearly is the impact of the 3 hyper-parameters (\alpha, \beta, and C) on the performance of MQES. To be more specific, why did you set the uncertainty ratio equal to 1.6? Finally, the empirical results are not discussed at all. It seems for example that the performance of MQES_G is more stable compared to that of MQES_Q. Moreover, the performance of DSAC is almost equal (or better) to that of MQES_Q. All these points should be explained or discussed by the authors.

review_title: MaxQ Entropy Search Blind Review #1  

final_score: rating: 5: Marginally below acceptance threshold
review_content: ---
Summary

This paper studies the problem of efficient exploration in continuous environment. It proposes a novel algorithm, Max-Q Entropy Search (MQES), and utilizes Distributional SAC (DSAC) to formulate the uncertainties. Experiments show that the proposed algorithm, MQES, outperforms the baselines (SAC, DSAC). 

---
Comments

However, MQES doesn't show significant improvement over DSAC. For example, Except Sparse-HalfCheetah-v2, MQES_Q has almost same performance as DSAC. Except Sparse-HalfCheetah-v2 and Ant-v2, MQES_G has almost the same performance as DSAC. 

Another question is: The horizon is cut to 100 while most papers and OpenAI Gym use 1000 by default. Why do the authors choose 100? How does MQES perform with longer horizon, like 1000? 

The paper shows that exploring using both aleatoric and epistemic uncertainty can improve the performance. What if we consider only one of them, e.g., using only aleatoric uncertainty? I'd like to see this as ablation. 


---
Writing Quality

The writing can also be improved. 

Table 1: Could you please highlight all algorithms within 1 std to the best? 

Sparse-HalfCheetah-v2: Could you please provide more details about the environment? 

What does the mutual information between $(Z^*, \pi^*)$ and $(Z^{\pi_E}, \pi_E)$ mean? Are $\pi^*$ and $\pi_E$ random variables? Moreover, in deterministic environments (as in Mujoco environments), $\pi^*$ is also deterministic, so is $Z^*(s_t, a_t)$. 

Eq 8: LHS is a scalar (probablity), while RHS is a policy. Please clarify notations to avoid confusion. 

Eq 9: What is $p$? The first input of MI is simply a $Z^*$ while the second is a pair $(Z^\pi, \pi(a_t | s_t))$. Please elaborate. 

> To measure the intractable distribution of $Z^*$ during training, we use the $\hat Z^*$ for approximation 

Please rephrase it and say that $\hat Z^*$ will be defined later. 

Eq 20: This is not an unbiased estimation, as $\mathbb{E}[X^{-1}] \neq \mathbb{E}[X]^{-1}$. Also please clarify K. 



review_title: Review

final_score: rating: 3: Clear rejection
review_content: The paper proposes an exploration scheme for RL in continuous action spaces using the principle of information maximization for globally optimal Q distribution.

1. I felt that the paper isn't well written and discusses a lot of different concepts in a haphazard manner.  There are a lot of equations and symbols in the text without proper explanation and context which make it difficult to gather the main contribution. The language used gets vague in many statements made in the paper. For ex. "Proposition 1. Generally, the posterior probability is as follows".

2. A lot of algorithm adaptations are proposed without actually carrying out ablations which make it difficult to discern if the proposed MI maximization is indeed responsible for performance. For ex. "Since the target for critic in the advanced algorithms, like SAC and TD3, is usually estimated pessimistically..". The authors should actually present ablations to support if a pessimistic estimate is indeed required for their adaptation for these methods. 

3. Why haven't the authors included OAC as a baseline given that it outperforms SAC in several tasks? Further the results show little difference in performance in comparison with DSAC on the mujoco tasks, given that only 5 seeds were used in evaluation, it brings the significance of the results under question. The authors should provide appropriate measures like P-values to support the experiments.
review_title: Are the empirical results significant enough to support the proposed exploration heuristic?

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper proposes an information-theoretic approach to exploration in model-free RL, by encouraging an exploration policy that is maximally informative about the optimal (distributional) value function. The authors discuss tractable approximation to this objective which can be implemented in continuous MDPs. The method is evaluated on benchmark continuous control tasks (Mujoco).

The basic underlying idea is interesting and, to the best of my knowledge, novel. However there are some major issues and/or limitations which should be addressed prior to publication.

**Clarity**
The paper is hard to follow and understand. 
* There is a use of terminology which might not be clear or known for the general RL/exploration audience ("acquisition functions", "heteroscedastic aleatoric uncertainty"). The entire discussion of the two types of uncertainty seems somewhat disconnected from the method itself. A short explanation of why and how the two types of uncertainty are important *for the exploration problem* would be very helpful.
* There are some notation obscurities or inaccuracies. This is most notable in Section 4.1, which is unfortunate since this is where the key ideas of the approach are discussed.
    * Equation 8, which is central to what follows, is rather confusing. The text mentions that "$\pi_E$ selects action $a_t$ that (...)", but the equations then seems to define an entire policy. And the optimization problem (in the same Eq.) is in itself dependent on $a_t$, so it's not even clear one gets a valid policy/distribution from something like $\pi_E = \arg\max_\pi F^\pi(s_t,a_t)$. 
    * Following the previous point, Eq. 9 is also confusing. It's not clear to me what the authors mean by measuring MI between $Z^*$ and *the pair* $(Z^\pi, \pi)$? It's also not clear what is the meaning of the "posterior distribution" denoted by $p$, and why the mutual information is measured for the Z parameters (return probs) but then re-expressed as the difference in entropies for the policies (action probs).

**Quality**
The paper has a good balance of a theoretically motivated algorithm, a practical implementation of it, and some basic empirical evaluation. Other than clarity issues discussed before, I have some concerns regarding the evaluation, and one more conceptual concern regarding the general idea:
* Since exploration here is encouraged by choosing informative actions about Q*, it's not clear that this method will be helpful in very sparse-reward settings (which are a central motivation for sophisticated exploration techniques). Put differently, relying on Q* to guide exploration ultimately couples exploration to the external reward, which seems rather undesirable to me. The method might be helpful from other perspectives (optimization, controlling the level of "over-exploration" etc), but it's not clear that it is helpful as an exploration method per se. 
* The empirical evaluation is done only against rather limited baselines (basically random exploration baselines). I would encourage the authors to compare their method to other forms of exploration. Particularly relevant to this paper is the work by Houthooft et al. 2016 (VIME) which also uses information-theoretic objective for exploration in continuous problems. The authors should at least cite this paper in their related work section.
* Following the last two points, a great improvement could be if other than just demonstrating performance in terms of reward, the authors would evaluate the exploratory behavior itself of an agent trained with their method, in some simple environment (i.e in terms of novel states visited / distance traveled / etc.)

There are some more minor issues which should be addressed as well: 
* In the abstract: the use of "optimism in the face of uncertainty" is definitely not a "recently"
* "The above methods are not efficient" (3rd paragraph, Introduction): This is not accurate. UCB (and other methods) **are** provably efficient for several problems/assumptions.
* Some relevant literature is missing from the related work. Most notable is the VIME paper mentioned earlier (Houthooft et al. NeurIPS 2016) and the Fox et al. 2018 ICLR paper (DORA The explorer) which combines counter-based like exploration with an optimism principle for high-dim MDPs.
* Section 3.2: $T^\pi$ is **not** the "Bellman optimiality operator" but rather the bellman operator for policy $\pi$.

**Conclusions**
This work has some interesting idea which could be useful for training RL agents in continuous problems. However in the current form of the paper it's hard to evaluate and understand some of the key ideas of the work. Given this, and together with the more conceptual concerns regarding evaluation and the basic approach, I think the paper is not ready for publication.


review_title: Review

Meta-review: The paper contributes to the community by introducing an approximation to distribution Q functions, based on the epistemic and aleatoric uncertainty. The reviewers believe the ideas make sense. However the presentation and its experiment results make it hard for them to understand some important details. For example, the reviewers are confused about why the empirical results show the proposed methods are better.   

The majority of the reviewers are negative about the paper. After rebuttal, the reviewers are not convinced. Based on this, the meta-reviewer recommends rejection. Authors can strengthen paper by improving its presentation and addressing the concerns from the reviewers.   
Decision: Reject