Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work proposes a method to disentangle epistemic from aleatoric uncertainty for avoiding the noisy TV problem which occurs when intrinsically motivated agents get rewarded for visiting states that have high irreducible uncertainty.
main_review: The main weakness of the current work is that it misses relevant work which does the same thing while it seems to misinterpret certain parts of the literature. As an example, the authors state that "tractable epistemic uncertainty estimation with high dimensional data is an unsolved problem", citing Gal 2016 which demonstrates a method for estimating epistemic uncertainty in high dimensional data (that's MC Dropout). Also, there is a similar approach using Deep Ensembles to assess epistemic uncertainty. Although far from optimal, such approaches are already being used in epistemic uncertainty driven exploration. For example, in Planning to Explore ([https://arxiv.org/abs/2005.05960](https://arxiv.org/abs/2005.05960)) the authors use epistemic uncertainty estimation as an intrinsic reward. In fact, the authors in Related Work last paragraph, they cover a few efforts using epistemic uncertainty estimation as intrinsic rewards, while they don't mention what's the shortcomings of such work which they aim to solve with their proposal.

In terms of experimentation, the selected baselines are not sufficient to demonstrate how much better this method is. I'd expect to see a comparison with MC Dropout or Deep Ensembles based methods of assessing epistemic uncertainty (in the presence of heteroscedastic noise), or DUE ([https://arxiv.org/pdf/2102.11409.pdf](https://arxiv.org/pdf/2102.11409.pdf)).

**Detailed comments:**

C1: "this work presents aleatoric mapping agents which use single network deterministic uncertainty estimation" - is the reference correct there? Kendall & Gal work uses a Bayesian Neural Network, whereas deterministic uncertainty estimation was proposed here [https://openreview.net/forum?id=Fu7D6kQPzs4](https://openreview.net/forum?id=Fu7D6kQPzs4) (not cited in the text)

C2: For the epistemic and aleatoric uncertainty you cite Hullermeier & Waegeman. Can you also cite the earlier references? e.g. "Hora S (1996) Aleatory and epistemic uncertainty in probability elicitation with an example from hazardous waste management. Reliability Engineering and System Safety 54(2–3):217–223"

C3: "However, as far as we are aware, we are the first to compute aleatoric uncertainties with a scalable curiosity framework to reduce intrinsic rewards for those state transitions with aleatoric uncertainty" - I don't understand what is the novelty here. There is a lot of work disentangling epistemic from aleatoric uncertainty which works at scale. Can you please state as precise as possible what's the contribution in terms of the computation of aleatoric uncertainty?

C4: "We implicitly incentivise agents to seek epistemic uncertainties by removing the aleatoric component from the total prediction error" - If I'm not mistaken that's the common way to disentangle the epistemic from the aleatoric uncertainty which is usually derived by the law of total variance. What is the novelty here?

C5: "Possible because regularisation terms could absorb some epistemic uncertainty" - What does it mean to absorb some epistemic uncertainty? Epistemic uncertainty can be reduced or increased. Do you mean that through the regularization, epistemic uncertainty is reduced? If so, can you explain why this can happen?

C6: In the Minigrid experiments, can you try to implement the intrinsic reward based on the epistemic uncertainty as evaluated from ensemble Kendall & Gal 2017 models? That should be the Variance over expectations of the deep ensemble components $Var_{\theta \sim p(\theta \mid D)}[ \mu^\theta_{s_{t+1}} ]$.

C7: It's very difficult to follow the experimental protocol in section 5. How do you estimate the Acetylcholine in the simulations?

Finally, I'd like to suggest to the authors to try to use \citep instead of \cite whenever possible. Also try to use a more accessible colour for the citations as currently, it makes it challenging to read.
summary_of_the_review: My recommendation is influenced by the following issues:
- The novelty of the contribution is not clear to me. I'd like to understand C3-C6 because overall it seems to me that epistemic uncertainty driven intrinsic rewards have been used before and here it's not compared thoroughly against such baselines (which are mentioned in related work)
- C7 - I don't understand how this section contributes towards understanding what type of uncertainty acetylcholine signals.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper suggests an intrinsic bonus for exploration that avoids noisy TV by adding a penalty for the estimated variance of the reached state $S_{t+1}$ given previous state $S_{t}$.

For this, they fit an independent normal model of the new state $S_{t+1}$ with mean ($\mu_{t+1}$) and var ($\sigma_{t+1}^2$) predicted from the previous state $S_{t}$.
Then, the bonus is given by the error of prediction $(S_{t+1} - \hat{\mu_{t+1}})^2$ minus the variance penalty $\sigma_{t+1}^2$. 

That way, a noisy TV will always be fitted with an higher variance that should have a similar order than the error of prediction, so both predictable and unpredictable transitions bonus will end up to zero (in average) after learning the model of dynamics of the whole environment.

main_review: I am surprised that this idea was not explored in an earlier paper. Taking the prediction error minus the variance looks like the very first solution to try in order to prevent agents to stay watching a noisy TV. While my knowledge to this domain is limited, to my mind this is the first paper suggesting this idea. Probably other reviewers will prove me wrong. 


I expect this approach to work well, and I trust the good experimental results exposed in the paper. I believe this could have a nice impact on the curiosity/exploration efforts in the RL community.


However, there are many weaknesses that makes this paper not ready for a publication:

First, all the mathematical justifications are unclear and slovenly written with too much errors in equations:

Eq 3) is wrongly reported. It should be $S_{t+1}$ in the left of the noise term, and $\hat{S_{t+1}}$ in the right of the variance term.

In Eq 5), $\mu_{S_{t+1}}$ should actually be $S_{t+1}$ (except it's an average over a batch of states $S_{t+1}$ with the same previous $S_t$, but I guess it's not)

In Eq 6) $\hat{S}$ should be $\hat{\mu}$ for consistency.

Also I feel that there is a lot of paragraphes just to describe a very simple gaussian model used to fit $S_{t+1}$ given $S_{t}$ by maximum log-likelihood (in Eq 5). Sigma is the estimated variance, and mu is the estimated mean. Eq 5 is just the log of Eq 4 (with lambda = -1/pi).
Then in Eq 7, it's just R = error - estimated variance. No need for $\eta$, the point is that these terms should have the same expectation at convergence of the fitted model with $\eta=1$.


The bio-inspired aspect needs more content to be convincing: 

for example, reporting models or experimental results from the biological papers (for ex, the actual original curves that fig 5 is trying to reproduce).

What I suggest:

The experimental section looks good to me. But all the method section needs to be re-written from scratch. After introducing the MDP notations, I would directly start with the final equation 7: we take the error minus the variance.

Then, show that the mean of this bonus is zero when the model is learned (so whatever is the unpredictability, the bonus will go to zero).
Optionally, look at the sub-optimality behaviour of the bonus: is the error higher or lower than the variance? How does it affect the agent's curiosity?

Finally explain how the mean and variance are learned, here with a two-headed network trained by maximum LLH.
Also, discuss what happen if the action are taken into account in the learned model (so far the bonus is on-policy and this should appear somewhere and be discussed). Here everything works as long as the agent is not changing his policy, but depending on the learning dynamics of the policy, the model may never converge, this would not happen with action taken into account.




summary_of_the_review: (+) Nice and novel idea
(-) Too poor mathematical justifications and unclear description of the method.

I sincerely hope the paper will be improved for a further submission because I believe the idea has an high potential, that would be missed if it appears in its present slovenly form.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper extends the works on curiosity in artificial agents by incorporating an explicit prediction of (non-reducible) aleatoric uncertainty into the agent’s action choice computation, such that the agents have a preference against the environmental states where uncertainty cannot be reduced by learning. The proposed framework has enabled efficient exploration in a set of tasks where non-reducible uncertainty has been introduced; the authors have also used it to propose a test to clarify the role of acetylcholine – the neurotransmitter related to uncertainty – in the brain.
main_review: *Strengths*

The paper provides a very nice, comprehensive review of the existing curiosity techniques, which was a pleasure to read.

I like the neuroscientific inspiration of the proposed algorithm. The field is full of claims about neuro-AI synergy but short of good examples. This work, further developed, may become a good example of such synergy.

The idea of explicitly accounting for aleatoric uncertainty in deciding on future actions seems highly meaningful and, potentially, fruitful.

In this formulation, the uncertainty is estimated with a single agent and not with ensembles of agents, as it was done before. This makes the model more useful.

The authors have performed an impressive set of experiments, involving a significant number of baselines.

I like the idea of proposing a test, which experimental neuroscientists may then use to improve our understanding of the brain.

*Weaknesses*

Provided the close relation of the proposed method to prior work (as mentioned by the authors), it would be reasonable to expect a strong Results part of the paper (see below).

Most of the experiments in this paper were performed in environments specially altered to introduce stochastic traps. The framework’s performance in unaltered environments did not exceed that of the baselines.

The exploration has been previously developed as a tool with a final goal of maximizing rewards in scarce environments. In this work, the authors have reported improvements in exploration but there are no experiments indicating improvement of total reward in standard unaltered environments (the authors do explicitly discuss this issue). It is unclear whether additional exploration, as suggested here, leads to higher rewards in the standard or real-world environment, which is the goal of exploration in the first place.

In the proposed experiment for establishing the role of acetylcholine in the brain, the prediction seemingly boils down to the fact that epistemic uncertainty decays over time, whereas aleatoric uncertainty remains constant. These predictions seem to directly follow from the definitions of such uncertainties, so it is unclear whether the proposed design of the experiment is necessary.

*Specific comments*

In Equation (5), is the last sign (+) correct?

Although the notation is clear, consider introducing the variables before using them.

Whereas the introduction is generally great, it may be useful to talk a bit more about Angela Yu and Peter Dayan’s model, as it forms the main neuroscientific motivation for the proposed framework, so that the readers won’t have to read their paper.

I would personally consider adding a short description of how Equation (5) was derived. It’s pretty clear for me, but may or may not be clear for everyone. The derivation is super simple and won’t take up a lot of space.

*Suggestions to the authors*

I found this work highly promising and would love to see future developments based on your framework. Perhaps, something along these lines:

It would be great to see the improvement of the received reward in standard or real-world environments. The claim here, supported by the literature, is that stochastic traps are the major problem in curiosity-driven exploration. This implies that there are enough standard environments where stochastic traps affect the efficiency of baseline curiosity-based algorithms. Such environments can be used to showcase the benefits of the proposed algorithm when it comes to rewards which, consequently, would skyrocket the impact of this work.

It would also be nice if the author follow up on their proposed acetylcholine experiment and collaborate with experimental neuroscientists to test this hypothesis. Determining that acetylcholine encodes one type of uncertainty or another would be an important result and another great way to ascertain the high impact of this work.

summary_of_the_review: An impressive amount of work has been done; the writing is mostly easy to follow, and the results seem promising; however, provided that the proposed framework is closely related to the existing ones, further research towards practical results is needed. The better exploration, proposed here, is important, but it may or may not lead to better-performing agents. I, therefore, think that the paper is better suited for a conference workshop rather than for the main track. Thus, overall, I tend to recommend rejection.
review_title: null

Meta-review: Authors present a method to disentangle epistemic from aleatoric uncertainty for avoiding the noisy TV problem during self-driven exploration. This is an important area where we need more ideas and experiments. The authors present a biologically inspired approach and through experiments. Although it doesn't present the state-of-the-art exploration in well-known RL environments, I acknowledge that new solutions to problems that were previously intractable often would face such an issue. The prediction to discriminate neuroscientific modulations that directly encode epistemic and aleatoric uncertainty is bold but not very specific. Unfortunately, as the reviewers noted, the manuscript in the current form doesn't quite meet the bar yet. I suggest comparing methods for directly estimating uncertainty. I also suggest adding discussion on the estimation bias for the epistemic uncertainty for the proposed method. I strongly encourage the authosr to continue this interesting line of work.
Decision: Reject