Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 10: strong accept, should be highlighted at the conference
review_content: summary_of_the_paper: In the paper "Sample Efficient Deep Reinforcement Learning via Uncertainty Estimation", a group of algorithms called "Inverse Variance Reinforcement Learning" is proposed that leverages different estimations of uncertainty to guide the loss of value-functions (and actors in actor-critic settings). IV-RL is evaluated for IV-DQN and IV-SAC in several benchmarks.
main_review: This paper is exceptionally well written and a pleasure to read. I literally found no weaknesses and strongly recommend acceptance.

Strengths:

1. It is an important research topic and of very high interest to the community.
2. The approach is very well embedded in the current literature.
3. The paper is very easy to follow. It is theoretically sound and for all critical aspects there are intuitions and explanations.
4. The evaluation is done very thoroughly (results over 25(!) seeds, hyperparameters are provided).
5. The benefits of IV-RL are shown in both the discrete and continuous setting.

Weaknesses:
None

Remark:
I would be interested in a more extensive discussion of policy degeneration in the LunarLander environment.
summary_of_the_review: I literally found no weaknesses and strongly recommend acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper presents inverse-variance RL, which combines an ensemble of probabilistic neural networks with batch inverse-variance weighting. The former provides a method for uncertainty estimation and the latter provides a method for incorporating that uncertainty, by multiplying the loss of each entry in each minibatch by the (normalized) inverse of its variance. Experiments on simple discrete and continuous control environments from the OpenAI Gym show that the resulting method, when added to DQN and SAC outperforms these baselines and SUNRISE.
main_review: **Clarity.**

The paper is well-written and fairly clear in its justification, background, and empirical results. However, the lack of clear math and pseudocode applying the method to the RL setting makes it unclear as to what exactly the method is and how to implement it. For the next iteration of the paper, this information is necessary. Including only equations for supervised learning (i.e., equations 1, 3, 10) for one of the main contributions of the paper is insufficient in an RL paper. Minor: The math in section 3.1.2 seems clear and correct, although should there be an $s,a$ in the RHS conditioning of equation 8?


**Novelty and significance.**

The main components of the proposed method (i.e., down-weighting by variance and using probabilistic ensembles) are well-known to the community, as cited already by this paper (e.g., Lee et al, 2020, Wu et al, 2021, and Lakshminarayanan et al, 2017). The particular proposed method for down-weighting is taken directly from supervised learning and applied to RL as a method for the above combination. Thus the novelty is just the combination of these, which is insufficient in my view to merit acceptance. The analysis of the sources of uncertainty (sec 3.1.2) is perhaps novel (I did not do a thorough literature review given the other issues with the paper) but insufficient to merit acceptance. 

Despite the lack of novelty, if the theoretical or empirical results were strong, then acceptance could be warranted. However, the empirical evidence here is limited and thus does not tip the balance. The environments are all very similar and none are particularly challenging, so it’s unclear the results will hold for a broader range of environments (for example, only 1 of the 3 discrete environments actually improve due to BIV). Further, if the argument is that the normalization used in BIV is crucial relative to UWAC, then I’d like to see this explicitly. I’d similarly like to see a more detailed set of experiments comparing the weighting schemes of UWAC, SUNRISE, and BIV (with all else identical) to know how they compare. 

**Question:** how does variance down-weighting interact with the benefits of optimism in the face of uncertainty in RL? Down-weighting by high variance elements of the batch could be interpreted as a form of pessimism, which would run counter to this.

summary_of_the_review: Overall, this paper lacks sufficient novelty for acceptance. It recombines a set of existing techniques in a slightly different way than has been done before and shows that this seems to improve performance on the small set of fairly similar domains presented. However, for such a minor change from what has come before, it lacks sufficiently strong theoretical or empirical evidence for acceptance.


----------

After rebuttal, the majority of my concerns have been addressed by a comprehensive update to the paper. To reflect this, I have updated my scores to accept.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper is about uncertainty quantification in Q-Learning/Actor-Critic arquitectures. The authors deal with the problem of noise in the targets from where the Q function or value/critic function is learned, by estimating their uncertainty (aleatoric mostly) during the training process, and the authors propose to use this uncertainty with batch inverse variance weighting (BIV), in order to weight samples by inverse variance.

This idea is simple and yet quite powerful, as results show that using target uncertainty and BIV, DQN and SAC models are able to learn faster and improve their sample complexity by reaching the same average return with less number of episodes in a series of standard RL benchmarks for discrete and continuous environments.

main_review: Strengths

- The proposed technique is theoretically sound, and I think it makes a lot of sense to use label uncertainty (aleatoric) in Reinforcement Learning in DQN/SAC in particular, and this improves sample complexity as more information is extracted from the training process.
- I believe that it is very important to work on improving sample complexity of Reinforcement Learning, as it is the most important challenge in order to apply RL to real-world problems.
-  There is a decent improvement in sample complexity when applying BIV to DQN and SAC with no drawbacks. In two of the tested discrete environments (LunarLander and CartPole noise) there is a clear improvement, while on MountainCar there is no improvement but also the method works with same performance as the baseline. For continuous environments, in three out of four there are clear improvements in sample complexity, while ant performance is kind of the same than the best baseline and slightly down. In all cases the IV-RL methods outperform the sample complexity of the original variations without IV-RL.
- The proposed IV-RL method seems to be an improvement without drawbacks (except for additional hyper-parameters), and overall I think the idea of estimating aleatoric uncertainty in the Q-function being learned is a great idea, as this extracts additional information from the training process, the information seems to be there and it should be exploted as standard part of the Q-learning or SAC training process. This additional information is extracted using a heteroscedatic Gaussian negative log-likelihood loss which can estimate aleatoric uncertainty of the targets without supervision.
- The authors perform ablations (Figures 3 and 5) that shows the effect of different components in their proposed network, namely if using ensembles, inverse variance weights, and aleatoric uncertainty quantification through the Gaussian NLL loss.

Weaknesses

- One important detail that I think was left out the paper is the effect of the multi-task loss parameter lambda (Eq 10), if I am not mistaken, there is no information on the paper to which value this parameter is set. The supplementary mentions that it is tuned using grid search, and only the range of this parameter is provided, not the actual value that was used to obtain the results in the paper or ablations to show the effect of this parameter on performance. This is important information for the reader and reproducibility that should be included in the paper or supplementary.
- A second important issue is Eq 10, where the authors claim that the loss function they use is a standard linear combination of the BIV and LA losses, and the BIV loss is defined in Eq 1. This equation shows that the BIV loss contains another sub-loss term ( L (f(xk , θ), ỹk) ) where I assume that it is a squared error loss like the one in Eq 5, but then Eq 3 which shows the LA loss also contains a squared error weighted with the inverse variance (without the ξ term), so it would not make much sense to combine the BIV and LA losses like that directly, I think there is a missing detail here or the author's proposed loss has two squared error terms which can be inconsistent or strange. Please clarify.

Minor Points

- I think that Sections 2.2.1, 2.2.2 and 2.2.3, I think the authors narrative about deep ensembles being sampled ensembles with variance prediction is incorrect, as the original deep ensembles paper (Lakshminarayanan 2017) does not use Dropout or Masks in the ensemble network members, and there is the inconsistency that the papers the authors argue are about sampled ensembles are from 2020, while the deep ensembles paper is from 2017. Also I do not like the term "probabilistic networks" and "probabilistic ensembles", as it is not clear here what makes them probabilistic, I think better terms are for probabilistic networks are variance networks (as they output mean and variance of the targets), and plain ensembles instead of probabilistic ensembles. Also please note that deep ensembles for regression uses individual variance networks as ensemble members, so the only difference between an ensemble and a variance network is the number of members in the ensemble, where an ensemble of one network is a variance network. I think these terms will make the paper more consistent with the uncertainty quantification literature, and should also be reflected in the label for Figures 3 and 5.
- I think the paper does not mention how and why the RL environments were selected I think it is better to evaluate on more environments or at least provide a good justification for the environments that were used. Papers like SUNRISE are evaluated on many more environments (ATARI, Gym, Deepmind Control Suite) which ensures that the results and conclusions are robust.
- I think in the labels of Figure 3 and 5, it is better to relate the components of your loss function (Eq 10) and to using ensembles or single networks, instead of using the names of probabilistic networks/ensembles, this connects also with my previous comment about Sec 2.2.2 and 2.2.3.
- I would like to point the authors to another paper submitted to ICLR 2022 ( https://openreview.net/forum?id=aPOpXlnV1T ) which deals with practical issues in aleatoric uncertainty estimation using the Gaussian NLL loss for regression, which could help improve their method. In that paper the authors also propose a weighting for each loss term based on their variance.

The issues pointed above have been addressed by the authors in their newest revision.
summary_of_the_review: I believe that this is a good paper, after rebuttal the issues I mentioned have been resolved. I like the idea, it is an important problem in RL, it seems to be correctly evaluated, and it will be a good contribution to the RL community as aleatoric uncertainty is generally not modelled in RL formulations.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper incorporates inverse variance weighting and probabilistic ensembles to account for sources of uncertainty present in the standard TD and Actor Critic learning paradigms. The authors present a systematic analysis of these sources of uncertainty and show how TD learning contributes heteroskedastic behavior, motivating the use of IV weighting of the samples used in agent training. Utlimately, a combination of complementary approaches is used to account for the variance in returns encountered in both the environment and approximation of the Q-Network. The proposed IV-RL algorithms are evaluated in both simple continuous control as well as some MuJoCo tasks, with favorable performance improvements over related prior approaches.
main_review: Overall the paper is very well written with clear motivations for the modeling choices used to develop the proposed IV-RL algorithms. The contributions listed at the end of Section 1 are each interesting and valid improvements to the current literature seeking to develop robust Deep RL paradigms. The paper covers a lot of supporting material used to ground the proposed approach and appropriately covers sufficient technical details to appropriately frame the contributions put forward. In particular, I enjoyed the connections drawn to frame TD learning as regression on heteroskedastic noisy labels. This insight drove home the necessity of using BIV to account for the uncertainty induced by approximating the Q function.

*This however leads to the first major weakness in the paper.* It is unclear, as presented at the end of Section 3.1.2, what exact quantities or network structures (ensembles and etc.) are used in BIV and LA exactly. As good as the framing and motivation are, there is a major lack of specificity or explicit formulation of the objectives used to learn the parameters of the policies. The paper would be significantly improved if more careful and direct development of the IV-RL loss was presented. In particular, what forms the estimate for $\sigma_{\bar{Q}}^2$? Is it an ensemble of networks? How does that connect to the various terms in Equation 1? What is the Loss function in the BIV Loss? Is it the standard TD-error? In the end, it would be very helpful if the separate components which were introduced in context of the grounding literature were extended to more explicitly communicate how they fit into the DRL pipeline. Clarifying this component is one of two major weaknesses that, if sufficiently addressed, will motivate me to raise my score.

A minor weakness connected to the lack of clarity just described is that the description of the various baselines are quite vague. It's not clear what is meant by a "non-IV version of the improvements of IV-RL" nor what the ablations in Figure 3 are really showing. The abbreviations/acronyms included in Figures 3 and 5 are not that informative. Again, being more specific here will make the following experimental analyses much easier to understand. 

*The second major weakness* that I want to point out is that there it is not abundantly clear what the major emphasis of IV-RL is. At the outset of the paper the Introduction positions the work as different from the DRL literature harnessing uncertainty to investigate exploration/exploitation tradeoffs, emphasizing that this work focuses instead on the quality of supervision while maximizing the expected reward. However, much of the discussion in Section 4 (Experimental Results) uses concepts of exploration or lack thereof as a justifying reason behind some of the presented performance. This inconsistency dilutes the major contributions and significance of the paper as outlined in Section 1. This lack of precision affects the major takeaways and is evident in the Conclusion as there are very few concrete points that motivate further use of IV-RL. I was hoping to read what the authors felt their proposed IV-RL approach was particularly well suited for, what types of problems they feel are easier to solve because of the advances put forward in the paper. In addition to more concrete conclusions, it would be really nice if the authors were able to express limitations to the proposed framework as currently implemented. Are there particular inefficiencies and computational overhead that need to be considered before implementing IV-RL? What other weaknesses are there? 

Beyond these specific points I do want to commend the authors for their extensive and rigorous empirical analysis when comparing to baseline and ablated algorithms. The presented results are convincing and there is some decent discussion (it could be sufficiently tightened a little bit as discussed above) to elaborate the findings. Overall, it appears that the proposed IV-RL algorithm works quite well and improves over prior approaches.
summary_of_the_review: Fantastic framing and decomposition of the sources of uncertainty present in learning an RL policy. The paper introduces inverse variance weighting to RL and does so in an interesting and well motivated manner. Despite some weaknesses in firmly anchoring the foundational pieces of the proposed IV-RL approach, the empirical results are shown to improve over comparative algorithms across a variety of domains. As expressed in my main review, greater clarity about the contributions of each component of the IV-RL loss, framed in a more specific terms, as well as more consistent focus on the proposed contributions throughout the paper will help me raise my assessment from borderline acceptance to full acceptance.
review_title: null

Meta-review: The reviewers unanimously appreciated the clarity of the work as well as the framing of the proposed method. Congratulations.
Decision: Accept (Spotlight)