Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: This paper proposes to use uncertainty estimates from an ensemble of action-values, to provide a weighting on the updates in Q-learning. The main idea is to use the sigmoid of the negative of this uncertainty in the next state, to produce a weighting between 0.5 and 1 to downweight updates with high uncertainty targets. This uncertainty estimate from the ensemble is also used to improve exploration, in a combined algorithm called Sunrise that leverages learning an ensemble in these two ways. 

The idea of using weighted Bellman updates is, as far as I know, novel. The evidence for the idea, however, needs more work. First, the weighted update in Eq (4) is not motivated from first principles. Second, the empirical evidence is weak because the experiments highlighting the role of the weighting do not demonstrate significant differences.  

The first issue is the justification for the approach. The ensemble of Q-learning agents is trained using the weighting, derived from that ensemble. There are natural questions as to the interaction between the ensemble uncertainty estimates and the ensemble estimates. Does it result in any instability? What is the final point of convergence? Does it change the solution?

But, one could argue that that is not much of a problem, since the weighting w(s,a) is always between 0.5 and 1, so it is not that skewed. Then the question arises how much it is helping, and why this small reduction in weight helps. This is particularly important to ask, considering the algorithm requires an ensemble to be learned, with subsets of data used for each action-value. There is a lot of effort expended for that weighting.   

The experiments then do include ablations, to examine the effect of these weightings. Unfortunately, the results are inconclusive. The experimental time spent must have been large to get all the results in this paper, across so many environments and algorithms. But, the ablations themselves are not sufficiently in-depth to provide insight into the idea and algorithm. The results in Figure 2 are key, since that figure examines Sunrise with and without the weighting. Due to the variance across runs, with only 4 runs, there are large standard errors (and so even larger 95% confidence intervals); it is hard to conclude that weighting is helping. The additional results in Figure 5 in the appendix have a similar issue.

The results in Figure 3, which motivate the exploration utility, are more clear in Cartpole. This provides some motivation for learning ensembles, so they can be used for exploration. But, this exploration approach with ensembles is an existing method. The main novelty in this work is the weighting. 

I highly recommend taking a few domains and carefully studying the impact of the weight. More runs would help for significance, as well as parameter sensitivity analysis to gain insight into the generality of the improvement. Sometimes performance gains are from hyperparameter tuning, rather than from the utility of an idea; here, you really want to know if and why this weighting improves performance. 

As a more minor comment, Sunrise is pitched as combining three ideas for using ensembles: your weighting, bootstrapping and UCB exploration. However, I see Sunrise as combining two ideas: weighting and UCB exploration. The Bootstrap DQN approach gives you a way to learn your ensemble of bootstrap models, so that it provides a useful uncertainty estimate. Given that ensemble, you can then use it to compute a weighting and optimistic action. It would be more clear to separate it out that way, rather than saying "Furthermore, since our weighted Bellman backups rely on maintaining an ensemble, we investigate how weighted Bellman backups interact with other benefits previously derived from ensembles: (a) Bootstrap; (b) UCB Exploration." The bootstrap is arguably not a benefit, but an approach to obtain confidence (uncertainty estimates). 
 
Minor comments:
1. Bootstrap DQN is listed under "Ensemble Methods in RL", rather than under "Exploration in RL", but is it an exploration approach.
2. "Recently, Kumar et al. (2020) showed that this error propagation can cause inconsistency and unstable convergence." The terms inconsistency and unstable convergence should be explained, since they seem like technical terms. 
3. Bellman backup seems to be used to describe the squared error to the expectation over next action, in Equation (2), and then to a stochastic sample of the action in (4). Which is it?
4. What is meant by the signal-to-noise in Q-updates? 
5. A natural baseline to include is to tune an agent that uses random weights between 0.5 and 1 in the update, but keeping other parts of Sunrise the same. The ablation removes the weighting all together, which is also important to include. But, it's worthwhile observing if random weights performs similarly, especially if that agent is tuned. 

------------ Update
Thank you for the clear reply. Unfortunately, I remain concerned about the significance of experiments. I mentioned above that 4 or 5 runs is typically not enough, and because the standard errors are overlapping, the differences could be due to chance. The addition of a result with 10 runs is a good step. But, as part of the reply, the authors state: "Figure 3(a) shows the learning curves of all methods on the SlimHumanoid-ET environment over 10 random seeds. First, one can not that SUNRISE with random weights (red curve) is worse than SUNRISE with the proposed weighted Bellman backups (blue curve). Additionally, even without UCB exploration, SUNRISE with the proposed weighted Bellman backups (purple curve) outperforms all baselines. This implies that the proposed weighted Bellman backups can handle the error propagation effectively even though there is a large noise in reward function." However, if you look at this figure, the error bars all still overlap. 10 random seeds is still not enough. 

I am also not confident that the issue will be remedied, as the authors additionally state in the rebuttal: "we believe that SUNRISE is evaluated in a broad collection of domains in the RL literature and the performance gap is also noticeable." An insignificant gap across many domains does not tell us anything. Actually, if you take the runs and tried to do significance tests by pooling all the runs across environments, then maybe the result might actually be significant. But, of course, there will be higher variance due to differences in the environments, so it is not obvious this would be true. Nonetheless, this could be a natural next step.
review_title: Official Review for Reviewer3

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: = Overview = 

The paper proposes SUNRISE, an approach to reinforcement learning that leverages ensembles of agents to build more robust RL updates. SUNRISE comprises a number of similar agents  (in the paper, SAC agents) that perform parallel updates. Sample transitions for which there is larger variability (across the ensemble) in the estimates of the next-step Q-values are down-weighted in the computation of the loss, thus potentially rendering the learned Q-function more robust to noise.

The proposed approach is then combined with bootstrapping masks and UCB exploration, and is shown to outperform a number of state-of-the-art approaches in several benchmark domains from the RL literature.
 
= Positive points =

The paper is clearly written. Additionally, the proposed approach is sensible and the empirical evaluation is, in my perspective, quite comprehensive: SUNRISE is evaluated in a broad collection of domains in the RL literature. 

= Negative points =

The paper would benefit, in my opinion, from additional discussion regarding: (a) the impact of the use of bootstrap with random initialization; and (b) the computational complexity of SUNRISE (even if the paper does briefly discuss the latter in Section 5.2)

= Comments = 

I quite enjoyed reading the paper. The problem addressed is a relevant problem in RL, and the approach proposed in the paper is, in my opinion, simultaneously simple and sensible. The paper provides a solid empirical evaluation, covering a broad range of domains and comparing with multiple state of the art approaches from the literature. The results show that SUNRISE compares favorably -- in terms of performance -- with several of these other methods in multiple domains.

There are, however, two aspects that I would like to see discussed at greater length. On one hand, the paper proposes the use of bootstrapping masks and random initialization to induce variety in the ensemble. While the paper introduces both bootstrapping and UCB exploration as a "useful complement", it seems to me that this is quite central to the performance of the algorithm. Is this correct? In fact, without this device, the agents in the ensemble would essentially train from the same replay buffer, so variability would only come from the initialization. It is a pity that this particular element isn't included in the ablation study, for I would like to gain a clearer understanding on how critical this device is for the performance of the algorithm.

One other aspect that I would like to see discussed is regarding the computational complexity of the proposed approach. The paper remarks that SUNRISE is more computationally efficient than competing methods such as POPLIN and PETS, and being an ensemble method, I expect it to be naturally heavier than non-ensemble approaches such as standard SAC. However, I would like to understand how much more computation such a method involves. In particular the computation of the Bellman weights requires multiple passes through the critic network, as does the UCB exploration policy, and I was wondering how much more computation this entails.

In spite of the above aspects, I again remark that I quite enjoyed the paper.
review_title: The approach proposed in the paper is interesting and the results suggest that it is successfully able to outperform current state-of-the art approaches in several benchmark domains from the literature.

final_score: rating: 5: Marginally below acceptance threshold
review_content: This submission developed an ensemble based approach to weight the Bellman backups from different agents. As claimed by the authors, the proposed method can improve the signal-to-noise ratio. To further boost the performance, the authors combined the weighted backups with a few other techniques, including UCB exploration and Bootstrap. The authors finally tested the proposed method on both continuous and discrete reinforcement learning tasks, and showed the improved or competitive performance, compared with baselines.

The proposed ensemble-based approach is interesting and the authors conducted an extensive experiments to verify its empirical performance, which I really appreciate. Compared with other baselines, the performance gap is also noticeable. 

On the other hand, I am a bit concerned about whether the improvement is indeed because of the weighted backups. For example, Figure 3(a) showed that if removing UCB, the performance for SUNRISE dropped a lot. I tried to see if there are any other ablation studies w.r.t. UCB on the results in tables (removing UCB and keeping other steps the same in Algorithm 1), but did not find them. Use of UCB seems orthogonal with the weighted backup, as one is focused on exploration and the other for Q updates. Therefore, it's a bit questionable whether UCB or the proposed weighted backups is the main factor for performance improvement.

The authors claimed a few times "signal-to-noise ratio". I hope there could be more rigor here. What exactly is the definition for this term? What are the signal and noise here?

Furthermore, I also doubt about the fairness in Table 3: The results there are only for 100K interactions; however, when comparing with Figure 8, Rainbow has not become stable at 100K and the scores for some games are just too low (e.g., Breakout), compared with results in the Rainbow paper.

Could you comment on the increased complexity, when employing multiple agents? There are a few recent papers on the weighted Q updates as well, e.g.,

Song, Z., Parr, R. and Carin, L., 2019, May. Revisiting the softmax bellman operator: New benefits and new perspective. In International Conference on Machine Learning (pp. 5916-5925).

Kim, S., Asadi, K., Littman, M. and Konidaris, G., 2019, August. Deepmellow: removing the need for a target network in deep Q-learning. In Proceedings of the Twenty Eighth International Joint Conference on Artificial Intelligence.

These papers avoid the need of multiple agents and show the benefits of weighted updates, which the authors need to discuss.
review_title: Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: ### Summary   

This paper proposes to weight the Bellman backups according to the empirical std of Q-functions estimated by ensemble method. The paper claims that this proposed idea stabilizes and improves the learning in both continuous and discrete control tasks. It then integrates this proposed weighted Bellman backups with two of the existing advantages of ensembles: bootstrap and ucb exploration to form a unifying framework namely SUNRISE. SUNRISE is then compared with Actor-Critic and Rainbow in both discrete and continuous control tasks.  

### Strong points 
-	Clarity: The paper is well written and organized 
-	Empirical significance: The integrated framework SUNRISE appears to have promising (yet seemly not fully convincing) performance as compared to the prior frameworks in both discrete and continuous control tasks.

### Weak points
-	Novelty: This work seems to have low novelty and low technicality. It combines several known results to integrate into a unifying framework using ensembles. The only new idea here is perhaps a specific way of reweighting the Bellman backups though the idea of reweighting the Bellman backups to stabilize the learning is already known e.g., Kumar et al. 2020. 
-	Significance: In addition, at the present form I find it hard to be convinced both empirically and theoretically (or at least more elaborate explanation or intuition) why the proposed weighted Bellman backups using empirical std of Q-functions improve the signal-to-noise in Q-updates as claimed in the paper (see Questions for the authors).   

###  Questions for the authors 
-	In the last sentence of Section 4.1, the paper claims that “the proposed objective … has a better signal-to-noise ratio”. I would like the authors to elaborate in this claim. What is exactly considered signal and what is exactly consider noise in this context? Why down weighting the sample transitions with high variance across target Q-functions result in a better signal-to-noise ratio? Why does the weighting proposed in this paper have better signal-to-noise ratio than the weighting in Discor (Kumar et al. 2020)? 
-	The paper claims that the proposed weighted Bellman updates improve signal-to-noise in Q-updates but appears to show only one experimental setting (presented in Fig. 2) where the reward is perturbed with *a standard Gaussian noise*.   For simplicity for the moment let’s call by “reward-to-noise ratio” the ratio of the magnitude of the original reward signal r(s,a) to the magnitude of the added noise. Since Section 5.3, I assume that the “reward-to-noise ratio” has something to do with the signal-to-noise ratio mentioned in the paper. Then, how the performance of the proposed weighted Bellman updates when the “reward-to-noise ratio” varies?    

###  My initial recommendation
Overall, I vote for weak rejecting for the weak points mentioned above. 

### My final recommendation 
The authors did not fully address my points. I remain my initial score and recommend for rejection. 

review_title: An encouraging empirical result but low technical novelty and insufficient experiments to make a reliable conclusion about the role of the proprosed weighted Bellman backups

Meta-review: The paper is acknowledged by all the reviewers as making a novel contribution -- the proposal to reweight state-action pairs depending on the variation in their Q-value estimates during learning. However, despite its extensive reporting of numerical experiments, its arguments in favor of the proposed approach are found to be wanting on both empirical and theoretical fronts. Reviewer 3 points out (correctly, in my opinion) that 5 (or even 10 in the updated version) independent trials are not sufficient to establish the validity of the approach up to statistical significance, and that even a well-reasoned heuristic explanation of why reweighting is expected to work in terms of reducing Q-value estimation error is missing. I agree with this point, which also struck me when reading the submission myself, that at the very least, the submission ought to contain a basic (and not necessarily rigorous) argument as to why the variance reduction ostensibly achieved due to reweighting should lead the estimation algorithm to the right Q-function in a general function approximation setting. For instance, even in the simplest multi-armed bandit setting, it is of interest to ask why this procedure should perform consistently without introducing unwanted bias in an unforeseen sense, and a clear explanation offered for this would be interesting. Another important concern that most reviewers are left with is about the lack of sufficient insight into the action of the UCB mechanism against the backdrop of the reweighting procedure (reviewers 1, 2, 3). I hope that the author(s) assimilate the feedback to strengthen the paper's main pitch further and make a strong case in the near future. 
Decision: Reject