Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: 629
This paper studies data efficiency and asymptotic convergence rate of deep reinforcement learning and proposed a new algorithm called AQE and claims the algorithm are both fast in both senses. The paper is mostly empirical with results on Mujoco, and compared with TQC and SAC and showed competitive performance. There is also a theorem about the update estimate bias, which tells the orderings of the bias for when K or N is increasing. 

I've read the authors' feedback and other reviews. I think the paper goes hard as the asymptotic rate in experiments is not supported well. The theorem is not helping much for the two claims. There are also other problems (presentation, algorithmic discussions and comparisons) that can be much improved. Keep trying!
main_review: 
Questions:
All Line9: how is the K determined? Do you mean take K minimal Q values from the complete set of N such Q values? Why do you take K minimal Q values for computing the target? Usually the Q-learning style algorithms takes the maximum. I found this is a mysterious part of the algorithm. 

Follow up the previous question, have you studied taking average from all N Q targets? The maximum of the N or K Q values, and what have you found? In the ensemble papers, people usually takes the maximum of Q values from ensembles, e.g., see eq 8 of 

https://arxiv.org/abs/1811.02696

It is also interesting to see that this paper together with TQC and REDQ all uses critic ensembles. However, the above paper (ACE) uses ensembles from actors. This is an interesting alternative. It is interesting to discuss the connections and strength of both approaches. 


The computation of y in line 10 of Alg. 1 uses log of the policy, this wasn’t explained. Is it a regularization effect? Why is it important for your algorithm? Have you studied this effect? (Ablation studies?)

Interestingly, in the actor update, the whole ensemble is used to average for a Q value estimate. Could you explain the choice of the complete instead of a subset of ensemble here?

The results were run for 1 million frames: this may not be able to support asymptotic behaviour. In particular, in (b)Walker2d, clearly the TQC may possibly be further faster than AQE. Could you provide more frames than shown?

In the theory part, you said we would like E(Z_KN) \appr 0. What does the approximation mean here? Do you want to minimize it? Is it positive or negative? Your theorem 1 shows this is positive, then how bigger is it than zero? Your result 2 in Th1. shows the smaller K is preferred? Is this the message? While result 3 shows when K is fixed, N should be as big as possible? How to explain this intuitively?



Minor:
The algorithm name from Q-learning is a little unconventional. There is a policy network, which is usually not called a Q-learning style algorithm. 

Sec 3:
The averaging over the ensembles for Q estimates is not Polyak averaging. Polyak averaging refers to temporal averaging over the past. Here your average is nothing about time. 


summary_of_the_review: It appears an interesting paper, but the algorithm wasn't explained with much detail which especially didn't address critical components. 

The results show sample efficiency. The asymptotic rate is not very much supported. 

The theory's message isn't very clear.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes Aggressive Q-Learning with Ensembles (AQE), which improves the sample-efficiency performance of TQC and the asymptotic performance of REDQ, thereby providing overall state-of-the-art performance during all stages of training. Empirical studies verify the effectiveness of AQE.
main_review: [Strengths]:
1. AQE is simple and effective. Implementation requires little change to REDQ.
2. Empirical studies show that AQE achieves a much higher average return compared to SAC, TQC and REDQ in most environments.

[Weaknesses]:
1.	One of the major concerns is the similarity of this paper to a prior work REDQ. The similarity is of various aspects, including the algorithmic tables, sentences, results, analyses and even the format! On top of that, the main idea is actually very much similar, and as far as I can tell the only interesting difference seems to be the replacement of Q value from the min value of a randomly sampled set in REDQ to the average of the min K values in AQE. The authors also don’t give a convincing explanation why this replacement can improve the performance, either theoretically or intuitively.
2.	The most similar approach REDQ which the authors cited multiple times is not tested in Figure 4 of Appendix B. Compared with the baselines in that figure (SAC-5, TQC-5), REDP sounds much better in terms of reducing the Q estimation error. I am quite curious how REDQ on the bias.
3.	It’s kind of strange that a model with low bias and std provides poor performance sometimes. For example, in figure 3, the orange line (K=10) has lower bias and std than the red line (K=16) but provides worse performance. The same phenomenon appears in Figure 5 (g) (between the blue line and red line) and Figure 5 (j) (orange line and red line), etc. The authors don’t provide an explanation.
4.	Have the authors tried to adjust K dynamically, such as using a small K in the early stage to avoid overestimating and use large K in the late stage to reduce the estimation variance?

[typos]
Not sure whether or not the authors intended to say “AQE improves the sample-efficiency performance of TQC and the asymptotic performance of REDP” in the abstract? Because TQC provides SOTA asymptotic performance while REDP achieves high sample efficiency...

summary_of_the_review: Overall I think the contribution of this paper is not significant to the high standard of ICLR. I am leaning to the rejection.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: **Summary**

This work introduces Aggressive Q-learning with Ensembles (AQE), a simple model-free reinforcement learning algorithm. AQE can be considered as a modified version of the Randomised Ensembled Double Q-Learning (REDQ) algorithm. Specifically, AQE uses a number ($1 \leq K \leq N$) of the ensemble members ($N$) with the lowest Q values for calculating the target, instead of sampling a number ($1 \leq M \leq N$) of ensemble members and keeping the one with the lowest Q values as in the case of REDQ. Similarly to REDQ, AQE also employs a high update-to-data ratio ($G > 1$) to improve the sample efficiency. Empirical analyses have been conducted on five MuJoCo environments showing that the performance AQE is competitive to that of Truncated Quantile Critics (TQC) and improves the sample efficiency performance of REDQ. 
main_review:  **Pros**

* The proposed AQE algorithm is simple and seems to be effective based on the presented empirical results.
* In general the paper is well written. The proposed algorithm and the empirical results are presented in a clear way.
* Empirical analysis on five MuJoCo environments shows that AQE is competitive to TQC and improves the sample efficiency performance of REDQ. 
* An extensive ablation study is provided, shedding light on the impact of the three free parameters ($K, N, G$) in the performance of AQE. 

**Cons and Comments**

* The novelty of this work is a little bit incremental as it can be considered as a simple modification of the Randomised Ensembled Double Q-Learning (REDQ) algorithm.
* The connection between AQE and REDQ is not clear for someone that is not familiar with the REDQ algorithm. Actually, the reader should check the paper of REDQ first to understand its connection to the AQE. 
* The number of K (first row) and N (second row) should be also presented at Figure 3 that illustrates the results of the ablation study. In general, figures/legend should contain all the necessary information.
* What is the performance of AQE in the extreme case where $G=1$? According to the paper $G$ should  be greater than $1$ ($G>1$), but it would be really interesting to check the impact of updating the model multiple times on the performance of AQE. 
* What is the impact of the hyper-parameter $\rho$ on the performance of AQE?
* How will the replacement of SAC with another off-policy algorithm affect the performance of AQE? 

*Minor Typos*

* p. 4: infeasible in our our 
summary_of_the_review: In general, the proposed model-free reinforcement algorithm, called AQE, is simple and its performance is competitive with that of other baselines on five MuJoCo environments. An ablation study is also provided at the same time. My main concern as regards this work is its novelty. As aforementioned, AQE constitutes a slight variant of the already existing REDQ algorithm.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors show that the asymptotic performance of  REDQ is improved by replacing the minimum of Q network outputs with the average of the multi-head Q network outputs.
main_review: Strengths: 
+ source code to replicate the proposed method is provided. 
+ the proposed method is simple. 
+ the performance (sample efficiency) of the proposed method is good. 

Weaknesses: 

- Incremental improvement from the REDQ paper: 

1. AQE algorithm (Algorithm 1) is almost the same as REDQ algorithm. 
The main difference between AQE and REDQ is that AQE uses (1) the average of their outputs (i.e., model averaging) to target and (2) multi-head Q functions. 
The idea of using multi-head functions instead of ensembles is not brand new [1]. 
And the idea of using averages Q-estimates as targets is not new [2]. 
IMO, the authors' primal contribution is to show that incorporating these into REDQ improves performance. 
However, it is not clear whether the performance improvement shown in the experimental results (e.g., Figure 1) is very meaningful. 
For example, is it difficult to achieve the AQE performance obtained in the experiment simply by tuning the REDQ's hyper parameters, policy and Q functions architecture?

2. the theoretical analysis (the flow of the proof of Theorem 1) is almost identical to the one done in the REDQ paper.

[1] Stefan Lee, Senthil Purushwalkam, Michael Cogswell, David J. Crandall, and Dhruv Batra. Why M
heads are better than one: Training a diverse ensemble of deep networks. CoRR, 2015.  
[2] Oron Anschel, Nir Baram, and Nahum Shimkin. Averaged-DQN: Variance reduction and stabilization
for deep reinforcement learning. In Proceedings of the 34th International Conference on
Machine Learning, pp. 176–185, 2017.

- Theorem 1 and its proof are not correct:  
At the beginning of pp. 18, the authors states ''Since for any state $s$,  $\max_a \bar{Q} _ {K+1, N}(s, a) \geq \max_a \bar{Q} _ {K, N} (s, a)$ .''  
However, since $ \bar{Q} _ {K, N}(s, a) $ is defined as $\bar{Q} _ {K, N}(s, a) = \frac{1}{K} \sum _ {j E_ {K,N} } Q^j(s, a)$, the above is not an inequality but an equality: $\max_a \bar{Q} _ {K+1, N}(s, a) = \max_a \bar{Q} _ {K, N}(s, a)$  
This means that changing the value of $K$ does not theoretically change the expected value of the estimation bias $  \mathbb{E}[Z _ {k, N}] $.  
The remaining part of the proof also contains flaws.  
For example,  $\bar{Q} _ {1, N} (s, a) = \min _ {1 \leq j \leq N} Q^j (s, a)$ is assumed in the proof of the 4th property of Theorem 1, but it is not consistent with Eq.5 and Algorithm 2.  

Minor comments:  
Algorithm 1:  Adding a more detailed explanation of how the multiple (h) estimates of each Q function are aggregated into a single scalar value in lines 10 and 14 would make the paper easier to understand.  

\>\> algorithmic base, it also employs an update-to-data ratio  
\>\> The REDQ paper uses G = 20 for the update-to-data ratio  
update-to-data -> UTD  

\>\> Hado van Hasselt. Double q-learning.  
Double q-learning -> Double Q-learning  

\>\> In International Conference on Machine Learning, 2020.  
\>\> In International Conference on Learning Representations,  2021.  
In Proceedings of ...  
summary_of_the_review: I recommend rejecting the paper.
A method that performs better than REDQ is beneficial from an engineering perspective. 
However, as noted above, the innovation made in the paper is incremental from what made in the REDQ paper. 
Also, the paper contains flaws in the theoretical analysis section that should be substantially revised.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a new method for mitigating the overestimation bias in Q-learning-like algorithms. An ensemble of critics of size N is trained while only the small K individuals are used for computing the bootstrapping targets. Empirical results are provided to support the claim.
main_review: My major concern is that the empirical results are not informative at all.

As shown by Figure 5 of Henderson et al. (2018), 5 random seeds are clearly not enough for empirical comparison. Consequently, I cannot draw any conclusion from the figures or convince myself of any conclusion the authors draw from the figures. The tables are even worse since they do not contain any confidence estimation. I am disappointed in that the authors indeed cited Henderson et al. (2018) regarding the concern of reproducibility of RL research but still consider only 5 random seeds.

Moreover, the paper considers only 5 domains. The contribution of this work purely empirical so I would expect the authors to evaluate the algorithms in much more domains, e.g., in DeepMind control suite. I cannot convince myself of any conclusion with only 5 domains, especially when the performance of the proposed algorithms and the baselines are of the same order.

The theory also looks weird. The authors assume that averaged Q estimation across the ensemble is accurate (the first equation in page 9). If this assumption indeeds holds, we can simply use the average without using any method for reducing the overestimation bias. Further, under this assumption, I believe Theorem 1 is just trivial.

I do appreciate that the authors achieve good performance with simpler approaches than existing work. That being said, it might be possible to present the paper from a "rethinking" perspective.
summary_of_the_review: My major concern is that the empirical results are not informative at all.
review_title: null

Meta-review: This paper introduces a model-free RL algorithm claiming SOTA performance. All but one reviewer agreed on rejection.

#1 The empirical results are based on only 5 seeds (too low) and the plots across 5 domains show no clear evidence of improved performance due to overlapping error bars. The paper's poor empirical practice does not support the main contribution.

#2 The proposed method builds on REDQ, but the authors maintained in the response that their method performed better than REDQ (failing to articulate significant algorithmic novelty) . Even the most positive reviewer (iNq8) did not agree when the authors claimed "our performance improvements are achieved by the innovations we introduced in our algorithm". iNq8 responded "it is unclear whether this performance improvement is really meaningful". The authors never responded to iNq8's followup questions about overlapping error bars and differences in the behaviours produced by the new method.

Points #1 and #2 combine to form the clear conclusion that this work is not ready in its current from for publication.
Decision: Reject