Aggressive Q-Learning with Ensembles: Achieving Both High Sample Efficiency and High Asymptotic Performance
Authors: Yanqiu Wu
Xinyue Chen
Che Wang
Yiming Zhang
Zijian Zhou
Keith W. Ross
Venue: ICLR 2022 Submitted (2022)
Keywords: deep reinforcement learning
off-policy
model-free
sample efficiency
ensembles

Abstract: Recently, Truncated Quantile Critics (TQC), using distributional representation of critics, was shown to provide state-of-the-art asymptotic training performance on all environments from the MuJoCo continuous control benchmark suite. Also recently, Randomized Ensemble Double Q-Learning (REDQ), using a high update-to-data ratio and target randomization, was shown to achieve high sample efficiency that is competitive with state-of-the-art model-based methods. In this paper, we propose a novel model-free algorithm, Aggressive Q-Learning with Ensembles (AQE), which improves the sample-efficiency performance of REDQ and the asymptotic performance of TQC, thereby providing overall state-of-the-art performance during all stages of training. Moreover, AQE is very simple, requiring neither distributional representation of critics nor target randomization. 

Under review as a conference paper at ICLR 2022
AGGRESSIVE Q-LEARNING WITH ENSEMBLES: ACHIEVING BOTH HIGH SAMPLE EFFICIENCY AND HIGH ASYMPTOTIC PERFORMANCE
Anonymous authors
Paper under double-bling blind review
ABSTRACT
Recently, Truncated Quantile Critics (TQC), using distributional representation of critics, was shown to provide state-of-the-ar asymptotic training performance on environments from the MuJoCo continuous control benchmark suite. Also re- cently, Randomized Ensemble Double Q-Learning (REDQ), using high update- to-data ratio and target randomization, was shown to achieve high sample effi- ciency that is competitive with state-of-the-art model-based methods. In this pa- per, we propose a novel model-free algorithm, Aggressive Q-Learning with En- sembles (AQE), which improves the sample-efficiency performance of REDQ and the asymptotic performance of TQC, thereby providing overall state-of-the-ar performance during all stages of training. Moreover, AQE is very simple, re- quiring neither distributional representation of critics nor target randomization.
INTRODUCTION
Off-policy Deep Reinforcement Learning algorithms aim to improve sample efficiency by reusing past experience. A number of off-policy Deep RL algorithms have been proposed for control tasks with continuous state and action spaces, including Deep Deterministic Policy Gradient (DDPG), Twin Delayed DDPG (TD3) and Soft Actor Critic (SAC) (Lillicrap et al.] [016]] Fujimoto et al., 2018 HHaamoj al. 2018a b). TD3 introduced clipped double-Q learning. and was shown to be significantly more sample efficient than popular on-policy methods for a wide range of MuJoCo benchmarks. Soft Actor Critic (SAC) has similar off-policy structures with clipped double-Q learn- ing, but it also employs maximum entropy reinforcement learning. SAC was shown to provide excellent sample efficiency and asymptotic performance in a wide-range of MuJoCo environments, including the high-dimensional Humanoid environment for which both DDPG and TD3 perform poorly.
More recently. Kuznetsov al. (020) proposed Truncated Quantile Critics (TQC), model-free algorithm which includes distributional representations of critics, truncation of critics prediction, and ensembling of multiple erities. Instead of the usual modeling of the Q-function of the expecta- tion of return, TQC approximates the distribution of the return random variable conditioned on the state and action. By dropping several of the top-most "atoms" and varying the number of dropped atoms of the return distribution approximation, TQC can control the over-estimation bias. TQC's asymptotic performance (that is after a long period of training) was shown to be better than that of SAC on the continuous control MuJoCo benchmark suite, including a 25% improvement on the most challenging Humanoid environment. However, TQC is not sample efficient in that it generally requires large number of samples to reach even moderate performance levels.
Chen et al. (02T) proposed Randomized Ensembled Double Q-learnin((REDD), model-free algo- rithm which includes a high Update- Update-To-Data To-aata (UTD) ratio, an ensemble of Q functions, and in-target minimization across a random subset of Q functions from the ensemble. Using a UTD ratio much larger than one, meaning that several gradient steps taken for each environment interaction, im- proves sample efficiency, while the ensemble and in-target minimization allows the algorithm to maintain stable and near-uniform bias under the high UTD ratio. The algorithm was shown to at- tain much better performance than SAC at the early stage of training, and to match or improve the sample-efficiency of the state-of-the-ane model-base algorithms for the MuJoCo benchmarks. How-

---

Under review as a conference paper at ICLR 2022
ever, although REDQ is highly sample efficient for early-stage training, its asymptotic performance is significantly below that of TQC.
Is it possible to design simple, streamlined model-frre algorithm which achieve REDQ's high sample efficiency in early-stage training and also achieve TQC's high asymptotic performance in late state training? In this paper, we achieve this goal with new model-free algorithm, Aggressive Q- Learning with Ensembles (AQE). Like TQC and REDQ, AQE uses an ensemble of Q-functions, and like REDQ it uses UTD ratio > However AQE is very simple, requiring neither distributional representation of critics as in TQC nor target randomization and double-Q learning as in REDQ. AQE controls over estimation bias and the standard deviation of the bias by varying the number of ensemble members N and the number of ensembles K < N that are kept when calculating the targets.
Through carefully designed experiments, we provide a detailed analysis of AQE. We first show that for the MuloCo benchmark, AQE provides state-of-the-ar performance, surpassing the performance of SAC, REDQ, and TQC at all stages of training. We perform ablation study, and show that AQE is robust to choices of hyperparameters: AQE can work well with small ensembles consisting of 10-20 ensemble members, and performance does not vary significantly with small changes in the keep parameter K. We show that that AQE performs better than several variations, including using the median of all ensemble members and removing the most extreme minimum and maximum outlier in the targets. In order to improve computational time, we also consider different multi- head architectures for the ensemble of critics: consistent with the supervised convolutional network literature, we find that a two-head architecture not only reduces computational time but can actually improve performance for some environments. Additionally, we show that AQE continues to out- perform SAC and TQC even when these algorithms are made aggressive with a UTD > 1.
To ensure our comparisons are fair, and ensure our results are reproducible (Henderson et al. 2018 Islam et al., 2017. Duan et al., 20066). we provide open source code" For all algorithmic comparisons, we use the the authors' code.
ADDITIONAL RELATED WORK
Overestimation bias due to in target maximization in Q-learning can significantly slow learning (Thrun & Schwartz] 1993). For tabular Q-learning, Vvan Hasselt (2010) introduced Double Q- Learning, and showed that it removes the overestimation basis and in general leads to an under- estimation bias. |assellt al. (2016) showed that adding Double Q-learning to deep-Q networks can have a similar effect, leading to a major performance boost for the Atari games benchmark. As stated in the Introduction, for continuous-action spaces, TD3 and SAC address the overestimation bias using dlipped-dubbe Q-learming, which brings significant performance improvements (Fuji- moto et al. 2018 Haarnoja al. 20]8a
As mentioned in the Introduction, Kuznetsov et al. 20200) control the over-estimation bias by esti- mating the distribution of the return random variable, and then by dropping several of the top-most "atoms" from the estimated distribution. The distribution estimate is based on methodology de- veloped in Bellemare et al.] (2017); Dabney et al. (2018aji) which employs an asymmetric Huber loss function to minimize the Wasserstein distance between the neural network output distribution and the target distribution. In this paper, in order to counter over-estimation bias, we also drop the top-most estimates, although we do so solely with an ensemble of Q-function mean estimators rather than with an ensemble of the more complex distributional models employed in (Kuznetsov et al. 2020).
is well-known that using ensembles can improve the performance of DRL algorithms (Fauter & Schwenker 2015 Osband et al. 2016 Lee al. [022I) For Q-learning based methods, Anschell et al. (2017) ) use the average of multiple Q estimates to reduce variance. Lan et al.] (2020) intro- duced Maxmin Q-learning which uses the minimum of all the Q-estimates rather than the average. Agarwal et al. (2020) use Random Ensemble Mixture (REM), which employs random convex combination of multiple Q estimates.
2

---

Under review as a conference paper at ICLR 2022
Model-basets methods often attain high sample efficiency by using high UTD ratio. In particular, Model-Based Policy Optimization (MBPO) (Janner et al. 2019) uses a large UTD ratio of 20-40. Compared to Soft-Actor-Criee
(SAC), which is model-free and uses a UTD of 1, MBPO achieves much higher sample efficiency in the OpenAl MuJoCo benchmark (Todorov et al. 2012 Brockman et al., 2016). REDQ (Chen et al.) (02T), model-free algorithm, also successfully y employs a high UTD ratio to achieve high sample efficiency.
3 AI.GORITHM
We propose Aggressive Q-learning with Ensembles (AQE), simple model-free algorithm which provides state-of-the-art performance for the MuJoCo benchmark for both early and late stage of training. The pseudo-code is shown in Algorithm As is the case with most off-policy continuous- control algorithms, AQE has a single actor (policy network) and multiple critics (Q-function net- works), and employs Polyak averaging of the target parameters to enhance stability. Building on this algorithmic base, it also employs an update-to-dan (rather than just two as in TD3 and SAC), and targets that average all the Q-functions excluding the O-functions with the highest N K values. For exploration, it uses entropy maximization in SAC, although it could easily incorporate alternative exploration schemes.
ratio G > 1, an ensemble of N 2 3 Q-functions
The design of AQE borrows components from TQC and REDQ, but also excludes components from those algorithms. Like TQC, AQE does not employ clipped double Q-learning to control over estimation bias, but instead drops the highest Q values when calculating the targets. Like REDQ, AQE employs a UTD > 1 to improve sample efficiency during training. However, unlike TQC, AQE uses UTD 1, and does not employ distributional representations of critics but instead the simpler Q-function expectation estimates. Unlike REDQ which employs two randomly chosen ensemble members when calculating the target, AQE does not use target randomization and uses most of the ensemble members when calculating the target. As discussed in the theory Section [ this provides AQE more flexibility through the choice of the N. The resulting algorithm is not only simple and streamlined, but also provides state-of-the art performance.
AQE has three key hyperparameters, G, N, and K. If we set N K 1 and G 1, AQE is simply the underlying off-polic policy algorithm such as SAC. When N > 2, K = and G = 1, then AQE becomes similar to, but not equivalent to, Maxmin Q-learning (Lan et al. 2020).
3.1 MULTI-HEAD ENSEMBLE ARCUITECTURE
AQE uses an ensemble of Q networks (as does REDQ and TQC). Employing multiple networks, one for each Q-function output, be expensive in terms of computation and memory. In order to reduce the computation and memory requirements, we also consider multi- heati-headd architectures for generating multiple Q-function outputs. Instead of each network providing a single Q-estimate output, we consider N separate Q networks each with heads, providing a total of h N estimates. The h heads from one network share all of the layers except the final fully-connected layer. In practice, we have found h = 2 heads works well for AQE, consistent with work in ensembles of convolutional neural networks for computer vision tasks (Lee et al., 2013). When properly sharing low level weights, multi-headed networks may not only retain the performance of full ensembles, but can sometimes outperform them. We conduct ablation studies on the multi-hean architecture in AQE in Section 4.2
EXPERIMENTS RESULTS
We provide experimental results in this section for AQE, TQC, REDQ and SAC for the five most challenging MuloCo environments, namely, Hopper, Walker2d, HalfCheetah, Ant and Humanoid. To make a fair comparison, the TQC, REDQ and SAC results are reproduced using the authors' open source code, and use the same network sizes and hyperparameters reported in their papers. In particular, TQC employs critic networks with 25 distributional samples for a total of 125 atoms. TQC drops atoms per critic for Hopper, 0 atoms per critic for Half Cheetah, and 2 atoms per critic for Walker, Ant, and Humanoid. For REDQ, we also use the authors' suggested values of N = 10 and M = 2, where M is the number ensemble members used in the target calculation.
3

---

Under review as a conference paper at ICLR 2022
Algorithm Aggressive Q-Learning with Ensembles
1: Initial policy parameters 0, N Q-function parameters 9i, 1,.., N, empty replay buffer D. Set target parameters Purr,i f- 0i for = 1, ,,,,...,.. N.
2: repeat
3: Take one action a ~ (T(- [s:)s Observe reward rt, new state 8(41-
4: Add data to replay buffer: D D u {(st, ae St+1)
5: for G updates do
6: Randomly sample mini- B {(s, s')) from D. for each (s, a, r, 5 E do
7:
8:
Sample ă' (-(sss)). Determine the K indices from i = I....,N N that minimize Quurer, (s', a)).
9:
Compute the Q target y:
y(s, a) =r+
K Σ *EK
Qdwnn (s', log To (ā(('(s))
11:
for i = 1..... N do Update Φ; with gradient descent using
12:
V BB
Σ
QQ. 3(s, a))
(s,,,r,')EB
13:
Update target networks with Ouarg, f Porarg,i + (1 - p) Update policy parameters 0 with gradient ascent using
14:
Qo (8, āo (s)) log (T(āa()))s)
as(s) ~ TO(-16)
N 1=1
#EB
15:
until Convergence
The REDQ paper uses G 20 for the update-to-data ratio, and provides results for up to 300K environment interactions. Using such high value for G computationally infeasible our our experimental setting, since we use 3 million environment interactions for Ant and Humanoid in order to investigate asymptotic performance as well early-stage sample efficiency. In the experiments reported here, we use value of G = for both REDQ and AQE.
For AQE, we use 10 Q-networks each with heads, producing 20 Q-values for each input. The AQE networks are the same size as those in the REDQ paper. AQE keeps 10 out of 20 values for Hopper, all 20 values for half-Cheetah, and 16 out of 20 values for Walker, Ant and Humanoid.
Figure I shows the training curves for AQE, TQC, REDQ, and SAC. For each algorithm, plot the average return of 5 independent trials as the solid curve, and plot the standard deviation across 5 seeds the shaded region. For each environment, we train each algorithm for exactly the same number environment interactions as done in the SAC paper. We use the same evaluation protocol in the TQC paper. Specifically, after every epoch, we run ten test episodes with the current policy, record the undiscounted sum of all the rewards in each episode and take the average of the sums the performance. more detailed discussion on hyperparameters and implementation details is given in the Appendix.
We see from Figure | that AQE is the only algorithm that beats SAC in all five environments dur- ing all stages of training. Moreover, it typically beats SAC by a very wide margin. Table shows that, when averaged across the five environments, AQE achieves SAC asymptotic performance ap- proximately 3x faster than SAC and 2x faster than REDQ and TQC. As seen from Figure [1 and Table in the early stages of training, AQE matches the excellent performance of REDQ all five environments, and both algorithms are much more sample efficient than SAC and TQC. As seen from Figure and Table 3 in late-stage training, AQE always matches or beats all other algorithms, except for Humanoid, where TQC is about 10% better. Table shows that, when averaged across all five environments, AQE's asymptotic performance is 26%, 22%, and 6% higher than SAC, REDQ, and TQC, respectively.
4

---

Under review as a conference paper at ICLR 2022
50003
12000
004000 We 14041
1000
+3300
8000
AOTioar))
2000
puu)
4000
1E00
1000
18:01
I8:00
TDC
SAC
SAC
SAC
0.
0.2
2.8
WnAtEameen intercctions
wtwhrotmmntt Interactione
andirotmmnt intenctione
(a) Hopper-v2
(b) Walker2d-v2
(c) HallCheetah--v2
2000
A20)jumn)
5000
2000
REDO
SAC
Aceloam)
2003
rac
REDG
1.0
20
30
05
25
30
(d) Ant-v2
(c) Humanoid-v2
Figure 1: AQE versus TQC, REDQ and SAC. AQE is the only algorithm that beats SAC in all five environments during all stages of training, and it typically beats SAC by wide margin.
Performance
SAC
TQC
REDQ
AQE
AQE/SAC
AQE/TQC
AQE/REDQ
Hopper at 3000
506K
184K
136K
77K
6.57
2.39
1.77
Walker2d at 4000
631K
371K
501K
277K
2.28
1.34
1.81
HalfCheetah at 10000
763K
737K
552K
304K
2.51
2.42
1.82
Ant 5500
1445K
1759K
1749K
632K
2.29
2.78
2.77
Humanoid at 6000
2469K
1043K
1862K
1345K
1.84
0.78
1.38
Average
3.10
1.94
1.91
Table 1: Sample efficiency comparison of SAC, TQC, REDQ and AQE. The numbers show the
amount of data collected when the specified performance level is reached (roughly corresponding to 90% of SAC's final performance). The last three columns show how many times AQE is more sample efficient than SAC, TQC and REDQ in reaching that performance level.
4.1 FIXED HYPERPARAMETER ACROSS ENVIRONMENTS
Following the TQC paper, in Figure we used different drop atoms for TQC for the different en- vironments. To make the comparison fair, we also used different keep values K for AQE for the different environments. In Figure 2 but now use the same hyperparameter values across environments for TQC (drop two atoms per network) and AQE (K 16). These choices fixed haper-parameters appear to give the best over- all performance for the two algorithms. We report detailed carly-stage and late-stage performance comparisons of all algorithms in Lable/4and Table
we repeat the experiment on the five MuJoCo environments,
We see from the results that with fixed hyperparamters, the conclusions for AQE remain largely unchanged, except for Hopper, where REDQ becomes the strongest algorithm. Table 4 shows that when averaging performance across environments, AQE still matches the high sample efficiency of REDQ during the early stages training. Furthermore, (S that, on average, AQE's asymptotic performance still 16%, 11% % and 9% higher than SAC, REDQ and TQC, respectively.
4.2 ABLATIONS
In this section, ablations to provide further insight into AQE. We focus the Ant environ- ment, and run the experiments up to 1M time steps. (In the Appendix we provide ablations for the other four environments.) As in the REDQ paper, consider not only performance but normalized
5

---

Under review as a conference paper at ICLR 2022
Amount of data
SAC
TQC
REDQ
AQE
AQE/SAC
AQE/TQC
AQE/REDQ
Hopper at 100K
1456
1807
2747
3345
2.30
1.85
1.22
Walker2d 100K
501
1215
1810
2150
4.29
1.77
1.19
HalfCheetah at 100K
3055
4801
6876
6378
2.09
1.33
0.93
Ant at 250K Humanoid at 250K
2107
2344
3279
4153
1.97
1.77
1.27
1094
3038
4535
3973
3.63
1.31
0.88
Average at early stage
2.86
1.61
1.10
Table 2: Etrly-stage performance comparison of SAC, TQC, REDQ and AQE. The numbers show the performance achieved when the specific amount of data is collected. On average, AQE performs 2.9 times better than SAC, 1.6 times better than TQC and 1.1 times better than REDQ.
Amount of data Hopper 1M Walker2d at IM
SAC
TQC
REDQ
AQE
AQE/SAC
AQE/TQC
AQE/REDQ
3282
3612
2954
3541
1.08
0,98
1.20
4134
5532
4637
5517
1.33
1.00
1.19
HalfCheetah at 1M Ant at 3M
10475
10887
11562
13093
1.25
1.20
1.13
5903
6186
5785
7345
1.24
1.19
1.27
Humanoid at 3M
6177
9593
6649
8680
1.41
0.91
1.31
Average at late stage
1.26
1.06
1.22
Table 3: Late-stage performance comparison of SAC, TQC, REDQ and AQE. The numbers show the performance achieved when the specific amount of data collected. The last three columns show the ratio of AQE performance compared to SAC, TQC and REDQ performance. On average, during late-stage training, AQE performs 1.26 times better than SAC, 1.06 times better than TQC and 1. 1.22 times better than REDQ.
6000
14000
12000
2000
12000
2000
3000
2UXU
AUE
4000
TOC
TDC
REDO
REDO
2000
REDO
0.
02
2.6
0.6
(a) Hopper-v2
(b) Walker2d-v2
(c) HalfCheetah-v2
ADE
8009
4000
.000
2000
cnnfformmnt Intoraations
Wi
(d) Ant-v2
(c) Humanoid-v2
Figure 2: Performance for AQE and TQC using same hyper- parameters across the five environments. AQE uses K = 16 and TQC uses atoms = 2 per critic.
bias and standard deviation of normalized bias as defined by the REDQ authors. We first look at how the ensemble size N affects AQE. The first row in F shows AQE with N equal to 2, 5, 10 and 15, with two heads for each Q network, and the percentage of kept Q-values unchanged. As the ensemble size N increases, we generally obtain a more stable average bias, lower std of bias, and stronger performance. When trained with high UTD value. relatively small ensemble size, for example, = 5, can greatly reduce bias accumulation, resulting in much stronger performance. This experimental finding consistent with the results in Theorem in Section 5

---

Under review as a conference paper at ICLR 2022
Amount of data
SAC
TQC
REDQ
AQE
AQE/SAC
AQE/TQC
AQE/REDQ
Hopper at 100K
1456
1719
2747
2294
1.58
1.33
0.84
Walker2d 100K
501
1215
1810
2150
4.29
1.77
1.19
HalfCheetah at 100K
3055
3594
6876
6325
2.10
1.76
0.92
Ant at 250K Humanoid at 250K
2107
2344
3279
4153
1.97
1.77
1.27
1094
3038
4535
3973
3.63
1.31
0.88
Average at early stage
2.71
1.59
1.02
Table 4: staly-stage performance comparison of SAC, TQC, REDQ and AQE when AQE and TQC when using the same hyperparameters across the environments. On average, AQE performs 2.71 times better than SAC, 1.59 times better than TQC and 1.02 times better than REDQ.
Amount of data Hopper IM Walker2d at 1M HalfCheetah at IM
SAC
TQC
REDQ
AQE
AQE/SAC
AQE/TOC
AQE/REDQ
3282
2024
2954
2404
0.73
1.19
0.81
4134
5532
4637
5517
1.33
1.00
1.19
10475
9792
11562
11293
1.08
1.15
0.98
Ant at 3M
5903
6186
5785
7345
1.24
1.19
1.27
Humanoid at 3M
6177
9593
6649
8680
1.41
0.91
1.31
Average late stage
1.16
1.09
Table 5: Late-stage performance comparison of SAC, TQC, REDQ and AQE when AQE and TQC when using the same hyperparameters across the environments. On average, AQE performs 16% better than SAC, 9% better than TQC and 11% times better than REDQ.
The second row in Figure shows the how the keep parameter can affect the algorithm's 's' perfor- mance: under the same high UTD value, as K decreases, the average normalized Q bias goes from over-estimation to under-estimation. Consistent with the theoretical result in Theorem [ by de- creasing we lower the average bias. When K becomes too small, the Q estimate becomes too conservative and starts to have negative bias, which makes learning difficult. We that K 16 has an average bias closest to O and also a consistently small std of bias. These results are similar for the other four environments, as shown in the Appendix.
The third row in Figure 3 shows results for variants of the target computation methods. The Me- dian curve uses the median value of all the estimates in the ensemble compute the Q target. The RemoveMinMax curve drops the minimum and maximum values of all the Q estimates in the ensemble compute the target. We see that these two variants give larger positive bias values.
We also considered different combinations of ensemble size N and the number of multi-heads h while keeping the total number of Q-function estimates N h fixed. We performed these experi- ments for all five environments, and the results are shown in Figure of the Appendix. In terms of performance, we found the two best combinations to be N 20, T and N 10, h 2, with the former being about 50% slower than latter in terms of computation time. In the Appendix we also consider endowing REDQ with the same multi-head ensemble architecture as AQE and find that it does not improve REDQ substantially.
In the Appendix, we also consider including a UTD ratio of = 5 in both SAC and TQC, and compare these aggressive versions with AQE, also with G 5. Although SAC becomes more sample efficient with G = 5, AQE continues to outperform both algorithms except for Humanoid, where once again TQC performs somewhat better than AQE for the final stage of training.
5 THEORETICAL RESULTS
In this section, characterize how changing the size of the ensemble N and the keep parameter K affects the estimation bias term in the AQE algorithm. We will restrict our analysis to the tabular version of AQE (see Appendix EE. Our analysis will follow similar lines of reasoning as Lan et al. (2020) and Chen et al. (202T) which extends upon the theoretical framework introduced in Thrun & Schwwrtz (193)..
7

---

Under review as a conference paper at ICLR 2022
Relo
000000 0400
5000
2000
05
1830
0.0
0.5
1.0
2.2
022
144
0:0
Wnstraemennt intercction
#nA.rcnmmmnt Internacione
WCTA3000mtrE intractioons
(a) Ensemble size: Perf
(b) Ensemble size: Bias
(c) Ensemble size:Std
7000
1000
010000 No: M031
5000
4020
Ar
2000
M
A
2000
1000
00
0.2
0.5
1.
02
02
0.4
0.6
(d) Keep value: Perf
(e) Keep value: Bias
(f) Keep value: Std
0000
wwwgg les 1000
5000
2.5
4050
8.0
1006
mnvt intarecionn
166
unvifemmmnt Inferacions
envionmmnn intrectiono
1nc.
(g)
Variations: Perf
(h) Variations: Bias
Variations: Std
Figure 3: AQE ablation results for Ant. The top row shows the effect of the ensemble size N. The second row shows the effect of keep number parameter K. The third row compares AQE to some variants.
For each E A. let EK,N (s, a) be the ensemble members in {1,... "i....,N, N) with the K lowest values of Q' (8, a). = 1,..., N. In the tabular case, the target for the Q networks take the form:
Σ 0'(6t,a))
+ 7 max
jEEx,N(s',o")
Define the post-update estimation bias as
ZK.N :
max
1
Σ
2(65.0)) - -
K
1
Σ
('6''
) max (5's
=7 max
K
(E
Under this definition, if E[ > there is tendency for the positive bias to accumulate during updates. Similarly, if E[Zx,N] O, then the expected post-update estimation bias is negative and there tendency for the negative bias to accumulate during updates. Ideally, would like E[Zk,N] R2 0
then the expected post-update estimation bias is positive and
Also let
(s, Q" (8, + +e (s.
(3)
8

---

Under review as a conference paper at ICLR 2022
where e (s, a) is independent and identically distributed enror term across all j's and all a's for each fixed s. We further assume that Eie' (s, a)] = Note that with this assumption
N
E
2 ==1
('((',
- (s)
N
that is the pre-update estimation bias is zero. The following theorem shows how the expected esti- mation bias changes with N and K:
Theorem 1. The following results hold for E|ZK,N];
1. E[ZN,N] 2 0 for all 21
2. 1[ - LE K for all K N.
3. [[[Z,N++] & E[Z
4. Suppose that eaa S c. for some > 0 for all 8, and j. Then there exists an N sufficiently large and K <, N such that Z <
Proof Sketch. Part 1 result of Jensen's Inequality, and Parts 2 and 3 can be shown by analyzing how the average of the K smallest ensembles changes when adding extra ensemble model. Given the first three parts, we only need to show that E[ < 0 to show that there exists K for sufficiently large N where the expected bias is negative. See supplementary material for full proof.
Theorem shows that we can control the expected post-update bias E[Z,,v] through adjusting K and N. More concretely, we can bring the bias term from above zero (i.e. over estimation) to under zero (i.e. under estimation) by decreasing K and/or increasing N.
We note also that similar to Chen et al. (002T. we make very few assumptions on the error term 6, This in contrary to Thrun & Schwartz (199)) and Lan et al. (2020), both of whom assume that the error term is uniformly distributed.
In REDQ, a random subset of ensemble models of size M chosen and for any fixed M, the bias does not depend on the number of ensemble models N (Chen et al.) 2021). We can thus see from Figure 7 in Appendix, with M fixed, increasing the size of ensemble with the multi-head architecture does not necessarily help the training of REDQ. Unlike REDQ, AQE can control the bias term through both the number of ensemble models used in the average calculation K and the total number of ensembles N, allowing for more flexibility. One other drawback for REDQ is that ignores the estimates all other ensemble estimates except for the minimal one in the randomly chosen set, which diminishes the power of the multiple ensemble sets. In contrast, AQE utilizes makes use of most of the ensemble models when calculating the target.
6 CONCLUSION
Perhaps the most important take away from this study is that simple model-free algorithm can do surprisingly well, providing state-of-art performance all stages of training. There no need for model. distributional representation of the return, or in-target randomization to achieve high sample efficiency and asymptotic performance.
The experimental results in this paper show that the AQE algorithm provides state-of-the-art perfor- mance not only when hyper- parametrsss are customized individual environments (as done in the TQC paper) but also when the parammtrrsettrss are held fixed across environments. The ablation study shows that AQE is robust to small changes in the hyper-parameters. The theoretical results complement the experimental results, showing that the estimation bias can be controlled by either varying the ensemble size N or the keep parameter K.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference Machine Learning. 2020.
Oron Anschel, Nir Baram, and Nahum Shimkin. Averaged-DQN: Variance reduction and stabiliza- tion for deep reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning, pp. 176-185, 2017.
Mare G. Bellemare, Will Dabney, and Rémi Munos. A distributional perspective reinforcement learning. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Con- ference Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp- 449-458. PMLR, 06-11 Aug 2017.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym, 2016.
Xinyue Chen, Che Wang, Zijian Zhou, and Keith W. Ross. Randomized ensembled double q- learning: Learning fast without a model. In International Conference on Learning Representa- fions, 2021.
Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80, pp. 100--111,, 2018a.
Will Dabney, Mark Rowland, Marc G. Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In Sheila A. Mcllraith and Kilian Q. Weinberger (eds.), Pro- ceedings of the Thirty-Second AAAI Conference Artificial Intelligence, 2018b.
Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep rein- forcement learning for continuous control. In Proceedings of The 33rd International Conference on Machine Learning, pp. 1329-1338, 2016.
Stefan Faußer and Friedhelm Schwenker. Neural network ensembles in reinforcement learning. Neural Process. Lett., pp. 55-69, 2015.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Proceedings of the 35th International Conference on Machine Learning, pp. 1587-1596, 2018
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference on Machine Learning, Pp. 1861-1870, 2018a.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Schoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algo- rithms and applications. Technical report, 2018b.
Hado van Hasselt Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, pp- 2194-2100.. AAAI Press, 2016.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In AAAI, 2018.
Riashat Islam, Peter Henderson, Maziar Gomrokchi, and Doina Precup. Reproducibility of bench- marked deep reinforcement learning tasks for continuous control. CoRR, 2017.
Michael Janner. Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model- based policy optimization. In Proceedings of the 33rd International Conference on Neural Infor- mation Processing Systems, 2019.
Arsenii Kuznetsov, Pavel Shvechikov, Alexander Grishin, and Dmitry Vetrov. Controlling overesti- mation bias with truncated mixture of continuous distributional quantile critics. In Proceedings of the 37th International Conference Machine Learning, pp. 5556-5566, 2020.
10

---

Under review as a conference paper at ICLR 2022
Qingfeng Lan, Yangchen Pan, Alona Fyshe, and Martha White. Maxmin q-leaming: Controlling the estimation bias of q-learning. In 8th International Conference on Learning Representations, 2020.
Kimin Lee, Michael Laskin, Aravind Srinivas, and Pieter Abbeel. SUNRISE: simple unified framework for ensemble learning in deep reinforcement learning. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, volume 139, pp. 6131-6141, 2021.
Stefan Lee, Senthil Purushwalkam, Michael Cogswell, David J. Crandall, and Dhruv Batra. Why M heads are better than one: Training diverse ensemble of deep networks. CoRR, 2015.
Timothy Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In 4th International Conference Learning Representations, 2016.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. In Advances in Neural Information Processing Systems, volume 29, 2016.
Sebastian Thrun and Anton Schwartz. Issues in using function approximation for reinforcement learning. In Proceedings of the Fourth Connectionist Models Summer School, pp. 255-263. Hills- dale, NJ, 1993.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In 2012 IEEF/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033, 2012.
Hado van Hasselt, Double q-learning. Advances in Neural Information Processing Systems, pp. 2613-2621, 2010.

---

Under review as a conference paper at ICLR 2022
A HYPERPARAMETERS AND IMPLEMENTATION DETAILS
Table 6 gives a list of hyperparameters used in the experiments. Most of AQE's hyperparameters made the same in the REDQ paper ensure fairness and consistency in comparisons, except that AQE has 2-head critic networks. As compared with AQE and REDQ, TQC uses a larger critic network with 3 layers of 512 units per layer. table 7 the number of Q values we keep in the ensemble to calculate the target in AQE.
we report the dropped atoms for TQC and
Hyperparameters
AQE
SAC
REDQ TQC
optimizer
Adam
learning rate
1-10-
discount(7)
0.99
target smoothing coefficient(p)
0.005
replay buffer size
1 106
number of critics N
10
2
10
5
number hidden layers in critic networks
2
3
size of hidden layers in critic networks
256
256
256
512
number of heads in critic networks h
25
number of hidden layers in policy network size hidden layers in policy network mini-batch size nonlinearity
2
256
256
ReLU
UTD ratio G
5
5
1
Table 6: Hyperparameter values.
Environment
Dropped atoms per critic
Kept Q values out of N values
Hopper
5
10
HalfCheetah
0
20
Walker
2
16
Ant
2
16
Humanoid
2
16
Table 7: Environment- hyper-pane apper-parameters for TQC and AQE.
12

---

Under review as a conference paper at ICLR 2022
B ADDITIONAL RESULTS FOR AQE, SAC-5 AND TQC-5
Figure 4 presents the performance of AQE, SAC-5 and TQC-5 for all the environments. SAC-5 and TQC-5 uses UTD ratio G for SAC and TQC, respectively. We can see that AQE continues to outperform both algorithms except for Humanoid, where TQC performs somewhat better than AQE in the final stage training. SAC becomes more sample efficient with G 5; however, AQE still beats SAC-5 by a large margin.
C ADDITIONAL RESULTS FOR PARAMETER K
Due to lack of space, Figure 3 only compares different AQE keep numbers K for Ant. Figure shows the performance, average estimation bias and standard deviation for all five environments. Consistent with the theoretical result in Theorem [ by decreasing K, we lower the average bias.
13

---

Under review as a conference paper at ICLR 2022
4020
2.0
A0E
20
AGE RAG
avengh nomra nomralzed 235
1.5
S00 nomanaooo tls
TOGS
2000
1006
AGE
SAC-5
00
1O0-0
00
0.2
24
0.5
10
-0.5
000
22
244
0.5
0.5
1.
02
0.4 envrrnment
0.5
0.3
interations
(a) Performance, Hopper
(b) Average normalized bias
(c) Std of normalized bias
AGE
SAC--5
200000 500m 2005 Bar
5000
T
3000
1000
0.5
0.5
0.4
wwuitmmment
ansiroement
(d) Performance, Walker
(e) Average normalized bias
(I) Std of normalized bias
AGE
AD" SAC-5
awergg sorna booc to
2.5
Sid somatadd Diee
SAC-
unonge Vol vt..0
100-
1026
18000
1.5
1.0
1.0
2000
0.0
ansfronmment Internatione
160
andirommmett Internatione
1nc
envimmmtnnt insecatiinn
160
(g) Performance, HalfCheetah
(h) Average normalized bias
(i) Std of normalized bias
1000
AGI
awagge nomn lond bot
SAG-3
SAL-2
8000
Rocs
800 womataaattttooooo
6000
1.5
20
3.0
0.5
2.5
00
wwuttommen interattions
anw.oommmnt Internatione
www.ecmmen Internaiions
(j) Performance, Ant
(k) Average normalized bias
(I) Std of normalized bias
20
SAC-6
AGE
anwagg rormateed boe
ADE
BAC-6
ADE
MAT1 300 18031
TOX-5
TOC--
0f nomaable
4000
2000
2.0
00
06
1.0
1.5
3.0
-03
00
0.5
1.0
20
2.5
60
3.5
1.0
20
2.5
(m) Performance, Humanoid
(n) Average normalized bias
(0) Std of normalized bias
Figure 4: Performance, average and std of normalized Q bias for AQE, SAC-5 and TQC-5. All of the algorithms in this experiment use UTD=5,
14

---

Under review as a conference paper at ICLR 2022
1.0
anwngg lust N0020
2ww-3g normatze C2
0.0
1.6
2.0
89
0.8.
Inferecions
Interactions
(a) Performance, Hopper
(b) Average normalized bias
(c) Std of normalized bias
meegge nomral 205
K-tu
4800
80 coma
3000
1806
envirommnn intrrctions
166
enwiromment Infereatione
100
envrrnnent mnrraciion
(d) Performance, Walker
(e) Average normalized bias
(I) Std of normalized bias
14300
10:20
awergg norms lond to
4.5
8--15
18080
Bid namatatddd
0.8
8000
0.5
0000
02
2.2
0.4
1.5
-1.0
60
22
-0
3.0
0A amvimmmmeet Irrecactions
aradronmment Internationns
andhronmment Internationee.
(g) Performance, HalfCheetah
(h) Average normalized
(i) Std of normalized bias
7000
1000
5000
average somm tomm
2000
4300
Si romatger
A
M
Am
2000
1000
0.0
0.2
5A
0.5
0.5
1.0
02
8A
C.5
1.0
0.2
0.4
3.0
0.5
(i) Performance, Ant
(k) Average normalized bias
(I) Std of normalized bias
1000
avergge nomm bond Mone
0.8
sringe bok wi.1
0.6
1.5
K--10
6200
8000
1006
522
0.4
00
wwwimmmmentt intenctoon
enstroreeent Intervation
arsdenmment Intennctions
(m) Performance, Humanoid
(n) Average normalized bias
(o) Std of normalized bias
Figure 5: Performance, average and std normalized Q bias for AQE with different values K.
15

---

Under review as a conference paper at ICLR 2022
D ADDITIONAL RESULTS FOR MULTI-HEAD ARCHITECTURE
Due to lack of space, Figure 3 only compares the different size of the ensemble N and the number of heads h for Ant. Figure shows the results for all five environments. We can see that the combination of N 10, h 2 and N = 20, h = 1 have comparable performance. However, N 10 and h 2 is faster in terms of computation time.
0000
12000
2000
19300
4000
0000
2000
NP2 Head-10
N-2 Hasd-10
g00
M-2 Feed-10 bed, Feeded
1800
Nec, Heedee
Ned. Headed
Ni10, Heudr12
NiN Hoas222
2000
N-10, Heedr2
10 Hasset
todn, Heares
0.
0.5
10
02
00 anwiotmmnnt Instcactions
0. andiotmmmnt Intenctions
24
enstrommen interactionss
(a) Hopper-v2
(b) Walker2d-v2
(c) HalfCheetah-v2
2003
6000
5000
5000
5900
N: 2_Heed 10
Head*4
NAO Heedee10
3009
N-12, llassk
Ne11, Hewrt
INTO Hood''
2009
2000
Necn, lendln
1000
1000
0.
02
0.4
38
10
00
3.2
0,4
0.8
100
(d) Ant-v2
(c) Humanoid-v2
Figure 6: Performance for AQE with different combinations of number of Q networks and number of heads.
Will the performance of REDQ match that of AQE if we also provide REDQ multi-head architec- ture? Figure 7 examines the performance of REDQ when it is endowed with the same multi-heal architecture as AQE. We see that the performance of REDQ does not substantially improve.
4000
6001
12300
12300
.000
8809
1806
REDO, rullhwad
REDO
2000
1000
REDO Bedd REDO
2000
4020
REDO
REDO rellheadd
0
ACE
AGE
AGE
0.0
0.2
BA
0.5
10
60
02
64
06
35
1/
00
02
24
26 Interactions
05
interatiions
Interaations
(a) Hopper-v2
(b) Walker2d-v2
(c) HalfCheetah-v2
REDO ru.iiwood
8000
REDO
AGE
2000
4000
4000
2009
2000
environmmnt interactins
10t
enviromment inaractions
(d) Ant-v2
(e) Humanoid-v2
Figure 7: Performance of REDQ with N=10 and heads 2 as compared with REDQ and AQE.
16

---

Under review as a conference paper at ICLR 2022
E TABULAR AQE WITH N ENSEMBLE MEMBERS AND d DROPS
Algorithm Tabular AQE
1: Initialize: Q' (2(8 a) for all E S, E A.j = I,....N.. ...
2: repeat
3: For some state E S, choose E A based on (Q' ك)} observe r, 8.
4: For each a A, let ( ,'', a') values of ('(ss I....,
be the ensemble members in (I,..., N) with the K lowest
5: Get target
=
max E'E. K
Σ
Q's''a))
jEEk,N(s'a))
6:
for j= N do
7:
Update each Q" (s
Q +al-))yyyylllio -la--(((a)
8: 8 +-+
9: until end
PROOFS
We first present the following lemma:
Lemma F.1 (Chen et al., (02I). Let X2....... be an infinite sequence of i.idd random vari-
ables with edf F(x) and let r = inf x F(x) O. Also let YN min{X1, X2...., XN). Then Y1, Yz,.. converges to almost surely.
Proof. See Appendix A.2 of Chen et al. 2021
Theorem 1. The following results hold for E[Z,,N]:
1. E[ 2 for all 1.
2. [[[Zx11.]] < [K for all K N.
3. E[Z,,N+1]] S E[ZK,N].
4. Suppose that ei S c for some e > for all 8, a and j. Then there exists an N sufficiently large and K < such that Z < O.
Proof.
1. By definition,
N Σ
Z
max
Q² Q'(s'
max ("
N
j=1
Q' (s', ,a)) )1 max Q (5',
1 N
(4)
2 max
Σ
-max
N
5=1
max ("(s'sa)) max Q=( .aa] =0 a'
2. Let
QK.N(8,0)= = K Σ (s))))).
(5)
JEEK.N
17

---

Under review as a conference paper at ICLR 2022
Since for any state 8, max, QK+I.N(S,a) max, QK.((s,a).
1[ZK+INN]
max QK+ max (axxQ"s',a')
17E máx QK,N(s max Q (5('sa"))
(6)
K|Z
3. Comparing K[Z and K|ZK,N+1] is equivalent to comparing (x.N(s,a) and (s, Since (s, a) is i.i.d., by extension Q' a) is also i..d. for j = 1,2, .... Suppose (s,, a) is drawn from some probability distribution F, then given (s, Q a) can be calculated by generating an additional ('(s, a) from The new sample Q" (s(s a) affects the calculation of ( a) under the following two cases:
If as) > ma (', a), then the lowest K values remain unchanged hence QK,N(s,a) = Q
Else if (s(ssa) a) < maxjeEx, Q' (s, a), then maxjeEx, (' a) would be re- moved from and ('(s, would be added to the set of lowest K values, therefore QK.N-1(S.a) S (s.
Combining the two cases QK S QK,((s, therefore E|Zk,N+1| & E[Z
4. Since E[ZN 2 0, E[Z < E[Z and E[ZK,N+1| || < E[Zx,x]- It is suffice to
show that E Z N
< 0 for some N. The rest of the proof largely follows Theorem of
Chen et al. 2021).
(= Fa(x) > 0} where Fa(x) is the edf of (' a). = 1,2. By Lemma 1, as, minisigN (2(s, a) converges almost surely to for each Since the action space is finite. it then follows that max, Q1,v a) converges almost surely to to r = max. Ta- Due to our assumption that (s, a) S and that (= a) is finite, it then follows that max, Q1.v (s, a) is also bounded above. By Part of the theorem, Q1,v a) is monotonoically decreasing w.r.t N. and since max, Q1, (s, a) is also bounded above and converges almost surely to 7, we have
ISJEN [[ = [[[max min Q' (s(s a)] - max Q"
=7 mmx Y max Q (x(ssa) a)) N=200 max T. max Q"x <
where the last equality follows from the assumption that the error e (s. a) is non-trivial, and hence To < max, (T a) for all a. Therefore for a sufficiently large N, there exists 1 < K < N such that Eg,N
18


Reviews:
[{"review_id": "vKcPDbDXkyu", "reviewer": "Reviewer_q4e8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Strengths]:\n1. AQE is simple and effective. Implementation requires little change to REDQ.\n2. Empirical studies show that AQE achieves a much higher average return compared to SAC, TQC and REDQ in most environments.\n\n[Weaknesses]:\n1.\tOne of the major concerns is the similarity of this paper to a prior work REDQ. The similarity is of various aspects, including the algorithmic tables, sentences, results, analyses and even the format! On top of that, the main idea is actually very much similar, and as far as I can tell the only interesting difference seems to be the replacement of Q value from the min value of a randomly sampled set in REDQ to the average of the min K values in AQE. The authors also don’t give a convincing explanation why this replacement can improve the performance, either theoretically or intuitively.\n2.\tThe most similar approach REDQ which the authors cited multiple times is not tested in Figure 4 of Appendix B. Compared with the baselines in that figure (SAC-5, TQC-5), REDP sounds much better in terms of reducing the Q estimation error. I am quite curious how REDQ on the bias.\n3.\tIt’s kind of strange that a model with low bias and std provides poor performance sometimes. For example, in figure 3, the orange line (K=10) has lower bias and std than the red line (K=16) but provides worse performance. The same phenomenon appears in Figure 5 (g) (between the blue line and red line) and Figure 5 (j) (orange line and red line), etc. The authors don’t provide an explanation.\n4.\tHave the authors tried to adjust K dynamically, such as using a small K in the early stage to avoid overestimating and use large K in the late stage to reduce the estimation variance?\n\n[typos]\nNot sure whether or not the authors intended to say “AQE improves the sample-efficiency performance of TQC and the asymptotic performance of REDP” in the abstract? Because TQC provides SOTA asymptotic performance while REDP achieves high sample efficiency...\n", "summary_of_the_paper": "This paper proposes Aggressive Q-Learning with Ensembles (AQE), which improves the sample-efficiency performance of TQC and the asymptotic performance of REDQ, thereby providing overall state-of-the-art performance during all stages of training. Empirical studies verify the effectiveness of AQE.", "summary_of_the_review": "Overall I think the contribution of this paper is not significant to the high standard of ICLR. I am leaning to the rejection.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "oaWnuGRtPxe", "reviewer": "Reviewer_DEA7", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": " **Pros**\n\n* The proposed AQE algorithm is simple and seems to be effective based on the presented empirical results.\n* In general the paper is well written. The proposed algorithm and the empirical results are presented in a clear way.\n* Empirical analysis on five MuJoCo environments shows that AQE is competitive to TQC and improves the sample efficiency performance of REDQ. \n* An extensive ablation study is provided, shedding light on the impact of the three free parameters ($K, N, G$) in the performance of AQE. \n\n**Cons and Comments**\n\n* The novelty of this work is a little bit incremental as it can be considered as a simple modification of the Randomised Ensembled Double Q-Learning (REDQ) algorithm.\n* The connection between AQE and REDQ is not clear for someone that is not familiar with the REDQ algorithm. Actually, the reader should check the paper of REDQ first to understand its connection to the AQE. \n* The number of K (first row) and N (second row) should be also presented at Figure 3 that illustrates the results of the ablation study. In general, figures/legend should contain all the necessary information.\n* What is the performance of AQE in the extreme case where $G=1$? According to the paper $G$ should  be greater than $1$ ($G>1$), but it would be really interesting to check the impact of updating the model multiple times on the performance of AQE. \n* What is the impact of the hyper-parameter $\\rho$ on the performance of AQE?\n* How will the replacement of SAC with another off-policy algorithm affect the performance of AQE? \n\n*Minor Typos*\n\n* p. 4: infeasible in our our ", "summary_of_the_paper": "**Summary**\n\nThis work introduces Aggressive Q-learning with Ensembles (AQE), a simple model-free reinforcement learning algorithm. AQE can be considered as a modified version of the Randomised Ensembled Double Q-Learning (REDQ) algorithm. Specifically, AQE uses a number ($1 \\leq K \\leq N$) of the ensemble members ($N$) with the lowest Q values for calculating the target, instead of sampling a number ($1 \\leq M \\leq N$) of ensemble members and keeping the one with the lowest Q values as in the case of REDQ. Similarly to REDQ, AQE also employs a high update-to-data ratio ($G > 1$) to improve the sample efficiency. Empirical analyses have been conducted on five MuJoCo environments showing that the performance AQE is competitive to that of Truncated Quantile Critics (TQC) and improves the sample efficiency performance of REDQ. ", "summary_of_the_review": "In general, the proposed model-free reinforcement algorithm, called AQE, is simple and its performance is competitive with that of other baselines on five MuJoCo environments. An ablation study is also provided at the same time. My main concern as regards this work is its novelty. As aforementioned, AQE constitutes a slight variant of the already existing REDQ algorithm.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "hOis2eW8ZK0", "reviewer": "Reviewer_fDzQ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "My major concern is that the empirical results are not informative at all.\n\nAs shown by Figure 5 of Henderson et al. (2018), 5 random seeds are clearly not enough for empirical comparison. Consequently, I cannot draw any conclusion from the figures or convince myself of any conclusion the authors draw from the figures. The tables are even worse since they do not contain any confidence estimation. I am disappointed in that the authors indeed cited Henderson et al. (2018) regarding the concern of reproducibility of RL research but still consider only 5 random seeds.\n\nMoreover, the paper considers only 5 domains. The contribution of this work purely empirical so I would expect the authors to evaluate the algorithms in much more domains, e.g., in DeepMind control suite. I cannot convince myself of any conclusion with only 5 domains, especially when the performance of the proposed algorithms and the baselines are of the same order.\n\nThe theory also looks weird. The authors assume that averaged Q estimation across the ensemble is accurate (the first equation in page 9). If this assumption indeeds holds, we can simply use the average without using any method for reducing the overestimation bias. Further, under this assumption, I believe Theorem 1 is just trivial.\n\nI do appreciate that the authors achieve good performance with simpler approaches than existing work. That being said, it might be possible to present the paper from a \"rethinking\" perspective.", "summary_of_the_paper": "The paper proposes a new method for mitigating the overestimation bias in Q-learning-like algorithms. An ensemble of critics of size N is trained while only the small K individuals are used for computing the bootstrapping targets. Empirical results are provided to support the claim.", "summary_of_the_review": "My major concern is that the empirical results are not informative at all.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "aQXiHheTdo7", "reviewer": "Reviewer_iNq8", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n+ source code to replicate the proposed method is provided. \n+ the proposed method is simple. \n+ the performance (sample efficiency) of the proposed method is good. \n\nWeaknesses: \n\n- Incremental improvement from the REDQ paper: \n\n1. AQE algorithm (Algorithm 1) is almost the same as REDQ algorithm. \nThe main difference between AQE and REDQ is that AQE uses (1) the average of their outputs (i.e., model averaging) to target and (2) multi-head Q functions. \nThe idea of using multi-head functions instead of ensembles is not brand new [1]. \nAnd the idea of using averages Q-estimates as targets is not new [2]. \nIMO, the authors' primal contribution is to show that incorporating these into REDQ improves performance. \nHowever, it is not clear whether the performance improvement shown in the experimental results (e.g., Figure 1) is very meaningful. \nFor example, is it difficult to achieve the AQE performance obtained in the experiment simply by tuning the REDQ's hyper parameters, policy and Q functions architecture?\n\n2. the theoretical analysis (the flow of the proof of Theorem 1) is almost identical to the one done in the REDQ paper.\n\n[1] Stefan Lee, Senthil Purushwalkam, Michael Cogswell, David J. Crandall, and Dhruv Batra. Why M\nheads are better than one: Training a diverse ensemble of deep networks. CoRR, 2015.  \n[2] Oron Anschel, Nir Baram, and Nahum Shimkin. Averaged-DQN: Variance reduction and stabilization\nfor deep reinforcement learning. In Proceedings of the 34th International Conference on\nMachine Learning, pp. 176–185, 2017.\n\n- Theorem 1 and its proof are not correct:  \nAt the beginning of pp. 18, the authors states ''Since for any state $s$,  $\\max_a \\bar{Q} _ {K+1, N}(s, a) \\geq \\max_a \\bar{Q} _ {K, N} (s, a)$ .''  \nHowever, since $ \\bar{Q} _ {K, N}(s, a) $ is defined as $\\bar{Q} _ {K, N}(s, a) = \\frac{1}{K} \\sum _ {j E_ {K,N} } Q^j(s, a)$, the above is not an inequality but an equality: $\\max_a \\bar{Q} _ {K+1, N}(s, a) = \\max_a \\bar{Q} _ {K, N}(s, a)$  \nThis means that changing the value of $K$ does not theoretically change the expected value of the estimation bias $  \\mathbb{E}[Z _ {k, N}] $.  \nThe remaining part of the proof also contains flaws.  \nFor example,  $\\bar{Q} _ {1, N} (s, a) = \\min _ {1 \\leq j \\leq N} Q^j (s, a)$ is assumed in the proof of the 4th property of Theorem 1, but it is not consistent with Eq.5 and Algorithm 2.  \n\nMinor comments:  \nAlgorithm 1:  Adding a more detailed explanation of how the multiple (h) estimates of each Q function are aggregated into a single scalar value in lines 10 and 14 would make the paper easier to understand.  \n\n\\>\\> algorithmic base, it also employs an update-to-data ratio  \n\\>\\> The REDQ paper uses G = 20 for the update-to-data ratio  \nupdate-to-data -> UTD  \n\n\\>\\> Hado van Hasselt. Double q-learning.  \nDouble q-learning -> Double Q-learning  \n\n\\>\\> In International Conference on Machine Learning, 2020.  \n\\>\\> In International Conference on Learning Representations,  2021.  \nIn Proceedings of ...  ", "summary_of_the_paper": "In this paper, the authors show that the asymptotic performance of  REDQ is improved by replacing the minimum of Q network outputs with the average of the multi-head Q network outputs.", "summary_of_the_review": "I recommend rejecting the paper.\nA method that performs better than REDQ is beneficial from an engineering perspective. \nHowever, as noted above, the innovation made in the paper is incremental from what made in the REDQ paper. \nAlso, the paper contains flaws in the theoretical analysis section that should be substantially revised. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Ecrx5N_3-Zc", "reviewer": "Reviewer_H9au", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nQuestions:\nAll Line9: how is the K determined? Do you mean take K minimal Q values from the complete set of N such Q values? Why do you take K minimal Q values for computing the target? Usually the Q-learning style algorithms takes the maximum. I found this is a mysterious part of the algorithm. \n\nFollow up the previous question, have you studied taking average from all N Q targets? The maximum of the N or K Q values, and what have you found? In the ensemble papers, people usually takes the maximum of Q values from ensembles, e.g., see eq 8 of \n\nhttps://arxiv.org/abs/1811.02696\n\nIt is also interesting to see that this paper together with TQC and REDQ all uses critic ensembles. However, the above paper (ACE) uses ensembles from actors. This is an interesting alternative. It is interesting to discuss the connections and strength of both approaches. \n\n\nThe computation of y in line 10 of Alg. 1 uses log of the policy, this wasn’t explained. Is it a regularization effect? Why is it important for your algorithm? Have you studied this effect? (Ablation studies?)\n\nInterestingly, in the actor update, the whole ensemble is used to average for a Q value estimate. Could you explain the choice of the complete instead of a subset of ensemble here?\n\nThe results were run for 1 million frames: this may not be able to support asymptotic behaviour. In particular, in (b)Walker2d, clearly the TQC may possibly be further faster than AQE. Could you provide more frames than shown?\n\nIn the theory part, you said we would like E(Z_KN) \\appr 0. What does the approximation mean here? Do you want to minimize it? Is it positive or negative? Your theorem 1 shows this is positive, then how bigger is it than zero? Your result 2 in Th1. shows the smaller K is preferred? Is this the message? While result 3 shows when K is fixed, N should be as big as possible? How to explain this intuitively?\n\n\n\nMinor:\nThe algorithm name from Q-learning is a little unconventional. There is a policy network, which is usually not called a Q-learning style algorithm. \n\nSec 3:\nThe averaging over the ensembles for Q estimates is not Polyak averaging. Polyak averaging refers to temporal averaging over the past. Here your average is nothing about time. \n\n", "summary_of_the_paper": "629\nThis paper studies data efficiency and asymptotic convergence rate of deep reinforcement learning and proposed a new algorithm called AQE and claims the algorithm are both fast in both senses. The paper is mostly empirical with results on Mujoco, and compared with TQC and SAC and showed competitive performance. There is also a theorem about the update estimate bias, which tells the orderings of the bias for when K or N is increasing. \n\nI've read the authors' feedback and other reviews. I think the paper goes hard as the asymptotic rate in experiments is not supported well. The theorem is not helping much for the two claims. There are also other problems (presentation, algorithmic discussions and comparisons) that can be much improved. Keep trying!", "summary_of_the_review": "It appears an interesting paper, but the algorithm wasn't explained with much detail which especially didn't address critical components. \n\nThe results show sample efficiency. The asymptotic rate is not very much supported. \n\nThe theory's message isn't very clear. \n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a model-free RL algorithm claiming SOTA performance. All but one reviewer agreed on rejection.

#1 The empirical results are based on only 5 seeds (too low) and the plots across 5 domains show no clear evidence of improved performance due to overlapping error bars. The paper's poor empirical practice does not support the main contribution.

#2 The proposed method builds on REDQ, but the authors maintained in the response that their method performed better than REDQ (failing to articulate significant algorithmic novelty) . Even the most positive reviewer (iNq8) did not agree when the authors claimed "our performance improvements are achieved by the innovations we introduced in our algorithm". iNq8 responded "it is unclear whether this performance improvement is really meaningful". The authors never responded to iNq8's followup questions about overlapping error bars and differences in the behaviours produced by the new method.

Points #1 and #2 combine to form the clear conclusion that this work is not ready in its current from for publication.
Decision: Reject