Dropout Q-Functions for Doubly Efficient Reinforcement Learning
Authors: Takuya Hiraoka
Takahisa Imagawa
Taisei Hashimoto
Takashi Onishi
Yoshimasa Tsuruoka
Venue: ICLR 2022 Poster (2022)
Keywords: Reinforcement learning

Abstract: Randomized ensembled double Q-learning (REDQ) (Chen et al., 2021b) has recently achieved state-of-the-art sample efficiency on continuous-action reinforcement learning benchmarks. This superior sample efficiency is made possible by using a large Q-function ensemble. However, REDQ is much less computationally efficient than non-ensemble counterparts such as Soft Actor-Critic (SAC) (Haarnoja et al., 2018a). To make REDQ more computationally efficient, we propose a method of improving computational efficiency called DroQ, which is a variant of REDQ that uses a small ensemble of dropout Q-functions. Our dropout Q-functions are simple Q-functions equipped with dropout connection and layer normalization. Despite its simplicity of implementation, our experimental results indicate that DroQ is doubly (sample and computationally) efficient. It achieved comparable sample efficiency with REDQ, much better computational efficiency than REDQ, and comparable computational efficiency with that of SAC.

Published as a conference paper at ICLR 2022
DROPOUT Q-FUNCTIONS FOR DOUBLY EFFICIENT REINFORCEMENT LEARNING
Takuya Hiraoka 1.2, Takahisa Imagawa 2, Taisei Hashimoto 2,3, Takashi Onishi 1.2
Yoshimasa Tsuruoka 2,3
NEC Corporation
2 National Institute of Advanced Industrial Science and Technology
3 The University of Tokyo
{t (takuya-hi, takashi onishi @nec. con, imagawa. t@aist. go. ip, (hashincto-taasse3eeee ecc,
wttee tsurucka@logos.c .t) u-tokyo. ac. ip
ABSTRACT
Randomized ensembled double Q-learning (REDQ) (Chen et all 2021b) has re- cently achieved state-of-the-art sample efficiency on continuous-action reinforce- ment learning benchmarks. This superior sample efficiency is made possible by using a large Q-function ensemble. However, REDQ is much less com- putationally efficient than non-ensemble counterparts such as Soft Actor-Critic (SAC) (Haarnoja et all 2018a). To make REDQ more computationally efficient, we propose method of improving computational efficiency called DroQ, which is a variant of REDQ that uses a small ensemble of dropout Q-functions. Our dropout Q-functions are simple Q-functions equipped with dropout connection and layer normalization. Despite its simplicity of implementation, our experimen- tal results indicate that DroQ is doubly (sample and computationally) efficient. It achieved comparable sample efficiency with REDQ, much better computational efficiency than REDQ, and comparable computational efficiency with that of SAC.
INTRODUCTION
In the reinforcement learning (RI.) community, improving sample efficiency of RL methods has been important. Traditional RL methods have been shown to be promising for solving complex control tasks such as dexterous in-hand manipulation (OpenAl et an. 2018). However, RL methods generally require millions of training samples to solve task (Mendonca ctall 2011)). This poor sample efficiency of RL methods is severe obstacle to practical RL applications (e.g., applications on limited computational resources or real-world environments without simulators). Motivated by these issues, many RL methods have been proposed to achieve higher sample efficiency. For example, Haarnoja et al. (201))b)) proposed Soft Actor-Critic (SAC), which achieved higher sample efficiency than the previous state-of-the-arr RL methods (Lillicrap all, 2015; Fujimoto et all, 2018 Schulman et al. 2)1.)
Since 2019, RL methods that use a high update-to-data (UTD) ratio to achieve high sample effi- ciency have emerged. The UTD ratio is the number of updates taken by the agent compared to the number of actual interactions with the environment. high UTD ratio promotes sufficiently train- ing Q-functions within a few interactions, which leads to sample-efficient learning. Model-Based Policy Optimization (MBPO) (Janner all 2019) seminal RL method that uses high UTD ratio of 20-40 and achieves significantly higher sample efficiency than SAC, which uses a UTD ratio of 1. Encouraged by the success MBPO, many RI. methods with high UTD ratios have been proposed (Shen M121:
Larcial 2020)
With such methods, randomized ensembled double Q-learning (REDQ) proposed by Chen et al (202/ is currently the most sample-efficient method for the MuJoCo benchmark. REDQ uses high UTD ratio and large ensemble D-ffucction. The use of high UTD ratio increases an estima- tion bias in policy evaluation, which degrades sample-efficient learning. REDQ uses ensemble of Q-functions to suppress the estimation bias and improve sample efficiency. Chen et all (20216) demonstrated that the sample efficiency of REDQ is equal even better than that of MBPO.

---

Published as a conference paper at ICLR 2022
However, REDQ leaves room for improvement in terms of computational efficiency. REDQ runs 1.1 to 1.4 times faster than MBPO @Chen.etall 202TH) but is still less computationally efficient than cun-ensemble-basee RL methods (e.g.. SAC) due to the use of large ensembles. In Section E.2 we show that REDQ requires more than twice as much computation time per update, and much larger memory. Computational efficiency is important in several scenarios, e.g., in RI. applications with much lighter on-device computation (e.g. mobile phones or other light-weight edge devices) (Chen etall 202Ta), or in situations in which rapid trials and errors are required for developing RL agents (e.g., hyperparameter tuning or proof of concept for RL applications with limited time resources). Therefore, RL methods that are superior not only in terms of sample efficiency but also in computa- tional efficiency are preferable.
We propose a method of improving computational efficiency. Our method is called DroQ and is a REDQ variant that uses a small ensemble of dropout Q-functions, in which dropout (Srivastand 2L ?ME)) and layer normalization (Ba et al 2076) are used (Section T. We experimentally
show that DroQ is doubly (computationally and sample) efficient: (i) DroQ significantly improves
computational efficiency over REDQ (Section Ewe) by more than two times and (ii) achieves sample
efficiency comparable to REDQ (Section I..
Although our primary contribution is proposing a doubly efficient RL method, we also make three significant contributions from other perspectives:
1. Simplicity of implementation. DroQ can be implemented by basically adding a few lines of readily available functions (dropout and layer normalization) to Q-functions in REDQ (and SAC). This simplicity enables one to easily replicate and extend it.
2. First successful demonstration of the usefulness of dropout in high UTD ratio settings. Pre- vious studies incorporated dropout into RL (Gall & Ghahramani, 2016: Farrgann 2016; Moerland
stall 2017; Gal all 2016: 2017: Kahn al, 2017: Daques et all 2019; He et all, 202IT (see Section B for details). However. these studies focused on low UTD ratio settings (i.c., UTD ratio
< 1). Dropout approaches generally do not work as well as ensemble approaches (Oshand et all 2016; Ovadia et all, 2019; Laakshinnrayaaan et al] 201; Durasov et all, 1021). For this reason, instead of dropout approaches, ensemble approaches have been used in RL with high UTD ratio settings (Chen all 20210 Janner et all 20T9; Shen atl MIII Hmaoka el all PI2T: La cl all 2020). In Section H we argue that DroQ achieves almost the same or better bias reduction ability and sample/computationallyy efficiency compared with ensemble-basee settings. This sheds light on dropout approaches once again and promotes their use as a reasonable alternative (or complement) ensemble approaches in high UTD ratio settings.
RL methods in high UTD ratio
3. Discovery of engineering insights to effectively apply dropout to RL. Specifically, we discov- ered that the following two engineering practices are effective in reducing bias and improving sample efficiency: (i) using the dropout and ensemble approaches together for constructing Q-functions (i.c., using multiple dropout (-functions) (Section I and Appendix A::): and (ii) introducing layer nor- malization into dropout Q-functions (Section 3 and Appendices D. E., and E). These engineering insights were not revealed in previous RL studies and would be useful to practitioners who attempt to apply dropout to RL.
2 PRELIMINARIES
2.1 MAXIMUM ENTROPY REINFORCEMENT LEARNING (MAXIMUM ENTROPY RL)
RL addresses the problem of an agent learning to act in environment. At each discrete time step t, the environment provides the agent with a state st, the agent responds by selecting an action az, and then the environment provides the next reward Fr and state 8t+1. For convenience, needed, we use the simpler notations of r, a, s', and a' to refer to a reward, state, action, next state, and next action, respectively.
We focus on maximum entropy RL, in which an agent aims to find its policy that maximizes the expected return with entropy bonus: arg max Et "Ea,a, Ire + (+)(s((sss)) Here, π S x
x
A + [0, oo) is a policy and H is entropy. Temperature a balances exploitation and exploration and affects the stochasticity of the policy.
2

---

Published as a conference paper at ICLR 2022
Algorithm 1 REDD
Initialize policy parameters 0, .N Q-function parameters 6i 1,.., N, and empty replay buffer D. Set target parameters Ø t Φi. for 1,.... N.
2: repeat
3: Take action d; mT)- Observe reward next state s + DU(s T2,
4: for G updates do
5: Sample mini-batch B = {(s, a, &,)) from D.
6: Sample set M M distinet indices from (I.,...,,N N}.
7:
Compute the Q target y (same for all N Q-functions):
min Qo (s',a') log (o(oo o' mo(ls')) iEM
y ツーr+7
8:
for i i= I, .... N do
9:
Update Φ; with gradient descent using
Σ
(Qo, -y²²
B
(s.a,T;"))EB
Update target networks with oi + pā: (1 (1-p) p)o: Update 9 with gradient ascent using
11:
(s, log logg a a~
8
B
N
SEB
1=1
2.2 RANDOMIZED ENSEMBLED DOUBLE Q-LEARNING (REDQ)
REDQ (Chen all 20216) is a sample-efficient model-free method for solving maximum-entropy
RL problems (Algorithm M.. It has two primary components to achieve high sample efficiency.
High UTD ratio: uses high UTD ratio G. which the number updates (lines 4-10) taken by the agent compared to the number of actual interactions with the environment (line 3). The high UTD ratio promotes sufficient training Q-functions within few interactions. However, this also increases overestimation bias in the Q-function training, which degrades sample-efficient learn- ing (Chen et all 20216).
2. Ensemble of Q-functions: To reduce the overestimation bias, it uses ensemble of N Q- functions for the target to be minimized (lines 6-7). Specifically, a random subset M of the en- semble selected (line then used for calculating the target (line 7). The size of the subset M is kept fixed and is denoted as M. In addition, each Q-function in the ensemble is randomly and independently initialized but updated with the same target (lines 8-9). Chen all (2021B) showed that using large ensemble (N 10) and small subset (M = 2) successfully reduces the bias.
Although using large ensemble of Q-functions is beneficial for reducing bias and improving sample efficiency, this makes REDQ computationally intensive. In the next section, we discuss reducing the ensemble size.
3 INJECTING MODEL UNCERTAINTY INTO TARGET WITH DROPOUT Q-FUNCTIONS
In this section, we discuss replacing the large ensemble of Q-functions in REDQ with small ensem- ble dropout Q-functions. We start our discussion by reviewing what the ensemble in REDQ does from the viewpoint of model uncertainty injection. Then, we propose to use dropout for model un- certainty injection instead of the large ensemble. Specifically, we propose (i) the dropout Q-function that is O-function equipped with dropout and layer normalization. and (ii) DroQ. variant of REDQ that uses a small ensemble of dropout f-uuctiooss. Finally, we explain that the size of the ensemble be smaller in DroQ than in REDQ.
3

---

Published as a conference paper at ICLR 2022
Dr (s,a) a)
a
LayerNorm
Weight
Dropout
LayerNou
ReLU
Weight
Dropout
ReLU
Weight
Dr
Q Dr.
Dr
Min
Figure 1: Dropout Q-function implementation (left part) and how dropout Q-functions are used in target (right part). Dropout Q-function implementation: Our dropout function is implemented by modifying that used by eh (202TT). Our modification (highlighted in red) is adding dropout (Dropout) and layer normalization (L.ayerNorm). "Weight" is a weight layer and "ReLU" is the activation layer of rectified linear units. Parameters Φ; represent the weights and biases in weight layers. How dropout Q-functions are used in target: M dropout Q-functions are used to calculate the target value min;=11..,,M Dr, (s,a).
We first explain our insight that, in REDQ, model (Q-function parameters') uncertainty is injected into the target. REDQ, the subset M of N Q-functions is used to compute the target value millie Q7, (d',a)) a') (lines 6-7 in Algorithm M.. This can be interpreted as an approximation for B [min;=1.....uu Qū (8', aa)], the expected target value with respect model uncertainty:
min 2s (5'd (a(6))00)=
E Eg.....
=
min (s,as' i=I,..,,M
II
plo.)do
i=I.....M
i=I.....M
min 1=.....M
as' 2mm
II
Jen (oi)dō
~
min (s',a)
iEM
(used in line 7 in Algorithm M.
On the left hand side in the second line, the model distribution p(o:) is replaced with proposal distribution Jen. which based on resampling from the ensemble in line in Algorithm n'. On the right hand side (RHS) in the second line, the expected target value is further approximated by one sample average on the basis of Jen- The resulting approximation is used line 7 in Algorithm MI In the remainder of this section, we discuss another means to approximate the expected target value.
We use M dropout Q-functions Qi for the target value approximation (Figure M. Here, QDr is Q-function that has dropout connections (Srivastava et all 2014). The left part of Figure [I shows Qu implementation by adding dropout layers to the Q-function implementation used by Chen et all (20216). Layer normalization (Ba all 2016) is applied after dropout for more effectively using dropout as with Vaswani all (2017;; ; Xn all (2019). By using QDr, the target value is approxi-
mated
:I.....M 2o (6
min
min Qú (S,a') i1....
II
qar( )dp
min M.
Q ,a)
Instead of gen. proposal distribution qär based the dropout is used on the RHS in the first line. The expected target value is further approximated by one sample average on the basis of qär in the second line. We use the resulting approximation min;=1.....M Q (s', a') for injecting model uncertainty into the target value (the right part of Figure M... For calculating min Q 3, (s',a'), M dropout Q-functions, which have independently initialized and trained parameters Ø1, ...; Tw- Using M dropout Q-functions improves the performance of DroQ (our RL method described in the next paragraph), compared with that using a single dropout Q-function (further details are given in A
We now explain DroQ, in which QDr is used for considering model uncertainty. The algorithmic description of DroQ is shown in Algorithm Ø DroQ is variant of REDQ, and the modified parts
We assume that YI &i independently follow identical distribution.
4

---

Published as a conference paper at ICLR 2022
Algorithm 2 DroQ
Initialize policy parameters 0, M Q-function parameters 6i i 1,,.. M, and empty replay
buffer D. Set target parameters Ø + @i, for i L,..,,
2: repeat
mT)- Observe reward next state St41; + DU(st, Fr 4: for G updates do
3: Take action a,
5: Sample mini-batch B = {(s, a,,r, from D.
6:
Compute the Q target y for the dropout Q-functions:
y =
min I....
Vo, - a logg)())))))) ((1
d' d' m
7:
for i I,.., M do Update Φ; with gradient descent using
8:
Σ
(s, =y)
B
(8.0,,,8')BBB
9:
Update target networks with oi + pK: + (1 p)o: Update 9 with gradient ascent using
M
Qoio, (s, (s a log alog [o([|s)) ~ ~
M
SEB
i=1
from REDQ are highlighted in red in the algorithm. In line 6, QD is used to inject model uncertainty into the target, as discussed in the previous paragraph. In lines 8 and 10, M dropout Q-functions are used instead of N Q-functions (N 2 M) to make DroQ more computationally efficient.
The ensemble size of the dropout Q-functions for DroQ (i.e., M) should be smaller than that of Q-functions for REDQ (i.e., N). M for DroQ is equal to the subset size for REDQ, which is not greater than N. In practice, M is much smaller than N (e.g., M == 2 and N 10 in Chen et all (2021H)). This reduction in the number of Q-functions makes DroQ more computationally efficient. Specifically, DroQ is faster due the reduction in the number of Q-function updates (line 8 in Algorithm L). DroQ also requires less memory for holding Q-function parameters. In Section 8.2 we show that DroQ computationally faster and less memory intensive than REDQ.
EXPERIMENTS
We conducted experiments to evaluate and analyse DroQ. In Section we evaluate DroQ's perfor- mance (sample efficiency and bias-reduction ability). In Section 2. we evaluate the computational efficiency of DroQ. Section !
we present an ablation study for DroQ
4.1 SAMPLE EFFICIENCY AND BIAS CAASRRDDUTIOON ABILITY OF DROQ
To evaluate the performances of DroQ, we compared DroQ with three baseline methods in MuJoCo benchmark environments (Iodoroo et all 2012 Broekman et all 2016). Following Chen et all (202I6); lanner
(20I)) we prepared the following environments: Hopper. Walker2d, Ant, and Humanoid. In these environments, we compared the following four methods:
REDQ: Baseline method that follows the REDQ algorithm (Chen all 20215) (Algorithm M.
SAC: Baseline method that follows the SAC algorithm (Haarnoja et all 2018at)) To improve the
sample efficiency of this method, delayed policy update and high UTD ratio G 20 were used, as suggested by Chen et (202TB).
DroQ: Proposed method that follows Algorithm
Double uncertainty value network (DUVN): Baseline method. This method is DroQ variant that injects model uncertainty into a target in the same manner as the original DUVN (Moerland et all 2017). The original DUVN uses the output of single dropout Q-function Q (s', without
5

---

Published as a conference paper at ICLR 2022
Table 1: Process times (in msec) per overall loop (e.g., lines 3-10 in Algorithm D). Process times
per Q-functions update (c.g., lines 4-9 in Algorithm 2) are shown in parentheses.
Hopper-v2
Walker2d-v2
Ant-v2
Humanoid-v2
SAC
910
870 (835)
888 (848)
893 (854)
REDQ
2328 (2269)
2339 (2283)
2336 (2277)
2400 (2340)
DUVN
664 (636)
762 (731)
733 (700)
692 (660)
DroQ
948 (905)
933 (892)
954 (913)
989 (946)
layer normalization for target as y = + Dr (s',a') aa))) 0. We introduce the original DUVN to DroQ by modifying target line 6 in Algorithm as y = + Y (Q.r,e (s', a') a log (s))) Here, layer normalization is not applied in Dr. (s', We use this method to investigate the performance of the existing single dropout Q-function (non-ensemble) method.
Following Chen all (20211), we the hyperparameters G 20, N 10, and M = 2 for all methods except DUVN (M- = 1). More detailed hyperparameter settings are given in Appendix HI
The methods were compared on the basis of average return and estimation bias.
Average return: An average return over episodes. We regarded 1000 environment steps in Hopper and 3000 environment steps in the other environments one epoch, respectively. After every epoch, we ran ten test episodes with the current policy and recorded the average return.
Average/std. bias: Average and standard deviation of the normalized estimation error (bias) of Q-functions Q (Chen all 20216). The error represents how significantly the Q-value estimate (Q"($, a)], where Q" (8, a) is the true Q-value under the current policy π and Q(s,, a) is its estimate. In our experiment, Q (s, a) was approximated by the discounted Monte Carlo return obtained with π in the test episodes. In addition, Q(s, a) was evaluated as as) 1/N EN (8,a) for SAC and REDQ and as ((s,a) = 1/M Σ QDi. (s, a) for DroQ and DUVN, respectively.
differs from the true one. Formally, the error is defined as !QT a) - Q(s,, alss,a)//B,aaa
The comparison results (Figure 2 indicate that DroQ achieved almost the same level of perfor- mance as REDQ. Regarding the average return, REDQ and DroQ achieved almost the same sample efficiency overall. In Walker2d and Ant, their learning curves highly overlapped, and there was no significant difference between them. In Humanoid, REDQ was slightly better than DroQ. In Hop- per, DroQ was better than REDQ. In all environments except Hopper, REDQ and DroQ improved their average return significantly earlier than SAC and DUVN. Regarding the bias, REDQ and DroQ consistently kept the estimaatn bias closer to zero than SAC and DUVN in all environments.
4.2 COMPUTATIONAL EFFICIENCY OF DROQ
We next evaluated the computational efficiency of DroQ. We compared DroQ with the baseline methods on the basis of the following criteria: (i) Process time required for executing methods; (ii) Number of parameters of each method; (iii) Bottleneck memory consumption suggested from the Pytorch profilerS, Bottleneck memory consumption is the maximum memory consumption recorded when running the methods. For evaluation, we ran each method on a machine equipped with two Intel(() Xeon(R) CPU E5-2667 v4 and one NVIDIA Tesla K80.
Process times per update (numbers in parentheses in Table M)) indicate that DroQ runs more than two times faster than REDQ. DroQ (and SAC) requires process times in the 800-900-mssec range. REDQ requires process times in the 2200-2300-msee range. Process times also show that learning Q-functions is dominant in an overall loop. This suggests that using compact (e.g., small numbers of) Q-functions is important for improving overall process times.
The number of parameters and bottleneck memory consumption of each method indicate that DroQ more memory efficient than REDQ. Regarding the numbers of parameters (Table D), we can see that those of DroQ (and SAC and DUVN) are about one-fifth those of REDQ. Note that the number of parameters of DroQ is equal to that of SAC since DroQ and SAC use the same number (two) of Q-functions. Regarding the bottleneck memory consumption (Table B), we can see that that for DroQ (SAC and DUVN) is about one-third that for REDQ. We can also see that the bottleneck
Harrigan (2016;; He (2121) also propose similar targets.
https: http://yyorrrcc.cccco http:///tttooooooottotoooooo a T htt//ecpppeee pes/ reci pes /ttp/rofffiic reci pe. htm

---

Published as a conference paper at ICLR 2022
Hopper-2
Hopperv?
Hopperi22
SAC
200C
DUVN
averagg bins
std olasss
10.0
200C
DroO
7.5
5.0
1000
2.5
0.0
-2
0.0
0.2
0-4
0.6
0.8
0.0
0.2
0.8
0.0
0.2
0.4
0.6
0.8
environment interections
environment interections
environiment interact ons
Walleer2d-v2
10
Welker2d-v2
15.0
WalkerZd-v2
5000
12.5
4000
aussage hlss
M
#t bf
2000
2000
0.0
0.0
0.5
10
1.
2.0
2.5
-20
0.0
0.5
100
155
2.0
2.5
0.0
0.5
1.0
1.5
2.0
2.5
environment interactions
le5
environment Interactions
leb
environment interactioo and
1e5
Ant-v2
Ant-v7
Antiv2
6000
12.5
average bins
4000
5dd of blas
10.0
7.5
2000
5.0
M
255
-2
environment intercctions
les
environment interactions Humanold-22
Humansidiv2
Humandic-w2
10
Average blar
10.0
5.0
1000
2.5
0.0
0.0
0.5
1.0
2.0
2.5
0.0
0.5
1.5
2.0
2.5
0.0
0.5
1.0
1.5
2.0 2.5
onvironment interactions
le5
anviroment Intoractions leb
environment interact ane le5
Figure 2: Average return and average/standard deviation of estimation bias for REDQ, SAC, DUVN, and DroQ. The horizontal axis represents the number of interactions with the environment (e.g., the number of executions of line 3 of Algorithm D) For each method, average score of five independent trials are plotted as solid lines, and standard deviation across trials is plotted as transparent shaded region.
Table Number of parameters
Hopper-v2
Walker2d-v2
Ant-v2
Humanoid-v2
SAC
141,826
146,434
152,578
166,402
REDQ
698,890
721,930
752,650
821,770
DUVN
139,778
144,386
150,530
164.354
DroQ
141,826
146,434
152,578
166,402
memory consumption is almost independent of the environment. This is because one of the most memory-intensive parts is the ReLU activation at the hidden layers O
'In experiment, the number of units in hidden layer is invariant environments (Appendix ED).
7

---

Published as a conference paper at ICLR 2022
Table 3: Bottleneck memory consumption (in megabytes) with methods. Three worst bottleneck
memory consumptions are shown in form lIs worst bottleneck memory consumption! 2nd worst bottleneck memory consumption / 3rd worst bottleneck memory consumption."
Hopper-v2
Walker2d-v2
Ant-v2
Humanoid-v2
SAC
73 64 62
73/ 64 62
73 64 62
/ 62
REDQ
241/211 200 200
241 200
241/211 200
241 / 201
DUVN
72/71/51
72 22/71/51
72/71/51
72/71 52
DroQ
73/72/60 69
73/72/68
73/72/70
73/72/70
Hopper-22
Hopper.v2
Hopper
Droo
200C
-D0
12.5
average bins.
5id of blas
10.0
-D0-LN
2000
7.5
5.0
1000
0.4
-2
0.0
0.6 environieent Interect
environment interections
le5
Walker2d-v2
WalkerZdiv2
10
15.0
4000
average blat
std of bles
3000
2000
1000
2.5
0.0
0.0
0.5
1.
2.0
2.5
0.0
0.5
15
2.0
2.5
0.0
0.5
1.0
1.5
2.0
25
onvironment Inferactions
le5
envirmment Interactions.
leb
environment interact and
le5
Ant--22
10
Ant-v2
15.0
Anter2
COCC
12.5
average bias
of bias
10.0
4000
7.5
2000
-2
0.0
0.5
15
2.0
2.5
1.0
interections
165
environment interections
les
environment interact
105
Humandic-v2
Humanoid v2
Hamencidvv2
5000
12.5
4000
average bias
of bles
10.0
3000
7.5
2000
5.0
1000
2.5
10
2.3
25
000
2.0 envimement Interactions
1.0
onvironment Interactions
le5
environment interactions ans
Figure 3: Ablation study results
4.3 ABLATION STUDY
As an ablation study of DroQ, we investigated the performance of DroQ variants that do not use either dropout, or layer normalization, or both. We refer to the one without dropout -DO, that without layer normalization as -LN, and that without both as -DO-LN. The results (Figure B) indi- cate that the synergic effect of using dropout and layer normalization is high, especially in complex environments (Ant and Humanoid). In these environments, DroQ significantly outperformed its ablation variants (-DO, -IN and -DO-LN) in terms of both average return and bias reduction.
8

---

Published as a conference paper at ICLR 2022
Table 4: Comparison between related studies and ours.
We classify related studies into five types
(c.g., "Ensemble Q-functions") on basis of three criteria (c.g. "Type of model uncertainty"),
Method for injecting model uncertainty
Type of model uncertainty
Focus high UTD ratio setting?
Type of study
Partially yes
Ensemble Q-functions
Ensemble
Q-functions
Chen et (202Th)
Partially yes e.g., lanner (2019)
Ensemble transition models
Ensemble
Transition models
Dropout Q-functions
Dropout
Q-functions
No
Dropout transition models
Dropout
Transition models
No
Normalization Our study
No
Dropout (with ensemhle)
Q- Q-functions
Yes
RELATED WORK
In this section, we review related studies and compare them with ours (Table B)).
Ensemble Q-functions: Ensembles of Q-functions have been used RL to consider model uncer- tainty (Canker & Schwenker, 2015: Oshand et al, 201G Anschel et all 2017: Osband al 2018
Agarwal al. 2020: etal M02T Lan all 2020; Chen all 202TE). Ensemble transition models: Ensembles of transition (and reward) models have been introduced to model-based RL, e.g., (Chua et all 2018; Kurutach et all, 2018; [annnr et all 2019; et 2020 eu 2021;
Lee all 2020, Eiraoka et all, 202I: Abraham et all, 2020). The methods proposed in the above studies use a large ensemble of Q-functions or transition models, thus are computationally intensive.
DroQ does not a large ensemble of Q-functions, and thus is computationally lighter.
Dropout transition models: Ga et all (2016; 2017); Kahn et all (2017) introduced dropout and
its modified variant to transition models of model-based RL methods. Dropout Q-functions: Gal & Ghahramani (2016) introduced dropout to a Q-function at action selection for considering model uncertainty in exploration. Harrigan (20I6;; Moerland et all (201 Daques et al] (2019); He et all (2021) introduced dropout to policy evaluation in the same vein as us. However, there are three main differences between these studies and ours. (i) Target calculation: they introduced dropout to single Q-function and used it for target value, whereas we introduce dropout to multiple Q-functions and use the minimum of their outputs for the target value. (ii) Use of engineering to stabilize learning: their methods do not engineering to stabilize the learning of dropout Q-functions, whereas DroQ uses layer normalization to stabilize the learning. (iii) RL setting: they focused on a low UTD ratio setting, whereas we focused on a high UTD ratio setting. A high UTD ratio setting increases high estimation bias, and thus is a more challenging setting than a low UTD ratio setting. In Sections [ and A we showed that their methods do perform successfully in a high UTD setting.
Normalization in RL: Normalization (e.g., batch normalization (loffe & Szegedy, 2015) or layer normalization (Ba all, 20116 has been introduced into RI. Batch normalization and its variant introduced in deep deterministic policy gradient (DDPG) (Lillicrap et all. 2016) and twin delayed DDPG (Fujumoto et all 20IX) (Bhatt ctall 202()) Layer normalization introduced in the imple- mentation of maximum a posteriori policy optimisation (Abdolmalcki el all 2018; Hollman el all 2020). is also introduced in SAC extensions (Ma all 2020 Zhang et all, 2021). Unlike our study, the above studies did not introduce dropout to consider model uncertainty. In Section P.1 we showed that using layer normalization alone is not sufficient in high UTD settings. In Appendix EL we also show that using batch normalization does not contribute to performance improvement in high UTD settings.
6 CONCLUSION
We proposed, DroQ, an RL method based on a small ensemble of Q-functions that are equipped with dropout connection and layer normalization. We experimentally demonstrated that DroQ sig- nificantly improves computational and memory efficiencies over REDQ while achieving sample efficiency comparable with REDQ. In the ablation study, found that using both dropout and layer normalization had a synergistic effect on improving sample efficiency, especially in complex environments (e.g., Humanoid).

---

Published as a conference paper at ICLR 2022
REFERENCES
Abbas Abdolmalcki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Mar- tin Riedmiller. Maximum posteriori policy optimisation. In Proc. ICLR, 2018.
lan Abraham, Ankur Handa, Nathan Ratliff, Kendall Lowrey, Todd D Murphey, and Dieter Fox. Model-based generalization under parameter uncertainty using path integral control. IEEE Rohotics and Automation Letters, 5(2):2864-2871, 2020.
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In Proc. ICML, 2020.
Oron Anschel, Nir Baram, and Nahum Shimkin. Averaged-DQN: Variance reduction and stabiliza- tion for deep reinforcement learning. Proc. ICML, 2017.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 2016.
Aditya Bhatt, Max Argus, Artemij Amiranashvili, and Thomas Brox. CrossNorm: On normalization for off- off-ppoicy reinforcement learning, 2020.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAl Gym. arXiv preprint arXiv:1606.01544 2016.
Lili Chen, Kimin Lee, Aravind Srinivas, and Pieter Abbeel. Improving computational efficiency in visual reinforcement learning via stored embeddings. arXiv preprint arXiv:2103.0288 202 la
Xinyue Chen, Che Wang, Zijian Zhou, and Keith W. Ross. Randomized ensembled double Q-
learning: Learning fast without model. In Proc. ICLR, 2021b.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Proc. NeurlPS, 2018.
Nikita Durasov, Timur M. Bagautdinov, Pierre Baqué, and Pascal Fua. Masksembles for uncertainty estimation. In Proc. CVPR, 2021.
Stefan Faußer and Friedhelm Schwenker. Neural network ensembles in reinforeement learning. Neural Processing Letters, 41 41(1)555-69, 2015.
Scott Fujimoto. Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In Proc. ICML, pp. 1587-1596. PMLR, 2018.
Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In Proc. ICLR, pp. 1050-1059. PMLR, 2016.
Yarin Gal, Rowan McAllister, and Carl Edward Rasmussen. Improving PILCO with Bayesian neural network dynamics models. In Data-Efficient Machine Learning workshop ICML, 2016.
Yarin Gal, Jiri Hron, and Alex Kendall. Concrete dropout. In Proc. NeurlPS, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In Proc. ICML, pp. 1861- 1870. PMLR, 2018a
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, and Pieter Abbeel. Soft actor-critic algorithms and applica- tions. arXiv preprint
2018b.
Cosmo Harrigan. Deep reinforcement learning with regularized comvolutional neural fitted Q itera- tion. https:/ www htt.e//eeee....e........ooo/eaap
nfq. odE, 2016.
Qiang He, Chen Gong, Yuxun Qu, Xiaoyu Chen, Xinwen Hou, and Yu Liu. MEPG: A min- imalist ensemble policy gradient framework for deep reinforcement learning. arXiv:2109.1055 2021.
arXiv preprint
10

---

Published as a conference paper at ICLR 2022
Takuya Hiraoka, Takahisa Imagawa, Voot Tangkaratt, Takayuki Osa, Takashi Onishi, and Yoshimasa Tsuruoka. Meta-mosl Meta-model--a model-based meta-policy optimization. In Proc. ACML, 2021.
Matt Hoffmman, Bobak Shahriari, John Aslanides, Gabriel Barth-Maron, Feryal Behbahani, Tamara Norman, Abbas Abdolmaleki, Albin Cassirer, Fan Yang, Kate Baumli, Sarah Henderson, Alex Novikov, Sergio Gomez Colmenarejo, Serkan Cabi, Caglar Gulcehre, Tom Le Paine, Andrew Cowie. Ziyu Wang, Bilal Piot, and Nando de Freitas. Acme: A research framework for distributed reinforcement learning. arXiv preprint arXiv:2006.00979 2020.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proc. ICML, 2015.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model- based policy optimization. In Proc. NeurIPS, 2019.
Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint DEXiv:1907.00455 2019.
Gregory Kahn, Adam Villaflor, Vitchyr Pong, Pieter Abbeel, and Sergey Levine. Uncertainty-awant reinforcement learning for collision avoidance. arXiv preprint 1rXiv:1702.01182 2017.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In Proc. ICLR. 2015.
Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-ensemble trust-region policy optimization. In Proc. ICLR, 2018.
Hang Lai, Jian Shen, Weinan Zhang, and Yong Yu. Bidirectional model-based policy optimization. In Proc. ICML, 2020.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Proc. NeurlPS, pp. 6402-6413, 2017.
Qingfeng Lan, Yangehen Pan, Alona Fyshe, and Martha White. Maxmin Q-learning: Controlling the estimation bias of Q-learning. In Proc. ICLR, 2020.
Kimin Lee, Younggyo Seo, Seunghyun Lee, Honglak Lee, and Jinwoo Shin. Context-aware dynam- ics model for generalization in model-hased reinforcement learning. In Proc. ICMI,, 2020.
Kimin Lee, Michael Laskin, Aravind Srinivas, and Pieter Abbeel. SUNRISE: simple unified framework for ensemble learning in deep reinforcement learning. In Proc. ICML, 2021.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint wtxiv:1509.0297.1 2015.
Xiaoteng Ma, Li Xia, Zhengyuan Zhou, Jun Yang, and Qianchuan Zhao. DSAC: Distributional soft actor critic for risk-sensitive reinforcement learning. In Reinforcement Learning for Real Life Workshop at ICML 2019, 2020.
Russell Mendonca, Abhishek Gupta, Rosen Kralev, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Guided meta-policy search. In Proc. NeurlPS, pp. 9653-9664, 2019.
Thomas M Moerland, Joost Broekens, and Catholijn M Jonker. Efficient exploration with double uncertain value networks. arXiv preprint arXiv:1711. 2017.
OpenAI, Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Józefowicz, Bob McGrew, Jakub W. Pachocki, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, Jonas Schneider, Szymon Sidor, Josh Tobin, Peter Welinder, Lilian Weng, and Wojciech Zaremba. Learning dexterous in hand manipulation. CoRR, abs/1808.001 2018.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped DQN. In Proc. NeurlPS, 2016.

---

Published as a conference paper at ICLR 2022
Ian Osband, John Aslanides, and Albin Cassirer. Randomized prior functions for deep reinforcement learning. Proc. NeurIPS, 2018.
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. In Proc. NeurlPS, 2019.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1rXiv:1707.06347 2017.
Jian Shen, Han Zhao, Weinan Zhang, and Yong Yu. Model-based policy optimization with unsuper- vised model adaptation. In Proc. NeurIPS, 2020.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting Journal of Machine Learning Research, 15656):1999-195, 2014.
Emanuel Todorov, Tom Erez, and Yuval Tassa. MuloCo: physics engine for model-based control. In Proc. IROS, pp. 5026-5033. IEEE, 2012.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proc. NeurIPS, 2017.
Yuxin Wu and Kaiming He. Group normalization. In Proc. ECCV, 2018.
Jingjing Xu, Xu Sun, Zhiyuan Zhang, Guangxiang Zhao, and Junyang Lin. Understanding and improving layer normalization. In Proc. NeurlPS, 2019.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: model-basee offline policy optimization. In Proc. NeurIPS, 2020.
Amy Zhang, Rowan Thomas McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. In Proc. ICLR, 2021.
12

---

Published as a conference paper at ICLR 2022
A EFFECT OF DROPOUT RATE ON DROQ AND ITS VARIANTS
A.1
DROQ
Hopper-22
10
Hopper.v2
15.0
Hoppery2
300C
0.1
12.5
avsrage blas
10.0
0.005
2000
1000
0.0
0
4.4
0.88
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
environment interactions
le5
envimmment Interactions
1eb
environment Interact ans
1e5
Walker2d
Walker2d-v2
Va Malleer2div2
5000
12.5
average bins
10.0
3000
7.5
2000
5.0
1000
2.5
-2
environment interactions
environment interections
environment
Ant-v2
10
000C
5000
average blas
4000
3000
5.0
2000
2.5
1000
0.0
0.0
0.5
10
1.
233
2.5
0.0
0.5
1.5
2.0
2.5
0.0
0.5
1.0
1.5
2.0
2.5
environment interactions.
le5
anviroment interactions
les
environment interact ans le5
Humenoid v2
10
Humanald-v2
15.0
Humaindl-v22
5000
12.5
4000
average bass
10.0
8000
7.5
2000
1000
0.0
2.0
2.5
1.0
les
environment interactions le5
environment interact
Figure 4: Average return and average/standard deviation of estimation bias for DroQ with different dropout rates. Scores for DroQ are plotted as solid lines and labelled in accordance with dropout rates (e.g.."0. "0.2"".
13

---

Published as a conference paper at ICLR 2022
A.2
DROQ WITHOUT LAYER NORMALIZATION
Hopper.v2
Hopper-22
Hopparv2
2500
0.2
12.5
200C
average binc
533 of ales
10.0
1500
0.005
7.5
2000
5.0
500
0.0
environment
environmentt interect
Walker2ddvv v2
10
Walker2d-v2
15.0
Welkker2-v2
5000
1000
average bar:
3000
1000
2.5
0.0
0.0
0.5
1.0
1.5
2.0
2.5
0.0
0.5
1.5
2.0
2.5
0.0
0.5
1.0
1.5
2.0
2.5
environmment interectionss
le5
environment Interactions
le5
environmeent interact oni
1e5
Aitt-12
10
Ant-v2
15.0
Ant-v2
12.5
average bias
10.0
7.3
2000
0.0
0.5
15
2.0
2.5
1.0
interactions
le5
environment interections
le5
Humanaid
Humencid-v2
4000
3000
average blas
10.0
2000
7.5
5.0
1000
2.5
0.0
-0.0
0.0
0.5
10
1.5
2.0
2.5
2.0 environmnnt intoractions
0.0
0.5
1.0
2.0
2.5
environment interections
environmentt interact
Figure 5: Average return and average/standard deviation of estimation bias for DroQ without layer normalization (i.c., -LN in Section ( with different dropout rates
14

---

Published as a conference paper at ICLR 2022
A.3 SIN- SIN-DROQ: DROQ VARIANT USING A SINGLE DROPOUT Q-FUCCTION
DroQ (Algorithm Zi uses an ensemble of multiple (M) dropout Q-functions. This raises the question: "Why not single dropout Q-function for DroQ?" To answer this question, we compared DroQ with Sin-DroQ, a variant of DroQ that uses a single dropout Q-function. Specifically, with Sin- DroQ, the target line in Algorithm 0 is calculated by evaluating Q (s', a'), single dropout Q-function, M times:
y =-r+y min Dr (s',d),.. Dr (s', a log re(d'ss)) d' =e(1-')))
evaluation results Qu.,a (s"
Note that, the output of Qr. (s', a') can differ in each evaluation due to the use of dropout connection. It should be also noted that this target is the same the one proposed in Jaques et al. (201%). The remaining part of Sin-DroQ is the same as DroQ.
From the comparison results of DroQ and Sin-DroQ (Figure K), we that the average return of Sin-DroQ was lower than that of DroQ. This result indicates that using multiple dropout Q-functions preferable using a single dropout Q-function with DroQ.
15

---

Published as a conference paper at ICLR 2022
Hopper-2
10
Hopper-v2
15.0
Hopper.v2
300C
0.2
12.5
average bibs
10.0
200C
0.005
100C
0.0
0.4
0.2
0.4
0.6
0.6
0.6 environment Interact ons
environmmnt interections
les
environment interactions les
1e5
Welker2div2
Walker2div2
5000
12.5
auotage biss
std ba D5
10.0
3000
7.5
2000
5.0
2.5
-000
ervironment interactions
les
environment interactions
cnvironment ons
Ant-v22
10
15.0
6000
12.5
average bibs.
of alss
2000
2.5
0.0
0.0
0.5
10
1.5
2.
2.5
0.0
0.5
1.0
1.5
2.0
2.5
0.0
0.5
1.0
1.5
2.0
2.5
environientt interectionss
le5
environment interections
les
environment interact ons
105
Humandie.v2
10
Humanoid-v2
15.0
Humandldi422
5000
12.5
m
4000
aurtage has
10.0
0000
7.5
2000
1000
22.00
2.0 interactions
1.0
anvironment interactions Ie5
interact
Figure 6: Average return and average/standard deviation of estimation bias for DroQ and Sin-DroQ Scores for DroQ are plotted as dash lines. Scores for Sin-DroQ are plotted as solid lines and labelled in accordance with dropout rates (e.g.. "0.2").
16

---

Published as a conference paper at ICLR 2022
B REDQ WITH DIFFERENT ENSEMBLE SIZE N
In Section E. we discussed comparing DroQ with REDQ, which uses an ensemble size 10 (i.e., N=10). To make more detailed comparison, we compared DroQ and REDQ by varying the ensem- ble size for REDQ. We denote REDQ that uses an ensemble size of N as "REDQN" (e.g., REDQS for REDQ with an ensemble size of five).
Regarding average return (left part of Figure 2), overall, DroQ was superior to REDQ2-5. DroQ was comparable with REDQ2 and REDQ3 Hopper but superior in more complex environments (Walker, Ant, and Humanoid). In addition, DroQ was somewhat better than REDQS in all envi- ronments. Regarding estimation bias (middle and right parts of Figure Z) overall, DroQ was sig- nificantly better than REDQ2 and comparable with REDQ3-10, Regarding the processing speed (Table HD, DroQ ran as fast as REDQ3 and faster than REDQS by 1.4 to 1.5 times. Regarding memory efficiency (Tables B and [Α) DroQ was less memory intensive than REDQ3-10.
Hopper-22
Hopper.v2
Hopper-v2
DroQ
300C
REDO2
12.5
average bias
REDOS
d2 of ales
10.0
2000
REDQ10
7.5
5.0
100C
0.4
0.0
environment interactions
Walker2d
10
Walker2d-v2
15.0
Walker2d-v2
5000
4000
average biat
blas
10.0
3000
2000
5.0
1000
2.5
0.0
0.0
0.5
10
15
23
25
0.0
0.5
15
2.0
2,5
0.0
0.5
1.0
2.0
255
environment interactionis
le5
environment interactions
le5
environment interoct and
Ant-w2
10
Ant-v2
15.0
Ant.v2
C0DC
12.5
average bibs
10.0
4000
755
2000
0.0
0.5
1.0
15
2.0
2.5
1.0
interactions
leb
environment interactions
le5
interact
Humtnnic
Humanold-v2
Humencid-v2
5000
4000
average bins bian
10.0
3000
of
7.5
2000
5.0
1000
2.5
0.0
-2.00
0.0
0.5
10
1.5
2.3
2.5
2.0 environment interactions
0.0
0.5
1.0
2.0 2.5
environment interactions
environment interact ons
Figure 7: Average return and average/standard deviation of estimation bias for DroQ and REDQ. Scores for DroQ are plotted as dash lines. Scores for REDQ are plotted as solid lines and labelled "REDQN".
17

---

Published as a conference paper at ICLR 2022
Table 5: Process times (in issc) per overall loop (e.g., lines 3-10 in Algorithm D Process times
per Q-functions update (e.g., lines 4-9 in Algorithm are shown in parentheses.
Hopper-v2
Walker2d-v2
Ant-v2
Humanoid-v2
DroQ
948 (905)
933 (892)
954 (913)
989 (946)
REDQ2
832 (792)
802 (768)
675 (641)
820 (773)
REDQ3
1052 (1014)
876 (838)
919 (881)
950 (906)
REDQ5
1414 (1368)
1425 (1378)
1373 (1327)
1552 (1503)
REDQIO
2328 (2269)
2339 (2283)
2336 (2277)
2400 (2340)
Table 6: Number of parameters
Hopper-v2
Walker2d-v2
Ant-v2
Humanoid-v v2
DroQ
141,826
146,434
152,578
166,402
REDQ2
139,778
144.386
150,530
164,354
REDQ3
209,667
216,579
225,795
246,531
REDQ5
349,445
360,965
376,325
410,885
REDQI0
698,890
721,930
752,650
821,770
Table 7: Bottleneck memory consumption (in megabytes) with methods. Three worst bottleneck
memory consumptions are shown in form of "Ist worst bottleneck memory consumption 2nd worst bottleneck memory consumption / 3rd worst bottleneck memory consumption."
H2pper-v2
Walker2d-v2
Ant-v2
Humanoid-v2
DroQ
73 69
73 69
73 73//2 72 70
73 70
REDQ2
53/51/51
73/51/51
73/51 /5
73 52/2//
REDQ3
9 64 / 62
94 / 64 62
94 / 64 162
94 65/65/ / 62
REDQS
136 106 100
136/ 106 100
136/ 106 100
136/107 101
REDQIO
241 241/211/2 200
241
200
241 / 200
241 / 2112/12// 201
18

---

Published as a conference paper at ICLR 2022
C ADDITIONAL ABLATION STUDY OF DROQ
Dropout is introduced into three parts of the algorithm for DroQ (i.e., lines 6, 8 and 10 of Algorithm in Section B). In this section, we conducted an ablation study answer the question "which dropout introduction contributes to overall performance improvement of DroQ?" We compared DroQ with its following variants:
DO@@ TargetQ: A DroQ variant that does not use dropout in line 6. Specifically, dropout is not used in QD (8', a') in the following part in line 6.
y = +
min A I=.....
DD (s', - log mo(a'ls') d' ~ TT(-|s')
-DO @ CurrentQ: A DroQ variant that does not use dropout in line 8. Specifically, dropout is not used in Qrr,o (8, a) in the following part in line 8.
V B
Σ
(s, y)²
-DO@ -DO@@ PpticyOpt: A DroQ variant that does not use dropout line Specifically, dropout not used in Qr a) in the following part in line 10.
M
1
V 0 B
Quió (s, a) alog alogmo((ass)
tT(ss)
-DO: A DroQ variant that does not use dropout in lines 6, 8, and 10.
In this ablation study, we compared the methods on basis of average return and estimation bias.
The comparison results (Figure M) indicate that using dropout for both current and target Q-functions (i.e., QD. (s',a') and QD. (s, a) in lines 6 and 8) is effective. Regarding average return, the DroQ variants that do not use dropout for either target Q-functions in line 6 or current Q-functions line perform significantly worse than that of DroQ. For example, in Ant, the variants that do not use dropout for target Q-functions (-DO @ TargetQ and -DO) perform much worse than DroQ. In addition, in Humanoid, the variants that do not use dropout for current Q-functions (-DO@ CurrentQ and -DO) perform much worse than DroQ. Regarding estimation bias, that of the variants that do not use dropout for target Q-functions (-DO@) @'argeet and -DO) is significantly worse than that of DroQ in all environments.
These results complement the results presented He (2021). They show that using dropout for both current and target Q-functions improves the performance of the maximum entropy RL method (SAC) in low UTD settings. Our results presented in the previous paragraph implies that their insight (i.e., "using dropout for both current and target Q-functions improves the performance") is valid, even in high UTD settings.
19

---

Published as a conference paper at ICLR 2022
Hopper-22
Hopper.v2
Hoppariv2
Dro0
200C
Doggargeno DogCurento
average bias
10.0
D0gPolicyOpt
2000
7.5
5.0
1000
2.5
0.0
0.0
0.0
0.2
0.4
0.6
0.6
0.2
0.0
0.2
0.6
errii invirommmnt interections
environment interactions Walker2d-v2
environmentt interact ons
Walker2d v2
10
15.0
Walkee22dv2
5000
12.5
4000
zuerage han
2000
1000
0.0
-2.0
0.0
0.5
10
155
2.0
2.5
0.5
1.0
155
2.0
2.5
0.0
0.5
1.0
1.5
2.0
2.5
anvi/onment Interactions
les
environment interactions
le5
environment interact 18b
Ant-v2
Ant.v2
Ant-r2
12.5
average binc
Rd of ales
10.0
4000
7.5
5.0
2000
0.0
ervironment interactions
environment
Humancic-v2 v2
10
Humanold-v2
15.0
Humeneid-v2
average bls
10.0
st blos
2000
5.0
1000
2.5
0.0
0.0
0.5
1.0
1.5
2.0
2.5
0.0
0.5
15
2.0
2.5
0.6
0.5
1.0
2.0
25
environmeent interectiones
1c5
environment Interactions
le5
environment
Interoctions
leb
Figure Additional ablation study result about dropout for DroQ
20

---

Published as a conference paper at ICLR 2022
220
Ar1-2
200
Ar1-2
0.59
An--2
0.10
Antro2
0.00
G0-4
0.30
20
20
0.00
30
35
1.
-0.0
03
35
1.0
155
2.5
25
30
45
10
interartionn
Humar 68.42
ECO
0.35
0.00
300
420
9:5
200
Seceago
9.10
220
+025
50
envronment Interactione 165
envrrrmmnt Interaction 105
andicrmmets Interaccion 323
anvironmenti Interactions
Figure 9: Average/standard deviation of the Q-function loss ("average Q loss" and "std of Q loss") and its gradient with respect to parameters ("average Q gradient" and "std of Q gradient"). Q-function loss is calculated as 37 Ei Li, where L, (s.aa) - (min Qa, (s', (s), a log (o('ss')))) and its gradient is calculated as M EE 8L, The definition of method labels (e.g.. "DroQ" "-DO") in the legend the same as that used in Figure
Do:
D WHY IS THE COMBINATION OF DROPOUT AND LAYER NORMALIZATION IMPORTANT?
Figure B in Section 3 showed that combining dropout and layer normalization improves overall performance, especially in complex environments (Ant and Humanoid).
The main reason for this performance improvement could be that layer normalization suppresses the learning instability caused by dropout. Using dropout destabilizes the Q-functions learning. Figure B shows that (i) the Q-function loss of the method using dropout (-LN) oscillates more sig- nificantly than that of the method not using dropout (-DO-LN), and that (ii) the variance of the gradient loss with respect to the Q-function parameters also oscillates accordingly, Layer normaliza- tion suppresses such learning instabilities. The right part of Figure & shows that the method using both dropout and layer normalization (DroQ) suppresses the oscillations of the gradient more sig- nificantly than the -LN. This stabilization of learning could enable better Q-value propagation and consequently improve overall performance.
21

---

Published as a conference paper at ICLR 2022
E EFFECTS OF OTHER NORMALIZATION METHODS
In Section k and Appendix D. we demonstrated that combining dropout and layer normalization is effective in improving performance. In this section, we examine the effectiveness of normalization methods other than layer normalization.
We examine the effects of (i) batch normalization (lotte & Szegedy, 2015), (ii) group normaliza- tion (Wi & Hc, (0TT), and (ii) layer normalization without variance re-scaling, Batch normaliza- tion an off-the-shelf and popular normalization method, and has been introduced into RL in low UTD settings (Bhatt all 2020). We introduce this popular method to verify whether it is effective in high UTD settings. Group normalization is a method of normalization using the average and variance over the elements of a target input, which is similar to layer normalization We introduce this method to verify whether method equipped with a similar mechanism to layer normalization works well in high UTD settings. Layer normalization without variance re-scaling is the layer nor- malization variant that does not re-scale the target input's 's variance. et all (2019) pointed out that variance re-scaling is important component for layer normalization to work well. We introduce this method to verify whether variance re-scaling is an important component also in our settings.
We compare six DroQ variants that these three normalization methods. We denote -DO vari- ants using batch normalization, group normalization, and layer normalization without variance re- scaling, instead of layer normalization, +BN, +GN, and +LNwo VR, respectively. We also denote DroQ variants using batch normalization, group normalization, and layer normalization without variance re-scaling, instead of layer normalization, +DO+BN, +DO+GN, and +DO+LNwoVR, respectively. We compare these six methods (+BN, +GN, +LNwoVR, +DO+BN, +DO+GN, and +DO+LNwoVR).
From the experimental results of the six methods (Figures IQ and IT],, we get the following insights: Batch normalization does not work well. Figure IIO shows that the methods using batch normal- ization (+BN and +DO+BN) do not significantly improve the average return and estimation bias in Humanoid and Ant. Further, Figure M shows that the Q-function learning with these methods is very unstable.
2. Group normalization, which has a similar mechanism to layer normalization, works well when combined with dropout. Figure I shows that the method using the group normalization and dropout (+DO+GN) successfully improves the average return and estimation bias in Humanoid and Ant. The figure also shows that the method that uses only group normalization (+GN) fails in improving the average return and estimation bias.
Layer normalization without variance re-scaling does not work well. Figure III shows that the methods that introduce this layer normalization variant (+LNwoVR and +DO+LNwoVR) do not significantly improve the average return and estimation bias well especially in Humanoid. Further, Figure I shows that the Q-function learning is unstable compared to +DO+GN (and DroQ in Fig- ure 9 in both Ant and Humanoid. This result implies that one of the primal factors for the synergistic effect of dropout and layer normalization is the variance re-scaling.
'Layer normalization normalizes all input elements using their mean and variance. On the other hand, group normalization divides the input elements into subgroups (two groups our setting) and normalizes each group using the intra-group mean and variance. "rroQ without dropout, which is introduced in Sections B.3 and [
22

---

Published as a conference paper at ICLR 2022
Ant-v2
Antov2
Arit-22
6000
+DO+LNwoVR
4000
+00+BN
average bibs
Rd of ales
10.0
V/WWVV
7.5
2000
5.0
2.5
0.0
0.0
6.5
10
15
2,0
2,5
0.0
2.0 environment interactions
0.0
0.5
1.0
2.0
environment Interactions
1e5
environment interact
Humanaid-vv
Humancic v2
10
15.0
12.5
average has
bas
10.0
3000
1000
0.0
0.0
0.5
100
1.5
2.0
2.5
6.0
0.5
1.0
1.5
2.0
2.5
0.0
0.5
1.0
1.5
2.0
2.5
arvironment interactions
les
enviromment interactions le5
environment interact and leb
Figure 10: Average return average/standard deviation of estimation bias for six methods using batch normalization, group normalization, and the variant of layer normalization.
Ar
0.10
0.10
Axt
+D2+BB
3.00
DD0-0M
0.35
0.38
3.00
20
0.00
2.0
25
35
25
0.00
Interactiona
anv/renment Insaractions
0.12
60o
400
0:35
300
0.00
420
035
3110
8.05
0.0
25
20
25
1.0
2.0
25
LG
Intersctione
07NP01EAAT 840 383
wnw.remment Inceraction
Figure 11: Average/variance of the Q-function loss (left part) and its gradient with respect to param- eters (right part).
23

---

Published as a conference paper at ICLR 2022
Algorithm 3 DroQ with ensemble of N dropout Q-functions (DroQN)
Initialize policy parameters 0, N Q-function parameters Φi 1, N, and empty replay buffer D. Set target parameters Ø < Φi. for I...,N.
2: repeat
3: Take action a, ~ (T)-)s;). Observe reward next state Sr41; + DU(st, Fr.
4: for G updates do
5: Sample mini-batch B = {(s, a,, (,,,,")) from D.
6:
Sample set M M distinct indices from {1, 2,., N}.
7:
Compute the Q target y for the dropout Q-functions:
y =
min Dr. (6',d) - ar log =o(('16')) -0loo_((aal's))) iEM
d' ~ TT((16)
8:
for 1= N do
9:
Update Φ; with gradient descent using
Σ
(0, y)²
(s.a,,s"))B
Update target networks with の; pā: (1 (1-p)) p) Update 0 with gradient ascent using
11:
QD. (8, a) (log a~ mo(1s)
V
N
#EB
F RELATION BETWEEN (1) ENSEMBLE SIZE AND (2) THE EFFECT OF LAYER NORMALIZATION AND DROPOUT ON OVERALL PERFORMANCE
In Section + and Appendix D we showed that using layer normalization and dropout improves overall performance. Further, Sections [.] and A. we showed that using multiple (two) dropout Q-functions improves overall performance. These results raise two new questions: (i) what happens we use more than three dropout Q-functions? and (ii) what the effect of layer normalization and dropout would be in such cases? In this section, we conduct additional experiment to answer questions (i) and (ii).
For our additional experiment, we introduce the DroQ with ensemble of N dropout Q-functions (DroQ.N) algorithm (Algorithm R) This algorithm is a variant of DroQ algorithm (Algorithm 2 which uses bootstrapping of a subset M of N dropout O-functions (line 6). This algorithm can also be regarded as a variant of REDQ algorithm (Algorithm M1 which uses N dropout Q-funttions. With DroQN algorithm, the ensemble size be changed by varying the value of N.
We compare the method based on the DroQ algorithm and its ablation variants: D-oNN--NN the DroQ variant that does not use layer normalization for the dropout Q-function QDr.d
DroON-DO-LA the DroQN variant that does not use layer normalization and dropout for QD Note that BroQN-DO-LL is identical with REDQN presented in Appendix H We evaluate the performance of these methods for the case of N 2,3, 5, 10.
Learning curves of the methods are shown in Figure M-
For the question (i), in complex environments (Ant and Humanoid), increasing the size of the en- semble improves the sample efficiency. In Ant, DroQ5-10 are more sample efficient than DroQ2-3. In Humanoid, the sample efficiency monotonically improved when the ensemble size of DroQ increased as N = 2,3,5. 10.
For the question (ii), in the complex environments, the synergy of layer normalization and dropout
is significant especially when the value of N is small (N 2). In Ant and Humanoid, DroQ2- 3 significantly improves the sample efficiency compared to DroQ2-3-DO. On the other hand, the effect of layer normalization itself becomes dominant when the value of N is large (N 5). For
24

---

Published as a conference paper at ICLR 2022
Hopperiv2
Hopper-2
Hopperi22
Drog2
Droq2
Dxoq2
3000
Dreg3 DO LN
300C
Dro03 DO
Dre03
average return
average return
200C
Dro010 DO
2000
Dro010 DO
2000
Dro010
100C
1000
1000
44
0.8
0.6
environment interactions
le5
interactions
le5
environment interactions
Walker22-42
Walker22-42
Walker2d-42
5000
5000
4000
4000
2verage rettmm
pvrrage retumn
4000
1000
3000
2000
2000
1000
1000
0
16
0.5
0.0
1.5
2.5
interactions
leb
interactions
les
Ant-v2
Ant-v2
Ant-v2
6000
6000
6000
5000
ausrae returr
avrrage return
4000
4006
4000
2000
2000
2000
2000
1000
10
2.0
0.5
2.5
2.5
environmmntt interections
le5
environment interactions
lc5
environmenn interactions Humancldwv22
165
Humonoic-v2
Humandidiv22
000C
5000
4000
Sverage return
average return
4000
3000
3000
3000
2000
2000
2000
100C
1000
1000
0
10
2.3
0.5
2.5
2.5
environment interactions
le5
environment interactionss
1c5
environment interactions le5
Figure 12: Additional ablation study result (average return). Left part (dotted lines): DroQN-DO- LN, middle part (dashed lines): DroQN-LN, right part (solid lines): DroQN
Ant and Humanoid, we can see that DroQ5-10-DO significantly improves the sample efficiency compared DroQ5-10-DO-LN, However, DroQ5-10 does not significantly improve the sample efficiency compared to DroQ5-10-DO.
Based on the above result, suggest (1) using layer normalization and dropout together for small ensemble cases, and (2) using layer normalization alone for large ensemble cases.
25

---

Published as a conference paper at ICLR 2022
G EFFECT OF DROPOUT Q-FUNCTIONS ON SAC
Ant-v2
Ant-v2
Ant-V2
600C
DroQ
12.5
SAC+DO
averaae blas
10.0
SAC LN
SAC+DO-LN
sta of
7.5
5.0
2000
2.5
0.0
1.0
2.
2.0 envirement Intoractions Humanaid-vv2
orvironment Interactions
165
environment
Humencid-v2
10
15.0
12.5
average biss
4000
of bibs
3000
2000
1000
2.5
0.0
0.0
0.5
10
1.5 2.0 2.5
0.0
0.5
1.0 155 2.0 2,5
0.0
0.5
1.0 1.5 2.0 2.5
ervironment interections
le5
environment interections les
environment interactions 1es
Figure 13: Average return and average/standard deviation of estimation bias for DroQ, SAC, and its variants. SAC+DO is SAC variant using dropout, SAC+LN is SAC variant us- ing layer normalization, and SAC+DO+LN malization. Note that the only difference between DroQ (M = 2) and SAC+DO+LN the policy update part (line 10 Algorithm D). DroQ (M Va TBT E.EB (1 (i (s,a) a lo (alss) On the other hand, SAC+DO+LN updates policy as Vo TTT EvEB (3 ming (8,,,(s,00 a log no(als)).
a SAC variant using dropout and layer nor-
2) updates policy as
26

---

Published as a conference paper at ICLR 2022
H HYPERPARAMETER SETTINGS
The hyperparameter settings for each method in the experiments discussed Section E are listed in Table &. Parameter values, except for (i) dropout rate for DroQ and DUVN and (ii) M for DUVN, were set according to Chen all (2021h). The dropout rate (i) was set through line search, and M for DUVN (ii) was according Harrigan (2016);; Moerland et (2017)).
Table 8: Hyperparameter settings
Method
Parameter
Value
SAC, REDQ, DroQ, DUVN
optimizer
Adam (Kingma Ba, 2015)
learning rate
3 10a
discount rate (r)
target- smoot--m coefficient (p) replay buffer size number hidden layers for all networks
0.005
106
number of hidden units per layer
256
mini-batek size
256
random starting data
5000
UTD ratio G
20
REDQ and DroQ
i-target miniizzttion parameter M
REDQ
ensemble size N
10
DroQ and DUVN
dropout rate
0.01
DUVN
in-tar minimization parameter M
I EXPERIMENTS ON THE ORIGINAL REDQ CODEBASE
Experiments presented in the main content of this paper have been implemented top of the SAC codebase ftttps://q1thub..o cww http://....oooee//e//aaa--trr-r-
pytooo- In this section, report the results of our experiments on the original REDQ codebase (nttps: //qithub. com/ wat charnyu/ (REC))). Our experiments are to replicate experiments in the main content of this paper. To evaluate the computational efficiency, we plot the wallclock time required to complete certain number of interactions with the environment, instead of the standard deviation of bias. In addition, dropout rate values are re-tuned for each environment (Table H) Fig- ure [4 shows the replica of the results presented in Section Iill Figure [15 shows the replica of the results provided
Section A.II Figure I shows the replica of the results provided in Section DI Figures IX and [T show the replica of the results provided in Section Figure X shows the replica the results provided in Section
Section H. Figure IG shows the replica of the results provided
Table 9: Dropout rate setting for Appendix
Environment
Value
Hopper-v2
0.0001
wa
0,005
Ant-v2
0.01
Humanoid-v2
0.1
27

---

Published as a conference paper at ICLR 2022
Hopperyv2
Hopper
Hopper-v2
400C
SAC REDO
1000
average bias
Droo
2000
Time
20000
10000
...c
0.50
9.00
0.50
0.75
1.00
ervironment interactions
environment interactions Walloor2d.vk
environment Interactions
Wilker2d-42
10
Walker26-v2
6000
5000
00000
average biass
400C
80000
3000
Tire
40000
2000
1000
0.0
0.5
1.0
2.5
2.0
255
0.0
0.5
1.0
1.5
2.0
2.5
0.0
0.5
1.0
1.5
2.0
2,5
envirromment interectiones
le5
environment interactions
les
envirroment interections
1e5
Ant-v2
Ant v2
10
Ant-v2
6000
80000
auerage has
4000
60000
Thme
2000
40000
20000
-2
0.0
2.0
interactions
leb
interactions Humanaid v2
le5
envlronment Interactions
Humaneió-v2
Humanold-v2
5000
80000
average bias
4000
60000
3000
40000
2000
1000
0
0.0
0.5
1.0
2.0
2.5
2.0
0.0
0.5 100 1.5
2.0 2.5
environmmnt interections
environment interactions
envvrromeent internetiiiiiiiin interections
Figure 14: Average return, average of estimation bias, and wallclock time (second) for REDQ. SAC, DUVN, and DroQ. The horizontal axis represents the number of interactions with the environment (c.g., the number of exccutions of line of Algorithm D) For each method, average score of five independent trials are plotted as solid lines, and standard deviation across trials is plotted as trans- parent shaded region.
28

---

Published as a conference paper at ICLR 2022
Hoppervv2
Hosper-22
10
4000
DrOO
15000
200C
AN
average bias
-OCHUM
200C
Time
1000
5000
0.00
0.25
0.50
0.75
1.00
8.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
environment intercctions
environment interections
envlronmart Interactions
1eb
Welker2d-v2
Walker2d-v2
10
Walker2d-v2
6000
40000
5000
aussage has
4000
30000
20000
2000
10000
0.0
0.5
1.0
2.
2.5
0.0
0.5
1.0
1.5
2.
2.5
1.5
anvironment interactions
le5
environment Interactions
le5
environment interections
le5
Ant-v?
Artt-2
6000
40000
average bins
4000
30000
2000
20000
10000
0.0
0.0
100
1.5
environment interactions
1e5
environmant Interactions
Humanoid-v2
Humancic-v2
10
Humanold-v2
5000
40000
average biat
1000
30000
2000
20000
2000
1000
0.0
0.5
1.0
2.0
2.5
0.0
0.5
1.0
1.5
2.0
2.5
0.0
0.5
1.0
1.5
2,0
2.5
anvironment interactions
le5
enviromment interactions
leb
environment inteeeeeeetttteeteeeee interections 1e5
Figure 15: Ablation study results
29

---

Published as a conference paper at ICLR 2022
Hopper-v2
Hopper v2
10
Hopper-v2
200C
0.1
15000
0.01
average has
2000
0,0011
Time
10000
1000
-..o
0.00
0,50
1,00
0.25
0.50
0.75
1.00
0.00
0.50
1.00
enviromment interactions
le5
environment interactions
le5
environment interactions le5
222
Walleer2d-2
Walker2d-v2
6000
40000
5000
average bins
4000
2000
Timre
20000
10000
1000
0.0
0.0
100
ervironment interactions
environmert Interactions
Ant-V2
Ant v2
10
6000
40000
average bist
4000
30000
Tire
20000
2000
10000
0.0
0.5
1.0
2.5
2.
255
0.0
0.5
1.0
1.5
2.0
2.5
0.0
0.5
1.0
1.5
2.0
2,5
environment interactionis
le5
environment interactions
le5
envirrommnt interactions
le5
Humanaid v2
Humanoicw42
10
Humanoid-v22
500C
40000
average bibs
400C
30000
300C
20000
2000
100C
10000
0.0
0.5
1.0
15
2.0
2.5
environment
environment interactions le5
envlronment Interactions
Figure 16: Average return, average of estimation bias, and wallclock time (second) for DroQ with different dropout rates. Scores for DroQ are plotted solid lines and labelled in accordance with dropout rates (e.g."0.22 "0.2").
30

---

Published as a conference paper at ICLR 2022
320
Di0
Art-2
0.28
Av
Antov2
80
0.33
3.06
0.38
0
25
1.0
2.5
2,5
60
10
1.
25
-030
0.00
14
Gnvrormatt intelcclooe
Cherormztt intorccloro
andicoment InSeRATIINS
365
aesimement inteection
820
Humana divz
300
humano drv
0.5
9:50
Humarste22
of grodeet
0.20
420
0.33
Tagh
300
100
1.
15
00
03
2.0
35
20
23
1.5 inpeactionn
Interatiion
305
Figure 17: Average/standard deviation of the Q-function loss ("average Q loss" and
"std of Q loss") and its gradient with respect to parameters ("average Q gradient" and
"std of Q gradient"). Q-function loss is caleulated as M Σ Li, where I
(Qor (s,a) a) - r (minge Qo, (s',a))== log =o(a's')))) and its gradient is calculated as
M7 Σ 1= 00.
OL.
v2
10
5000
4000
+DO+CN
+-00485
everage bias
averoge
2000
2200
0.0
1.5 enviranment Interactions Humanold.vv
environment interections
1e5
Hurnno d-u2
10
5000
everage return
4000
average bias
3000
1000
2..
0.0
0.5
15
20
2.5
1.5
environment interactions
1es
environment interactions
Figure 18: Average return and average estimation bias for six methods using batch normalization, group normalization, and the variant of layer normalization.
31

---

Published as a conference paper at ICLR 2022
320
+DO+LNNw
Artin2
Artw2
0.30
0.30
Antrv2
3.00
DD 201
0.35
40
2worgee
0.33
0.04
20
1.23
25
23
13
-0.00
35
30
35
24
25
1,4
23
165
Cierorment intorccaboo
800
Humano
d92
0.38
Humonaid
4c0
0.35
3.2
.000
120
0.33
3.15
0:30
0.65
0.
2.5
2.5
2.5
6.0
1.0
15
25
0000
0.5
1.0
15
2.0
2.5
3.0
1,4
2.5
UIV InnrrOmeT Interactions
enverommnnt aecioctoooo
3r3
snvironment rectorecooo
Figure 19: Average/variance of the Q-function loss (left part) and its gradient with respect to param- eters (right part).
Hopper-42
Hoppers22
Hosper-22
400C
Dre02-DO-LE
Drea2-00
Dro02
Dro03
3000
CroQs DC LN
zvvrnge renurr
200C
Croq5 DC
DvTragg return
DreQ5
Dro010:DO-IN
Dre010-00
Dro010
2000
200C
100C
100C
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0,50
0.75
1.00
environment interections
1e5
environmentt interections
le5
environment interections
Walker2div2
Walker2div2
Walker2div2
6000
5000
5000
4000
aursage reture
4000
awsrgge rohure
4000
3000
2000
2000
2000
2000
2000
1000
1000
0.0
0.5
1.0
2.5
2.0
2.5
0.0
0.5
1.0
2.5
2.0
2.5
0.0
0.5
1.0
2.5
2.0
2.5
environmeet interections
165
environmment inter interectionss
le5
environmmnnt interactions
Ant.
CO0G
6000
average return
awerage ehturn petrr
4000
4000
4000
2000
2000
0
2.5
1.5
2.5
2.5
2.0
ervironment interactions
les
environment interactions
le5
environment interactions
Humencid v2
Humancic
Humancid
600C
5000
500C
5000
4000
average return
4000
average rsturn
4000
3002
3000
2000
200C
1000
1000
2.5
0.5
1.5
2.5
2.0 environmantt interactions
ervironment interactions
leb
environment Interactions
leb
Figure 20: Additional ablation study result (average return). Left part (dotted lines): DroQN-DO- LN, middle part (dashed lines): DroQN-LN, right part (solid lines): DroQN
32

---

Published as a conference paper at ICLR 2022
J OUR SOURCE CODE
Our
source
code
available at
Dropout Q Functions for Doubly ReiieeennncPaattorrreeereeeaaaae
33


Reviews:
[{"review_id": "uzmfmfAydXk", "reviewer": "Reviewer_RMuR", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n\nThis paper is well-written and easy to understand. The authors present a compelling reason to use ensemble-based Q-learning and why ensembles can be harder to work with from a computational perspective. I thinkt he obser The results and analysis support the hypothesis that dropout and layer normalization can have the same effect as a larger ensemble. \n\n\nWeaknesses: \n\nI believe novelty is a major weakness for this approach. Many of the individual components presented in this paper (dropout for Q functions, ensembles, etc) have been discussed in previous works. The authors claim that these have not been studied in high UTD settings, however the authors do not present any evidence that there is a difference between how REDQ and their method work at high UTD or lower UTD. Such a set of experiments will really help make this paper's claim stronger. Additionally, since the authors use M=2, this approach is very similar to a vanilla SAC implementation. It would be good to see an analysis of SAC with the dropout architecture (essentially this method but without any of the improvements from REDQ). ", "summary_of_the_paper": "This paper attempts to address computational efficiency in Ensemble-based Q-learning. The authors claim that previous, REDQ is too inefficient computationally as it requires too many Q-networsk in the ensemble. Thus, they propose an alternative approach that employs dropout (and layer normalization to stabilize the dropout layers), in order to get the same benefit of a larger ensemble in uncertainty estimation, while keeping computational costs low. The results and analysis on standard continuous control benchmark tasks show that the proposed approach maintains all the benefits of REDQ without the computational costs. ", "summary_of_the_review": "The paper presents a useful and interesting improvement to REDQ but novelty claims are not backed by experiments. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Z9JhgIJNzSb", "reviewer": "Reviewer_1PuF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "======Strengths=======\n\n- Simplicity: the proposed method requires minimal modification to existing methods, making it easy to implement and use\n- Significance: REDQ has achieved very strong sample efficiency but the ensemble of Q network leads to slower computation. How to achieve the same sample efficiency while reducing computation seems to be a very important question to solve. \n- Analysis: comparison on computation and memory is good, shows significant reduction in computation and memory usage\n- Results: results is nice, showing that the proposed method can achieve same sample efficiency of REDQ while being much more computation and memory efficient.\n\n======Weaknesses=======\n\n- Clarity of writing: some of the writing is not very accurate or not clear and can be refined\n- Lack of in-depth investigation on why dropout and layernorm alone do not work well, and why dropout+layerrnorm works well, lacks comparison to other regularization techniques\n- A number of other issues in the paper\n- Proposed method lacks novelty: the proposed method (dropout+layernorm) seems to be a bit incremental. \n\nConcerns, comments and questions:\n- In page 1, \"the UTD ratio is defined as the number of Q-function updates...\", in the REDQ paper, this is actually defined as \"the number of updates taken by the agent compared to the number of actual interactions with the environment\", which is slightly different from your definition here (only the Q update), although for the main results reported in the REDQ paper, they focus on only changing the Q update number to isolate the effect of the policy update, you might want to be more clear as to prevent confusion. In your case, you can further clarify your writing, perhaps be more explicit and say its general definition and then say in your paper you define it to be only focused on Q update. \n- Authors should check the writing again for small typos and mistakes. For example, should be MuJoCo instead of Mujoco. Figure 2 typo \"and average and\"\n- top of page 2, \"REDQ runs more than 2 times slower than SAC\", you might want to further clarify here (and in other places in the paper where you mention this), are you talking about the computation time per update (where REDQ is a bit slower due to ensemble)? Or per environment interaction collected (where REDQ is a lot slower due to ensemble and high UTD)? Or the computation time for running until a performance level (where REDQ is not that slow due to reduced sample usage, I feel like this one is what you are talking about)? It is very important to make this point crystal clear since a majority of your discussion is on this subject. And helping the reader to gain a clear idea of exactly how much slower REDQ can be would help readers see the significance of your contribution.  \n- What is the full name of DUVN? Is this sth proposed in previous work? It is a bit unclear how this variant relates to your proposed method. Might want to explain it in a bit more detail. \n- page 6 definition of the error (bias), the current definition is a little weird, it might be better to define the error as the difference of Q values, and then say we can estimate this error with MC return, instead of defining Q as the MC return. (I believe typically Q value is defined as expectation of MC return, not the other way around)\n- end of page 6 \"the most memory-intensive process... (applying the ReLU layer)\" are you saying relu activation takes a lot of memory? \n- Figure 2 hopper result seems to be lower than reported in the REDQ paper, not sure why? \n- Ablation shows that layer norm helps with performance, but in the paper there is very limited discussion and analysis on why exactly layer norm makes dropout work better. What is so special about layer norm? Previously dropout had mixed results in DRL, why layer norm can fix the problems? What about other regularization schemes? Adding comparison to other schemes, and more analysis on how layer norm and dropout work together, either theoretically or empirically can help improve the significance of this paper. Otherwise the layernorm seems to be just a random hack added to boost performance. \n- Though the name of Dr.Q is technically different from DrQ, I'm still a bit concerned whether this will lead to confusion \n", "summary_of_the_paper": "Authors propose to use dropout and layer norm to make Randomized ensembled double Q-learning more computationally efficient. The proposed algorithm uses simple modifications to achieve comparable sample efficiency while being faster in computation. Authors also conduct empirical analysis on how dropout can be used effectively. ", "summary_of_the_review": "I think the paper has potential to make some good contribution, the authors already obtained some good results showing that Dr.Q can achieve the same performance as REDQ while greatly reduce computation and memory consumption, which is very nice. However, what is lack is some more in-depth investigation on why the proposed method works. Why dropout on its own does not work? And why layer norm will deal with the difficulties? What are the unique advantage of this combination over other methodologies? Very little discussion and empirical/theoretical analysis is provided in the paper. I think if the authors try explore these questions more, they should be able to make a bigger contribution and make this paper much stronger. \n\nThough the paper studies an important problem, and has some good results, due to this major concern, and considering other issues in the paper, I don't quite think the paper is ready for publication in its current state, thus I currently lean towards rejection. \n\n==============================================================\n\nPost-rebuttal: after reading the authors' response and the revision, I feel that a large number of my concerns are addressed and the new results ablation and analysis look quite interesting. Though I still have some concerns, I now increase my score to 6. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1HyHtweyyv", "reviewer": "Reviewer_fiFc", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nThis was a very interesting paper. Contrary to the ensemble approach for bias reduction which is rapidly gaining popularity in recent years, the authors proposed an alternative which allows an SAC-like algorithm to work in high UTD settings by adding dropout and layernorm, which in my opinion is the main contribution of this paper. The presentation of the paper is quite clear and it was very easy to read. Experiments are quite thorough and very well done. The ablation studies are also quite clear in presenting how the different components of the algorithm affect the performance of the algorithm.\n\nWeakness:\n\nMy main concern is that it is not entirely clear from the paper why the method presented by the authors work so well. Dropout, layernorm, and high UTD have all be separately applied to and studied in off-policy algorithms and it is not clear why combining all three leads to such a boost in performance. Intuitively dropout has similar effects as ensemble models since they add noise to the Q networks thereby injecting uncertainty into the target, but based on the authors' ablation studies and previous work, having dropout alone is not sufficient. More analysis on this front would do much to strengthen the authors' overall arguments.\n\nOne final minor point, I suggest modifying the name Dr. Q, I don't think adding the period after Dr will help with avoiding name conflicts with DrQ as the authors have suggested.", "summary_of_the_paper": "This work proposes a new algorithm based on REDQ. The new algorithm Dr. Q. uses a small ensemble of dropout Q functions along with layer normalization. The proposed algorithm achieves similar sample efficiency as REDQ but better computational efficiency. ", "summary_of_the_review": "Though the paper could benefit from additional analyze on explaining the performance of the proposed algorithm, I think this paper brings about some valuable insights which may benefit and impact how off-policy algorithms are implemented in the long-run.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The manuscript describes a method for improving the computational efficiency of randomized ensemble double Q-learning for continuous action RL, by using a small ensemble of Q-functions equipped with dropout and layer normalization, achieving matched sample efficiency at considerably less computational cost.

Reviewers praised the method's simplicity and achievement of its stated objective of reducing the computational cost of deploying ensemble Q functions. In general, the paper was found to be easy to understand and well written. Several expressed concern about the lack of interrogation of why this combination of dropout and layer norm worked so well and an overall lack of novelty. Other miscellaneous criticisms were well-addressed in rebuttal and extensive new analyses in the Appendix were noted by several reviewers as adding much to the work.

In the AC's opinion, this is an example of a simple but non-obvious combination of well-known ideas that works very well. The review process has improved the level of empirical rigor that has gone into understanding the properties and trade-offs of this method. I'm happy  to recommend acceptance, though would echo reviewers concerns that dubbing the method "Dr.Q" will lead to confusion and would strongly urge adopting another name for the camera ready.
Decision: Accept (Poster)