CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicity
Authors: Aditya Bhatt
Daniel Palenicek
Boris Belousov
Max Argus
Artemij Amiranashvili
Thomas Brox
Jan Peters
Venue: ICLR 2024 spotlight (2024)
Keywords: Deep Reinforcement Learning
Primary area: reinforcement learning

Abstract: Sample efficiency is a crucial problem in deep reinforcement learning. Recent algorithms, such as REDQ and DroQ, found a way to improve the sample efficiency by increasing the update-to-data (UTD) ratio to 20 gradient update steps on the critic per environment sample.
However, this comes at the expense of a greatly increased computational cost. To reduce this computational burden, we introduce CrossQ:
A lightweight algorithm for continuous control tasks that makes careful use of Batch Normalization and removes target networks to surpass the current state-of-the-art in sample efficiency while maintaining a low UTD ratio of 1. Notably, CrossQ does not rely on advanced bias-reduction schemes used in current methods. CrossQ's contributions are threefold: (1) it matches or surpasses current state-of-the-art methods in terms of sample efficiency, (2) it substantially reduces the computational cost compared to REDQ and DroQ, (3) it is easy to implement, requiring just a few lines of code on top of SAC.

Published as a conference paper at ICLR 2024
CROSSQ: BATCH NORMALIZATION IN DEEP REINFORCEMENT LEARNING FOR GREATER SAMPLE EFFICIENCY AND SIMPLICITY
Aditya Bhatt 1.4
Daniel Palenicek * 1.2
Boris Belousov 1.4
Max Argus
Artemij Amiranashvili
Thomas Brox
Jan Peters 1,2,4.5
"Equal contribution
'Itelligent Autonomous Systems, TU Darmstadt 'Hessian.AT University of Freihurg *German Research Center for Al (DFKI) "Centre for Cognitive Science TU Darmstadt
bbhtty..hatttefkkk.co
ABSTRACT
Sample efficiency is crucial problem in deep reinforcement learning. Recent al- gorithms, such as REDQ and DroQ, found a way to improve the sample efficiency by increasing the update-to-data (UTD) ratio 20 gradient update steps on the critic per environment sample. However, this comes at the expense of a greatly increased computational cost. To reduce this computational burden, we introduce CrossQ: A lightweight algorithm for continuous control tasks that makes careful use of Batch Normalization and removes target networks to surpass the current state-of-the-arr in sample efficiency while maintaining low UTD ratio of 1. No- tably, CrossQ does not rely on advanced bias-reduction schemes used current methods. CrossQ'ss s contributions are threefold: (1) it matches surpasses current state-of-the-an of-the-art methods in terms sample efficiency, (2) it substantially reduces the computational cost compared REDQ and DroQ, (3) it is easy to implement, requiring just a few lines of code on top of SAC.
INTRODUCTION
Sample Efficiency
Sample efficiency is a crucial concern when applying Deep Reinforcement Learning (Deep RL) methods on real physical systems. One of the first successful ap- plications of Deep RL to challenging problem of quadruped locomotion was achieved using Soft Actor- Critic (SAC, Haarnoja et al. (201 8a)), allowing a robot dog to learn to walk within 2h of experience (Haarnoja et al., 2018b). Subsequently, it was noted that the critic in SAC may be underfitted, as only a single gra- dient update step the network parameters is per- formed for each environment step. Therefore, Ran-
1QM Nort Return
1.5
1.0
0.5
0.0
5
Env Steps (1M)
Computational Efficiency
IQM Norm Retur
1.5
domized Ensembled Double Q-Learning (REDQ, Chen et al. (2021)) was proposed, which increased this num- ber of gradient steps. termed update-to-data (UTD) ratio. In addition, Dropout Q functions (DroQ, Hi- raoka et al. (2021)) improved the computational ef- ficiency of REDQ while maintaining the same sam- ple efficiency by replacing its ensemble of critics with dropout. This enabled learning quadruped locomotion in mere 20min (Smith et al., 2022). Thus, REDQ and DroQ represent the -fate-of-the-art in terms of sample efficiency in Deep RL for continuous control.
1.0
0.5
0.0
15
Gradieet Steps (1M)
CrossQ
DroQ
REDQ
Figure 1: CrossQ training performance aggregated over environments. CrossQ is more sample efficient (top) while be- ing significantly more computationally ef- ficient (bottom) in terms of the gradient steps, thanks to a low UTD = 1. Follow- ing Agarwal et al. (2021), performance by the maximum of REDQ in each environment.
Importantly, both REDQ and DroQ showed that naively increasing the UTD ratio of SAC does not perform well due to the critic networks' value estimation bias. Therefore, ensembling techniques were introduced for bias reduction (explicit ensemble in REDQ and implicit
normalize

---

Published as a conference paper at ICLR 2024
ensemble via dropout in DroQ), which allowed increasing the UTD to 20 critic updates per envi- ronment step. Higher UTD ratios improve sample efficiency by paying the price of increased com- putational cost, which manifests in higher wallclock time and energy consumption. It is, therefore, desirable to seek alternative methods that achieve the same or better sample efficiency at a lower computational cost, e.g., by using lower UTDs.
turns out that even UTD = can perform surprisingly well if other algorithmic components are adjusted appropriately. In this paper, we introduce CrossQ. a lightweight algorithm that achieves superior performance by removing much of the algorithmic design complexity that was added over methods. First, removes target networks,
the years, culminating in the current state-of-the an ingredient widely believed to slow down training in exchange for stability (Mnih et al., 2015; Lillicrap et al., 2016; Kim et al., 2019; Fan et al., 2020). Second, we find that Batch Normalization variants (loffe & Szegedy (2015); loffe (2017)), when applied in a particular manner, effectively stabilize training and significantly improve sample efficiency. This contradicts others' observations that it hurts the learning performance in Deep RL, e.g. Hiraoka et al. (2021). Third, CrossQ uses wider critic layers, motivated by prior research on the ease of optimization of wider networks (Ota et al., 2021). In addition to the first two improvements, wider networks enable even higher returns.
Contributions. (1) We present the CrossQ algorithm, which matches or surpasses the current state-of-the-art for model-free off-policy RL for continuous control environments with state ob- servations in sample efficiency while being multiple times more computationally efficient; (2) By removing target networks, we are able to successfully accelerate off off-policy Deep RL with Batch- Norm: (3) We provide empirical investigations and hypotheses for CrossQ'ss success. CrossQ'ss changes mainly pertain to the deep network architecture of SAC; therefore, our study is chiefly em- pirical: through a series of ablations, we isolate and study the contributions of each part. We find that CrossQ matches or surpasses the state-of-theearl algorithms in sample efficiency while being up to 4x faster in terms wallclock time without requiring critic ensembles, target networks, or high UTD ratios. We provide the CrossQ source code at giitub.com//ddiyy//Cross
2 BACKGROUND
2.1 OFF-POLICY REINFORCEMENT LEARNING AND SOFT ACTOR-CRITIC
We consider a discrete-time Markov Decision Process (MDP, Puterman (2014)), defined by the tuple (S, A. P, R. p.7) with state space S, action space A, transition probability 8++1 Pl(-s,,, a,), reward function rz R(s,, ar), initial state distribution 8o ~ and discount factor 7 E [0, 1). RL describes the problem of an agent learning an optimal policy π for given MDP. At each time step t, the agent receives a state and interacts with the environment according to its policy т. We focus on the Maximum Entropy RL setting (Ziebart et al., 2008), where the agent's objective is to find the optimal policy n², which maximizes the expected cumulative reward while keeping the entropy H high; arg maxxa Exgep (Ei (r - aH(TT( [s:)))] The action-value function defined by Q(s, a) Ex,P EE 7 (rr a log (a,) s;) 80 8, ag = a] and describes the expected reward when taking action a in state s. Soft Actor- (SAC, (Haarnoja et al., 2018 popular algorithm that solves the MaxEnt RL problem. SAC parametrizes the Q function and policy neural networks and trains two independent versions of the Q function, using the minimum of their estimates to compute the regression targets for Temporal Difference (TD) learning. This clipped double-Q trick, originally proposed by Fujimoto et al. (2018) in TD3, helps in reducing the potentially destabilizing overestimation bias inherent in approximate Q-learning (Hasselt, 2010).
2.2 HIGH UPDATE-TO-DATA RATIOS, REDQ, AND DROQ
Despite its popularity among practitioners and as foundation for other more complex algorithms, SAC leaves much room for improvement in terms sample efficiency. Notably, SAC performs exactly one gradient-based optimization step per environment interaction. SAC's UTD- = 1 setting is analogous to simply training for fewer epochs in supervised learning. Therefore, in recent years, gains in sample efficiency within RL have been achieved through increasing the UTD ratio (Janner et al., 2019; Chen et al., 2021; Hiraoka et al., 2021; Nikishin et al., 2022). Different algorithms, however, substantially vary in their approaches to achieving high UTD ratios. Janner al. (2019)
2

---

Published as a conference paper at ICLR 2024
def critic_less (Q params, policy_params, obs, acts, rews, next_obs] next_acts, next_logpi policy. apply paramsy_params, next_obs)
Concatenated forward pass
all al1_g, new new___params - Q.apply (Q_params,
concatenteee (lops, next_cbst eext_cbsl), jup.concatenate (aats, next_acts)
Split all_q predictions and stop gradient on next_q 9, next_q jtp. split (all_q, 2)
next_q inp min (next_e, axis=0)
min over double function next_q q jax 1ax. 1aw.1lxx.ooopo. http:ssttppgdidaae (next_q alpha next_ return jnp. mean (q (rews gamma * next_q)) .*2), new new_a_params
12
13
Figure 2: CrossQ critic loss in JAX. The CrossQ critic loss is easy to implement on top of an existing SAC implementation. One just adds the batch normalization layers into the critic network and removes the target network. As we are now left with only the critic network, one can simply concatenate observations and next observations, as well as actions and next actions along the batch dimension, perform a joint forward pass, and split up the batches afterward. Combining two forward passes into one grants a small speed-up thanks to requiring only one CUDA call instead two.
model to generate synthetic data, which allows for more overall gradient steps. Nikishin et al. (2022) adopt a simpler approach: they increase the number of gradient steps while periodically resetting the policy and critic networks to fight premature convergence to local minima. We now briefly outline the two high-UTI methods to which we compare CrossQ.
Chen et al. (2021) find that merely raising SAC's UTD ratio hurts performance. They attribute this to the accumulation of the learned Q functions' estimation bias over multiple update steps-despite the clipped double-Q trick-which destabilizes learning. To remedy this bias more strongly, they increase the number of networks from to ensemble of 10. Their method, called REDQ, permits stable training at high UTD ratios up to 20.
REDQ.
DroQ. Hiraoka et al. (2021) note that REDQ's ' ensemble size, along with its high UTD ratio, makes training computationally expensive. They instead propose using smaller ensemble of Q functions equipped with Dropout (Srivastava et al., 2014), along with Layer Normalization (Ba et al., 2016) stabilize training in response the noise introduced by Dropout. Called DroQ, their method is computationally cheaper than REDQ, yet still expensive due to its UTD ratio of 20.
3 THE CROSSQ ALGORITHM
In this paper, we challenge this current trend of high UTD ratios and demonstrate that we can achieve competitive sample efficiency much lower computational cost with UTD method. CrossQ our new state-of-the-ar off-policy actor-critic algorithm. Based on SAC, uses purely network- architectural engineering insights from deep learning to accelerate training. As result, it crosses ont much of the algorithmic design complexity that was added over the years and which led to the current state-of-re of-the-art methods. In doing so, present much simpler yet more efficient algorithm. In the following paragraphs, we introduce the three design choices that constitute CrossQ.
3.1 DESIGN CHOICE 1: REMOVING TARGET NETWORKS
Mnih al. (2015) originally introduced target networks stabilize the training of value-based off- policy RL methods, and today, most algorithms require them (Lillicrap et al., 2016; Fujimoto et al., 2018; Haarnoja al., (0). SAC updates the critics' target networks with Polyak Averaging
0 < -0 + +
where 0 are the target network parameters, and 0 are those of the trained critic. Here 7 is the target network smoothing coefficient; with a high т = (equivalent to cutting out the target net- work), SAC training diverge, leading to explosive growth in 0 and the Q predictions. Target networks stabilize training by explicitly delaying value function updates, arguably slowing down online learning (Plappert al., 2018; Kim et al., 2019; Morales, 2020).
3

---

Published as a conference paper at ICLR 2024
SAC:
CrossQ (Ours):
Qo(Si; A.) = qt.
S
A
1)
qt
Qe
Qo (S++1; As+1) =qi+1
S1+1
A
a++1
Cg = (q: - T - -7
La = (q: - rt 190++llaa) log))
Figure 3: SAC without BatchNorm in the critic Qa (left) requires target Q values lit1 stabilize learning. CrossQ with BatchNorm in the critic Qe (right) removes the need for target networks and allows for joint forward pass of both current and future values. Batches are sampled from the replay buffer B: St, A,, re, S4+1 ~ B and Ac+1 ~ (T((++1) from the current policy. lag denotes the s -ooo--rradi ent
ent operation.
Recently, Yang et al. (2021) found that critics with Random Fourier Features can be trained without target networks, suggesting that the choice of layer activations affects the stability of training. Our experiments in Section 4.4 uncover an even simpler possibility: using bounded activation functions or feature normalizers is sufficient to prevent critic divergence in the absence of target networks, whereas the common choice of relu without normalization diverges. While others have used normalizers Deep RI. before, we are the first to identify that they make target networks redundant. Our next design choice exploits this insight to obtain an even greater boost.
Action Demsities
3.2 DESIGN CHOICE 2: USING BATCH NORMALIZATION
BatchNorm has not yet seen wide adoption in value-based off- policy RL methods, despite its success and widespread use in su- pervised learning (He et al., 2016; Santurkar et al., 2018), attempts at doing so have fared poorly. Lillicrap et al. (2016) use BatchNorm layers on the state-only representation layers in the DDPG critic but find that it does not help significantly. Others use BatchNorm in de- coupled feature extractors for Deep RL networks (Ota et al., 2020; 2021), 1), but not in critic networks. Hiraoka et al. (2021) report that using BatchNorm in critics causes training to fail in DroQ.
04
We find using BatchNorm carefully, when additionally removing target networks, performs surprisingly well, trains stably, and is, in fact, algorithmically simpler than current methods.
Figure 4: Replay buffer and current policy actions are distributed differently. Darker colors denote higher density. Estimated from batch of 10 transitions (a, s') ~ B; a' ~ TT(ss), after 3 x 10 training steps on Walker2d; 04 and as are random action dimensions.
First, we explain why BatchNorm needs to be used carefully. Within the critic loss [Qo(S, - (r + 7Q0= (S', A'))], predic- tions are made for two differently distributed batches of state-action pairs; (S, A) and (S', A'), where A' ~ (S') is sampled from the current policy, while A originates from old behavior policies.
Just like the target network, the BatchNorm parameters are updated by Polyak Averaging from the live network (Equation 1). The BatchNorm running statisties of the live network, which were estimated from batches of (s, a) pairs, will clearly not have seen samples (s', (s()) and will further not match their statistics. In other words, the state-action inputs evaluated by the target network will be running statistics. is well known that the prediction quality of BatchNorm Naueuhoor- networks degrades in the face of such test-time distribution shifts (Pham et al., 2022; Lim et al., 2023).
given its mismatched BatchNorm
Removing the target network provides an elegant solution. With the target network removed, we can concatenate both batches and feed them through the Q network in a single forward pass, as illustrated Figure 3 and shown in code in Figure 2. This simple trick ensures that BatchNorm's normalization moments arise from the union of both batches, corresponding to 50/50 mixture of their respective distributions. Such normalization layers do not perceive the (s()) batch as being out-of-distribution This small change to SAC allows the safe use of BatchNorm and greatly accelerates training. We are not the only ones to identify this way of using BatchNorm to tackle the distribution mismatch; other works in supervised learning, e.g., Test-Time Adaptation (Lim al.,
4

---

Published as a conference paper at ICLR 2024
Humanoid
HumanoddStandup
Walker2d
IQM 1QM Return (1k)
400
8
300
200
2
Ant
HalfCheetah
Hopper
IQM Return (1k)
16
4
12
0
2
3
4
5
Env Steps (IM)
Env Steps (1M)
Env Steps (1M)
CrossQ (ours)
CrossQ (TD3)
DroQ
REDQ
TD3
SAC
Figure CrossQ sample efficiency. Compared to REDQ and DroQ (UTD = 20) CrossQ
(UTD 1) performs either comparably, better, or-for the more challenging Humanoid tasks- substantially better. These results directly transfer to TD3 as the base algorithm in CrossQ (TD3). We plot interquartile mean (IQM) and 70% quantile interval of the episodic returns 10 seeds.
2023), EvalNorm (Singh Shrivastava, 2019), and Four Things Everyone Should Know to Improve Batch Normalization (Summers & Dinneen, 2020) also use mixed moments to bridge this gap.
In practice CrossQ's 's actor and critic networks use Batch Renormalization (BRN, Ioffe (2017)), an improved version of the original BN (Ioffe & Szegedy, 2015) that is robust to long-term training instabilities originating from minibatch noise. BRN performs batch normalization using the less noisy running statistics after warm-up period, instead of noisy minibatch estimates in BN. In the rest of this paper, all discussions with "BatchNorm" apply equally to both versions unless explicitly disambiguated by BN BRN.
3.3 DESIGN CHOICE 3: WIDER CRITIC NETWORKS
Following Ota et al. (2021), we find that wider critic network layers in CrossQ lead to even faster learning As we show in our ablations in Section 4.4, most performance gains originate from the first two design choices; however, wider critic networks further boost the performance, helping to match or outperform REDQ and DroQ sample efficiency.
We want to stress again that CrossQ, a UTD = 1 method, does not use bias-reducing ensembles, high UTD ratios or target networks. Despite this, achieves its competitive sample efficiency at fraction of the compute cost of REDQ and DroQ (see Figures 5 and 6). Note that our proposed changes can just as well be combined with other off- oefipolicy TD-leaming methods, such as TD3, as shown in our experiments in Section 4.1.
4 EXPERIMENTS AND ANALYSIS
We conduct experiments to provide empirical evidence for CrossQ's performance, and investigate:
1. Sample efficiency of CrossQQ compared to REDQ and DroQ:
2. Computational efficiency in terms of wallclock time and performed gradient step;
3. Effects of the proposed design choices on the performance via Q function bias evaluations;
And conduct further ablation studies for the above design choices. We evaluate across a wide range of continuous-control MuJoCo (Todorov al., 2012) environments, with 10 random seeds each. Following Janner et al. (2019); Chen et al. (2021) and Hiraoka et al. (2021), we evaluate on the same four Hopper, Walker2d. Ant, and Humanoid d tasks, as well as two additional tasks: Hal HalfCheetta ah and the more challenging Humanoi dStiddsanneee andup from Gymnasium (Towers et al., 2023). We adapted the JAX version of stablehhaeeline (Raffin et al., 2021) for experiments.
5

---

Published as a conference paper at ICLR 2024
Humanoid
HumanoidStande
Wa Walker2d
IQM Return (1k)
400
8
300
200
100
2
8
Ant.
Ha SallfCheetal
Hopper
1QM 1QM Return (1k)
16
12
3
2
2
2
Wallclock [hours]
Wallclock [hours]
Wallclock [hours]
CrossQ (ours)
SAC
DroQ
REDQ
Figure 6: Computational efficiency. CrossQ trains an order of magnitude faster, taking only 5% of the gradient steps, substantially saving on wallclock time. The dashed horizontal lines are visual aids to better compare the final performance after training for 5 x 10 environment steps. We plot IQM and 70% quantile interval over 10 seeds. Appendix A.3 provides table of wallelock times.
4.1 SAMPLE EFFICIENCY OF CROSSQ
Figure compares our proposed CrossQ algorithm with REDQ, DroQ, SAC and TD3 terms of their sample efficiency, i.e., average episode return at given number of environment interactions. a proof of concept, we also present CrossQ (TD3), a version of CrossQ which uses TD3 stead of SAC as the base algorithm. We perform periodic evaluations during training obtain the episodic reward. From these, we report the mean and standard deviations over 10 random seeds. All subsequent experiments in this paper follow the same protocol.
This experiment shows that CrossQ matches or outperforms the best baseline in the pre- sented environments except on Ant, where REDQ performs better in the early training stage, but CrossQ eventually matches it. On Hopper, Wa Mark, and Ha fCheetah, the learning curves of CrossQ and REDQ overlap, and there is no significant difference. On the harder Huma no d and Humano dSt andup tasks, CrossQ and CrossQ (TD3) both substantially surpass all basclines.
4.2 COMPUTATIONAL EFFICIENCY OF CROSSQ
Figure 6 compares the computational efficiency of CrossQ to the baselines. This metric is where CrossQ makes the biggest leap forward. CrossQ requires 20x fewer gradient steps than REDQ and DroQ, which results in roughly 4x faster wallclock speeds (Table 2). Especially the more challenging Humanoid and Humanoids' andup tasks the speedup is the most pronounced. In view, this is noteworthy feature. On the hand, it opens the possibility of training agents in a truly online and data-efficient manner, such as in real-time robot learning. On the other hand, with large computing budgets CrossQ can allow the training of even larger models for longer than what is currently feasible, because of its computational efficiency stemming from its low UTD = 1.
4.3 EVALUATING Q FUNCTION ESTIMATION BIAS
All methods we consider in this paper are based on SAC and, thus, include the clipped double- Q trick to reduce Q function overestimation bias (Fujimoto et al., 2018). Chen et al. (2021) and Hiraoka et al. (2021) stress the importance of keeping this bias even lower to achieve their high performances and intentionally design REDQ and DroQ to additionally reduce bias with explicit and implicit ensembling. In contrast, CrossQ outperforms both baselines without any ensembling. Could CrossQ's 's high performance be attributed to implicitly reducing the bias as a side effeet of design choices? Using the same evaluation protocol Chen et al. (2021), compare the

---

Published as a conference paper at ICLR 2024
IQM Return (1k)
Mcan Norm Q Bias
Std Norm Q Bias
1.2
Humanoid
0.8
0.8
0.4
0.4
2
2
Ast
2
-2
2
10
6 Env Steps (100k)
Env Steps (100k)
Env Steps (100k)
CrossQ (ours)
DroQ
REDQ
SAC (UTD-20)
SAC
Figure 7: Q estimation bias does not reliably influence learning performance. Following the analysis of Chen et al. (2021), we plot the IQM and 70% quantile interval of the normalized Q function bias. REDQ generally has the least bias over 10 seeds. CrossQ matches or outperforms DroQ, REDQ and SAC while showing more function bias in all environments. The full set of environments is shown in Fig. 17 in the Appendix.
normalized Q prediction biases in Figure 4.3. Due to space constraints, here we show Hopper and Ant and place the rest of the environments in Figure 17 in the Appendix.
We find that REDQ and DroQ indeed have lower bias than SAC and significantly lower bias than SAC with UTD 20. The results for CrossQ are mixed: while its bias trend exhibits a lower mean and variance than SAC, in some environments, its bias is higher than DroQ, and in others, it is lower or comparable. REDQ achieves comparable or worse returns than CrossQ while maintaining the least bias. As CrossQ performs better despite having laving-perhaps paradoxically generally higher estimation bias, we conclude that the relationship between performance and estimation bias is complex, and one does not seem to have clear implications the other.
4.4 ABLATIONS
We conduct ablation studies to better understand the impact of different design choices in CrossQ.
4.4.1
DISENTANGLING THE EFFECTS TARGET NETWORKS AND BATCHNORM
CrossQ changes SAC three ways; of these, two explicitly aim to accelerate optimization: the removal of target networks, and the introduction of BatchNorm. Unfortunately, SAC without target networks diverges; therefore, to study the contribution of the first change, we need a way to compare AAC--ivergencee-free friteggenne-freewwihh with and without target networks. Fortunately, we find that such way ex- ists: according to our supplementary experiments in Appendix A.6, simply using bounded activation functions the critic appears to prevent divergence. This a purely empirical observation and an in-depth study regarding the influence of activations and normalizers on the stability of Deep RL is beyond the scope of this paper. In this specific ablation, we use tanh activations instead of relu, solely as a tool to make the intended comparison possible.
Figure& 8 shows the results of our experiment. The performance of SAC without target networks supports the common intuition that target networks indeed slow down learning to a small extent. We find that the combination of BatchNorm and Target Networks performs inconsistently, failing to learn anything in half of the environments. Lastly, the configuration of BatchNorm without target networks and the closest to CrossQ achieves the best aggregate performance, with the boost be- though
ing significantly bigger than that from removing target networks alone. In summary, removing target networks may slightly improve performance some environments, it is the combi- nation of removing target networks and adding BatchNorm that accelerates learning the most.

---

Published as a conference paper at ICLR 2024
Humanoid
Humanoi iamanoddStten tanduu
Walker 2d
IQM Return (Ik)
160
4.5
120
3
2
1.5
1
0
0
Ant
Hal fCheetah
Hopper
IQM Return (Ik)
2.4
3
4
1.6
1.5
2
0.8
0
0
0
100
200
300
0
100
200
300
40
80
120
Env Steps (Ik)
Env Steps (Ik)
Env Steps (Ik)
SAC (tanh TN + BN)
SAC (tanh + BN)
SAC (tanh TM)
SAC (tanh)
Figure 8: The effects of target networks and BatchNorm on sample efficiency. All SAC variants in this experiment use crities with tanh activations, since they allow divergence-free training without target networks, enabling this comparison. This ablation uses the original BatchNorm (BN, Ioffe &
Szegedy (2015)). Removing target networks (-TN) provides only small improvements over the SAC baseline with target nets. BatchNorm with target nets (+BN, green) is unstable. Using BatchNorm best.
after removing target nets (-TN+BN)- -te configuration most similar to We plots IQM return and 70% quantile intervals over 10 seeds.
4.4.2 ABLATING THE DIFFERENT DESIGN CHOICES AND HYPERPARAMETERS
In this subsection, we examine the contributions of the different CrossQ design choices to show their importance. Figure 9 shows aggregated ablations of these components and various hyperparameters, while Figure 10 ablates the BatchNorm layer itself.
CrossQ
SAC
1QM Percent Loss
0-
25
50-
75
100
Hyperparameters. CrossQ uses the best hyperpa- rameters obtained from a series of grid searches. Of these, only three are different from SAC's default values. First, we find that reducing the B1 momen- tum for the Adam optimizer (Kingma Ba, 2015) from 0.9 to 0.5 well the policy delay of 3 have the smallest impact on the performance. However, since fewer actor gradient steps reduce compute, this setting is favorable. Second, reducing the critic net- work's width to 256 the same small size SAC reduces performance and yet still significantly out- performs SAC. This suggests that practitioners may be able to make use of a larger compute budget, i.e., train efficiently across range of different network sizes, by scaling up layer widths according to the available hardware resources. Third, as expected, removing the BRN layers proves to detrimental and results in the worst overall performance. A natural question that comes to mind is whether other normalization strategies in the critic, such as Layer Normalization (LayerNorm, Ba et al. (2016)), would also give the same results. However, in our ablation, we find that replacing BatchNorm with LayerNorm degrades CrossQ's 's performance significantly, roughly to the level of the SAC baseline. Lastly, SAC does not benefit from simply widening critic layers 2048. And naively adding BRN to SAC while keeping the target networks proves detrimental. This finding is in line with our diagnosis of mismatched statistics being detrimental to the training.
w/o BRN
w/o double Q Layer Norm small network
policy delay =
B 0.9
BRN+Th
wide network
SAC
Figure 9: Ablations on CrossQ and SAC. Loss in IQM return in percent-relative to CrossQ=at 1M environment interactions. Aggregated over all environments and six seeds each, with 95% bootstrapped confi- dence intervals (Agarwal et al., 2021). Left shows CrossQ ablations; Right shows effects of adding parts on top of SAC. Figure 13 in Appendix shows individual training curves.
8

---

Published as a conference paper at ICLR 2024
Humancie
HumanoidStandup
Ant
Hopper
1QM Return (1k)
12
400
300
200
2
100
2
0
0
1
2
4
5
0
3
4
0
2
3
2
5
Env Steps (1M)
Env Steps (IM)
Env Steps (IM)
Env Steps (1M)
CrossQ (ours)
BRN critic only
BRN (0.999)
BRN (0.9)
BN (0.99)
Comparing BatchNorm hyperparameters. All variants have comparably strong and stable curves early in the training. Omitting normalization in the actor (BRN critic only) does not significantly affect CrossQ. Using the original Batch Normalization (BN, with moving--vvraag mo- mentum 0.99) is prone to sudden performance collapses during longer training runs. Using BRN permits stabler training, which improves with higher momentums; CrossQ's default 0.99 (black) and higher show no collapses. We plot IQM return and 70% quantile intervals over five seeds.
Figure 10:
Batch Normalization Layers. In Figure 10, ablate the BatchNorm versions (BN (loffe & Szegedy, 2015) and BRN (loffe, 2017)) and their internal moving-average momentums. Compared CrossQ's optimal combination-BRa with momentum 0.99-all variants have similar sample ef- ficiency in the early stages of training (1M steps). When using BN, we sometimes observe sudden performance collapses later training; we attribute these to BN's unique approach of using noisy minibatch estimutes of normalization moments. BRN's improved approach of using the less noisy moving-averages makes these collapses less likely; further noise-reduction via higher momentums eliminates these collapses entirely. Additionally, we find that using BatchNorm only in the critic (instead of both the actor and the critic) is sufficient to drive the strong performance of CrossQ;; however, including it in both networks performs slightly better.
CONCLUSION & FUTURE WORK
We introduced CrossQ, new off-policy RL algorithm that matches or exceeds the performance of REDQ and DroQ -he current state-of-the-ar on continuous control environments with state observations-in terms of sample efficiency while being multiple times more computationally effi- cient. To the best of our knowledge, CrossQ is the first method to successfully use BatchNorm to greatly accelerate off-policy actor-critic RL. Through benchmarks and ablations, we confirmed that target networks do indeed slow down training and showed a way to remove them without sacrificing training stability. We also showed that BatchNorm has the same accelerating effect training in Deep RI as it does in supervised deep learning. The combined effect of removing target networks and adding BatchNorm is what makes CrossQ so efficient. We investigated the relationship between the Q estimation bias and the learning performance of CrossQ, but did not identify a straightfor- ward dependence. This indicates that the relationship between the estimation bias and the agent performance is more complex than previously thought.
In future work, would be interesting to analyze the Q estimation bias more extensively, similar to Li et al. (2022). Furthermore, deeper theoretical analysis of the used BatchNorm approach in the context of RL would be valuable, akin to the works in supervised learning, e.g., Summers & Dinneen (2020). Although the wider critic networks do provide an additional performance boost, they increase the computation cost, which could potentially be reduced. Finally, while our work focuses the standard continuous control benchmarking environments, a logical extension would be applying CrossQ to a real robot system and using visual observations in addition to the robot RL, such as state augmentation (Laskin al., 2020; Yarats et al., 2021) and auxiliary losses (Schwarzer et al., 2021; He et al., 2022), also aim to learn efficiently from limited data. We believe some of these ideas could potentially be applied CrossQ.
state. Techniques from image-

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
We acknowledge the grant "Einrichtung eines Labors des Deutschen Forschungszentrum für Künstliche Intelligenz (DFKT) an der Technischen Universität Darmstadt" of the Hessisches Minis- terium für Wissenschaft und Kunst. This research was also supported by the Research Clusters "The Adaptive Mind" and "Third Wave of AI", funded by the Excellence Program of the Hessian Min- istry of Higher Education, Science, Research and the Arts, Hessian. Al and by the German Research Foundation (DFG): 417962828.
REFERENCES
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G Bellemare. Deep reinforcement learning at the edge of the statistical precipice. In Advances in neural infor- mation processing systems, 2021.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 167.:0660.0064500 2016.
Xinyue Chen, Che Wang. Zijian Zhou, and Keith Ross. Randomized ensembled double Q-learning: Learning fast without a model. In International conference on learning representations, 2021.
Jianqing Fan, Zhaoran Wang, Yuchen Xie, and Zhuoran Yang. A theoretical analysis of deep Q- learning. Learning for dynamics and control, 2020.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference machine learning. 2018.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learning, 2018a.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Schoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishck Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and appli- cations. arXiv preprint
2018b.
Hado Hasselt. Double Q-learning. In Advances in neural information processing systems, 2010.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. Conference on computer vision and pattern recognition, 2016.
Tairan He, Yuge Zhang, Kan Ren, Minghuan Liu, Che Wang, Weinan Zhang, Yuqing Yang, and Dongsheng Li. Reinforcement learning with automated auxiliary loss search. In Advances in neural information processing systems, 2022.
Takuya Hiraoka, Takahisa Imagawa, Taisei Hashimoto, Takashi Onishi, and Yoshimasa Tsuruoka.
Dropout fufuetions for doubly efficient reinforcement learning. In International conference on learning representations, 2021.
Sergey loffe. Batch renormalization: Towards reducing minibatch dependence in batch-normalized models. In Advances in neural information processing systems, 2017.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, 2015.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model- based policy optimization. In Advances in neural information processing systems, 2019.
Seungehan Kim, Kavosh Asadi, Michael L. Littman, and George Dimitri Konidaris. Deepmellow: Removing the need for a target network in deep Q-learning. In International joint conference on artificial intelligence, 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International conference on learning representations, 2015.
10

---

Published as a conference paper at ICLR 2024
Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Rein- forcement learning with augmented data. In Advances in neural information processing systems, 2020.
Qiyang Li, Aviral Kumar, Ilya Kostrikov, and Sergey Levine. Efficient deep reinforcement learning requires regulating overfitting. In International conference on learning representations, 2022.
Timothy P Lillicrap. Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Inter- national conference on machine learning, 2016.
Hyesu Lim, Byeonggeun Kim, Jacgul Choo, and Sungha Choi. TTN:
A domain-shift aware batch normalization in test-time adaptation. In International conference on learning representations, 2023.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533 2015.
M. Morales. Grokking deep reinforcement learning. Manning Publications. 2020.
Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In International conference machine learning, 2022.
Kei Ota, Tomoaki Oiki, Devesh Jha, Toshisada Mariyama, and Daniel Nikovski. Can increasing in- put dimensionality improve deep reinforcement learning? In International conference machine learning, 2020.
Kei Ota, Devesh K Jha, and Asako Kanezaki. Training larger networks for deep reinforcement learning, arXiv preprint arXiv:2102.07920, 2021.
Quang Pham, Chenghao Liu, and Steven HOI, Continual normalization: Rethinking batch normal- ization for online continual learning. In International conference on learning representations, 2022.
Matthias Plappert. Marcin Andrychowicz. Alex Ray, Bob McGrew, Bowen Baker, Glenn Pow- ell, Jonas Schneider, Josh Tobin, Macick Chociej, Peter Welinder, et al. Multi-goal reinforce- ment learning: Challenging robotics environments and request for research. arXiv preprint 1r0iv:1802.09444 2018.
Martin L Puterman. Markov decision processes: Discrete stochastic dynamic programming. John Wiley & Sons, 2014.
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dor- mann. Stable-baselines3: Reliable reinforcement learning implementations. Journal of machine learning research, 22(268): 2021.
Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch normal- ization help optimization? In Advances neural information processing systems, 2018.
Max Schwarzer, Ankesh Anand, Rishab Goel, Devon Hjelm, Aaron Courville, and Philip Bach- man. Data-efficient reinforcement learning with self-predictive representations. In International conference on learning representations, 2021.
Saurabh Singh and Abhinav Shrivastava. Evalnorm: Estimating batch normalization statistics for evaluation. In International conference on computer vision, 2019.
Laura Smith, Ilya Kostrikov, and Sergey Levine. walk in the park: Learning to walk in 20 minutes with model-free reinforcement learning. arXiv preprint 2tt.:22288.786000 2022.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of machine learning 2014.
research,

---

Published as a conference paper at ICLR 2024
Cecilia Summers and Michael J. Dinneen. Four things everyone should know to improve batch normalization. In International conference on learning representations, 2020.
Yuval Tassa, Yotam Doron, Alistair Muidal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, al. Deepmind control suite. arXiv preprint arXiv: 000900 2018.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In International conference on intelligent robots and systems, 2012.
Mark Towers, Jordan K. Terry, Ariel Kwiatkowski, John U. Balis, Gianluca de Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Arjun KG, Markus Krimmel, Rodrigo Vicent,,, An- drea Pierré, Sander Schulhoff, Jun Jet Tai, Andrew Tan Jin Shen, and Omar G. Younis. Gymna- sium, 2023.
Ge Yang, Anurag Ajay, and Pulkit Agrawal. Overcoming the spectral bias of neural value approxi- mation. In International conference on learning representations, 2021.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous con- trol: Improved data-augmented reinforcement learning. In International conference on learning representations. 2021.
Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, al. Maximum entropy inverse reinforcement learning. In AAAI conference on artificial intelligence, 2008.
12

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 DEEPMIND CONTROL SUITE EXPERIMENTS
Figure 11 presents an additional of experiments performed on the DeepMind Control Suite (Tassa et al., 2018). The experiments shown here are an extension to the experiments shown in Figure 5 in the main paper and have been moved the Appendix due to space constraints. For the presented tasks, we lowered the learning rate x 10 for all algorithms, and set the CrossQ policy delay 1. All other hyperparameters remained the same as for the main paper.
reacher-easy
reacher-hard
ba lall 1 in cup-catcc
1000
1000
1000
1QM Return
800
600
600
600
400
400
400
200
200
200
0.2
0.4
0.2
0.4
0.6
0.8
1.2
1.6
2
finger -spin
fish
Swim.
humanoid -iaaooiddd-dtddddd
1000
1000
1000
10M Return
800
800
800
600
600
200
0
0
2
10
20
30
40
20
30
40
50
Env Steps (100k)
Env Steps (100k)
Env Steps (100k)
Cross (ours)
REDQ
SAC
Figure 11: Sample efficiency of CrossQ on Deep.Mind Control. The experiments here were each performed on 5 different random seeds. Cross(''s good sample efficiency transfers well to the presented tasks from the DeepMind Control Suite.
13

---

Published as a conference paper at ICLR 2024
A.2 HYPERPARAMETERS
Experiment hyperparameters, used in the main paper. We adapted most hyperparameters that are commonly used in other works (Haarnoja et al., 201 Chen et al., 2021; Hiraoka al., 2021). The Moving-Average Momenium corresponds to minus the Moving-Average Update Rate as de- fined in both BatchNorm papers (Toffe & Szegedy, 2015; loffe, 2017).
Table
Learning Hyperparameters
Parameter
SAC REDQ DroQ
CrossQ (ours)
Discount Factor (7)
0.99
Learning Rate (Actor & Critic)
0.001
Replay Buffer Size
106
Batch Size
256
Activation Function
relu
Layer Normalization
No
Yes
No
Dropout Rate
N/A
0.01
N/A
BatchNorm / Version
N/A
BRN
BatchNorm Moving-Average Momentum
N/A
0.99
BatchNorm BRN Warm-up Steps
N/A
10a
Critic Width
256
2048
Target Update Rate (r)
0.005
N/A
Adam $1
0.9
0.5
Update- To-Data ratio (UTD)
1
20
1
Policy Delay
20
3
Number Crities
A.3 WALLCLOCK TIME MEASUREMENT
Wallclock times were measured by timing and averaging over four seeds each and represent pure training times, without the overhead of synchronous evaluation and logging, until reaching 5 x 10t environment steps. The times are recorded on an Nvidia RTX 3090 Turbo with an AMD EPYC 7453 CPU.
Table 2: Wallclock times. Evaluated for CrossQ and baselines across environments in hours and recorded on an RTX 3C90, the details of the measurement procedure are described in Appendix 4.2. Comparing CrossQ with CrossQ (Small) and SAC, it is apparent that using wider critic networks does come with performance penalty. However, compared to REDQ and DroQ, one clearly sees the substantial improvement in Wallclock time of CrossQ over those baselines.
Wallclock Time [hours]
SAC
CrossQ (small) CrossQ (ours)
REDQ
DroQ
HumanooidStanndup-vv
1.5
2.1
2.2
8.7
7.5
Walker2d-v4
0.9
0.9
1.1
4.0
4.1
Ant-v4
0.9
1.2
1.5
4.7
4.7
fh
0.8
1.2
1.5
4.1
4.4
Hopper-v4
1.0
1.1
1.3
4.1
4.2
14

---

Published as a conference paper at ICLR 2024
A.4 EVOLVING ACTION DISTRIBUTIONS
Training 61x10*
0.8 10
12210*
18x10
24x10' 3.0
0
04
a1 04
Figure 12: Replay and policy action distributions different, and evolve during training. We train an agent for 300, 000 steps on Walker2d. We take snapshots of the replay buffer B and policy n every 60, 000 steps. For each snapshot (one column), sample large batch of 10, 000 transitions (s, a. ss, a' = o (s)) and use this to compute a visually interpretable 2D kernel density estimate of the distributions of (blue) and a' (red), seen through the action-space dimensions and 5. The cross denotes the mean, and the dashed ellipse is one standard deviation wide for each of the dimensions. We observe that the distributions as well as the means and standard deviations of the off- aff-policy and on-policy actions are visibly and persistently different throughout the training run, and keep drifting the training progresses. This discrepancy implies that BatchNorm must be used with care in off-policy TD learning.
15

---

Published as a conference paper at ICLR 2024
A.4.1 ABLATING THE DIFFERENT DESIGN CHOICES AND HYPERPARAMETERS
Figure 13 depicts in detail the CrossQ and SAC ablations, previously shown in aggregate form by
Figure 9.
Humanoid id
Humanoi ieemooddStaaae dSt.andup
Wal ker2d
IQM Return (1k)
12
300
200
100
Ant
Hal aChefheetahh
Hopper
1QM Return (1k)
7.5
16
5
2.5
0
-2.5
2
8
10
2
10
0
6
10
Env Steps (100k)
Env Steps (100k)
Env Steps (100k)
CrossQ (ours)
SAC BRN-TN
SAC wide Net
Layer Norm
w/ BRN
CrossQ (small)
B1 = 0.9
policy delay
w/o double Q
SAC
Figure 13: CrossQ ablation study. We ablate across different hyperparameter settings and ar- chitectural configurations. Using the same network width SAC, CrossQ (small) shows weaker performance, yet is still competitive with CrossQ in four out of six environments. At the same time, SAC with wider critic does work better. Using the default Adam momentum B 0.9 instead of 0.5 degrades performance in some environments. Using policy delay of instead of 3 has very small effect, except on Ant. Using LayerNorm instead of BatchNorm results in slower learning; also trains stably without target networks. Removing BatchNorm results in failure of training due to divergence. Adding BatchNorm to SAC and reusing the live critic's s normalization moments in the target network fails to train. Training without double Q networks (single critic) harms performance.
16

---

Published as a conference paper at ICLR 2024
A.5 REDQ AND DROQ ABLATIONS
Figures 14 and 15 show REDQ and DroQ ablations on seeds each. They show both baselines with the CrossQ hyperparameters: wider critic networks as well 0.5. Neither baseline benefits from the added changes. In most cases, the performance is unchanged, while in some cases, deteriorates. The dashed black line shows CrossQ reference.
Humanoid
HumancidStandie
Walker2d
IQM Retum (1k)
400
300
200
100
Ant
HalfCheetah
Hopper
IQM Return (1k)
8
16
2
10
4
10
Env Steps (100k)
Env Steps (100k)
Env Stops (100k)
CrossQ (ours)
REDQ wide network
REDQ wide network
0.5
REDQ
REDQ 81 0.5
Figure 14: REDQ ablation. Showing performance for different combinations of the CrossQ hyper- parameters. The changes in hyperparameters do not help REDQ to get better performance. fact, in some cases, they even hurt the performance.
Humanoid id
Humanoi aSdanuupp
Walker2d
IQM Return (Ik)
12
400
300
200
100
Ant:
Ha 1 @Cheetah
Hopper
IQM Return (Ik)
12
8
0
2
10
2
10
0
6
Env Steps (100k)
Env Steps (100k)
Env Steps (100k)
CrossQ (ours)
DroQ wide network
DroQ + wide network B1 0.5
DroQ
DroQ B: = 0.5
Figure 15: DroQ ablation. The changes in hyperparameters do not help DroQ to get better perfor- mance overall. In Hopper and Ant. performance rises to the Cross( performance, however, the Humanoide it hurts performance.
17

---

Published as a conference paper at ICLR 2024
A.6 EFFECT OF ACTIVATIONS AND NORMALIZERS ON LEARNING STABILITY
Figure 8 depicts small exploratory experiment in which we remove target networks from SAC, and train it with different activation functions and feature normalizers. We do this only explore whether the boundedness of activations has an influence on training stability. We learn from this experiment that SAC with tanh activations trains without divergence, allowing us to conduct the study in Section 4.4.1. We also observe that at least two feature normalization schemes (on top of the unbounded relu activations) permit divergence- optimization.
For vectors relu_over max(() denotes simple normalization scheme using an underlying un- bounded activation: relu(æ)/ rclu(=)/max(()), max(x), with the maximum computed over the entire feature vec- tor. layernormed _relu simply denotes LayerNorm applied after the relu activations. Both of these schemes prevent divergence. Using LayerNorm before the relu activations also prevent divergence, and is already explored in the ablations in Figure 13. None of these normalizers perform strongly as BatchNorm.
A thorough theoretical or experimental study of how activations and normalizers affect the stability of Deep RL beyond the scope of this paper. We hope, however, that our observations help inform future research directions for those interested in this topic.
Hopper
Ant
Hal #Cheetah
IQM Norm Q Bias
2
0
-2
40
80
120
0
100
200
300
100
200
300
Walker2d
Humanoid d
HumanoidSt andup
1QM Norm Q Bius
0
-2
100
200
300
0
100
200
300
100
200
300
Env Steps (1k)
Env Steps (Ik)
Env Steps (Ik)
relu
layerormed, relu
elu
relu over max
relu6
glu
tanh
Figure 16: (In)stability of SAC without target networks. Observed through the Q estimation bias. In this small-scale experiment, we run SAC with unbounded (relu, glu, elu) and bounded (tanh, relu6, sin) activation functions, well as "indirectly" bounded activations through the use of two custom normalizers other than BatchNorm (relu_over max, layernormed SAC variants with unbounded activations appear highly unstable in most environments, whereas the variants with bounded activations (as well as the normalizers) do not diverge, maintaining relatively low bias.
18

---

Published as a conference paper at ICLR 2024
A.7 NORMALIZED Q BIAS PLOTS
Figure 17 shows the results of the Q function bias analysis for all environments.
IQM Return (Ik)
Mean Norm Q Bias
Std Norm Q Bias
1.2
1.2
Humano ast andup Humanoid
0.4
0.4
400
1.2
320
0.8
0.8
240
160
0.4
0.4
80
Walk Walker2d
0.8
0.8
4.5
0.4
Ant
4
0
Bal-Cheetab
15
1.2
1.2
0.4
0.4
1.2
1.2
Sopper
3.2
2.4
1.6
0.4
0.4
0.8
2
10
6
Env Stops (100k)
Env Stops (100k)
Env Steps (100k)
CrossQ (ours)
DroQ
REDQ
SAC (UTD-20)
SAC
Q estimation bias. Mean and standard deviation of the normalized Q function bias, computed as described by Chen et (2021). As in the main paper, we do not find straightforward connection between normalized Q function bias and learning performance. CrossQ generally shows the same or larger Q estimation bias compared REDQ but matches outperforms REDQ in learning speed, especially on the challenging Humanoid tasks.
Figure 17:
19


Reviews:
[{"review_id": "oY2bbXpjYd", "reviewer": "Reviewer_stzr", "summary": "The paper presents CrossQ, an algorithm for deep reinforcement learning, aiming to address the crucial issue of sample efficiency. CrossQ achieves state-of-the-art sample efficiency, reduces computational costs significantly compared to existing methods (REDQ and DroQ), and simplifies the implementation process by eliminating the need for advanced bias-reduction schemes and high update-to-data (UTD) ratios. The core innovations of CrossQ include the removal of target networks, strategic use of Batch Normalization, and wider critic networks.", "questions": "1. Can you elaborate on the specific scenarios or domains where CrossQ might outperform other algorithms significantly? Are there any limitations or challenges in which CrossQ may not be the most suitable choice?\n\n2. The paper focuses on empirical results; however, are there any insights or intuitions on why the chosen modifications (removal of target networks, Batch Normalization, wider networks) lead to the observed improvements? Providing additional context on the underlying mechanisms would enhance the paper's impact.\n\n3. To me, I think using wider critic networks played a pivot role in the performance boost, especially in humanoid-based environments. Can you draw SAC+wider networks to Figure 7?\n\nI would be happy to raise my scores if my concerns are addressed.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Well-written paper, easy to follow. The paper is well-structured and clearly articulates the motivation, methodology, and experimental results. The use of figures and code snippets aids in understanding the algorithm's implementation, making it accessible to readers with varying levels of expertise.\n\n2. CrossQ's approach stands out in the field of deep reinforcement learning by challenging the trend of high UTD ratios. The innovative removal of target networks, combined with Batch Normalization and wider critic networks, is a departure from conventional methods, leading to improved sample efficiency and reduced computational complexity.\n\n3. The paper demonstrates a rigorous empirical analysis of CrossQ's performance through ablation studies, clearly showcasing the algorithm's effectiveness. The use of Batch Normalization in a novel manner, coupled with wider networks, highlights a high-quality exploration of the algorithm's design space.", "weaknesses": "1. While the empirical results are compelling, the paper lacks theoretical analysis or justification for the success of CrossQ. Providing insights into why the proposed modifications lead to improved performance would enhance the paper's depth and contribute to a more comprehensive understanding of the algorithm.\n\n2. The experiments are only conducted in one continuous control domain (MuJoCo). It is dangerous to draw conclusion of sample efficiency only from one experimental domain.\n\n3. Also, this paper does not include image-based settings. There are tons of efficient RL works recently on image-based settings and some of the techniques (state augmentation [1, 2] and auxiliary losses [3, 4]) can be also applied to state-based settings. I think it is worth discussing the recent progress of efficient RL in the related work session or adding baselines for comparison.\n\n[1] Laskin, M., Lee, K., Stooke, A., Pinto, L., Abbeel, P., & Srinivas, A. (2020). Reinforcement learning with augmented data. Advances in neural information processing systems, 33, 19884-19895.\n\n[2] Yarats, D., Fergus, R., Lazaric, A., & Pinto, L. (2021). Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint arXiv:2107.09645.\n\n[3] Schwarzer, M., Anand, A., Goel, R., Hjelm, R. D., Courville, A., & Bachman, P. (2020). Data-efficient reinforcement learning with self-predictive representations. arXiv preprint arXiv:2007.05929.\n\n[4] He, T., Zhang, Y., Ren, K., Liu, M., Wang, C., Zhang, W., ... & Li, D. (2022). Reinforcement learning with automated auxiliary loss search. Advances in Neural Information Processing Systems, 35, 1820-1834.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "79GF3RRYrP", "reviewer": "Reviewer_enbH", "summary": "The paper proposes very simple adaptations to the well known SAC algorithm to make it more data and especially more compute efficient. While recent works focused on increasing UTD ratios to make more use of the collected data and thus becoming more data efficient at the cost of compute resources, the authors go the opposite way and show that even with UTD=1 competetive data efficiency is possible with only a fraction of the compute. The main ingredients for the new method are the addition of BatchNormalization, the removal of widely believed necessary target networks, different activation functions as well as wider critic networks.", "questions": "See weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The proposed method achieves competitive sample efficiency while needing only a fraction of the compute, as demonstrated in figures 1 & 5. The paper showcases some interesting findings to enable these advantages - target networks are widely believed to be necessary for stable learning, however the authors found a way around them by using BatchNorm in a novel fashion. Since target networks necessarily slow down learning, this is one part that enables the computationally more efficient learning process. The paper is well written and the major points come across clearly.", "weaknesses": "The paper makes very broad claims in terms of proposing the new state of the art in both sample and compute efficiency - they might be justified, however only relatively few baselines (DroQ, REDQ, SAC) are used for comparison. For statements like this, a broader comparison might be required. \n\nFurther, even though I am no expert on the the achievable performance in the environments used, I know that at least for some of them higher final performances are possible - e.g. on HalfCheetah I have seen policies do much better than 10k (interestingly the D4RL paper even reports 12k for SAC, whereas in Fig 4. it looks like you only get about 7.5k) - even if other algorithms use more data / compute: if the superior efficiency were only attainable for suboptimal performances, that would be quite limiting and needs to be examined. Please clarify\n\nIf you could show some evidence on the suspicion that the reason for prior approaches being unable to use BatchNorm effectively is the OOD-ness of the actions from the policy that are not sampled from the batch in the target networks, that would be great as well since it is I think one of the key insights (i.e. visualize the distributions of sampled and policy actions).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cXnt8YRhLe", "reviewer": "Reviewer_e62k", "summary": "The authors address the efficiency of Soft Actor-Critic (SAC) and related deep RL methods for continuous control. Recent methods such as REDQ and DroQ have large update-to-data (UTD) ratios when training the critic (as high as 20). The authors introduce a new method called CrossQ that\n\n1. reduces the critic's UTD to 1,\n1. uses batch normalization,\n1. removes target networks, and\n1. increases hidden-layer widths.\n\nIt is shown that these changes are still able to achieve high sample efficiency in MuJoCo environments while being significantly cheaper to run. It is also hypothesized that two of the architectural changes (batch norm and no target networks) are what enable stable training with such a low UTD ratio.\n\n**Score raised from 5 to 8 during rebuttal.**", "questions": "1. At the end of the introduction, you say that your success with batch norm “contradicts” another paper [2] that did not find batch norm to work well. Why do you think you were able to achieve better results? Is it because you removed the target network, or is there another reason?\n1. Figure 2’s caption says, “this removes the computational need for two individual forward passes through the same network” in reference to batching the observations and next observations together. But this doesn’t actually reduce computational cost, does it? The same number of forward passes are being done either way.\n1. Could you explain the evaluation procedure in the Q-estimation bias experiment (Figure 6)?\n1. Could you also explain what is meant by a policy delay of 3? How is this different from the UTD ratio?\n\n**References**\n\n[1] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. 2016.\n\n[2] Takuya Hiraoka, Takahisa Imagawa, Taisei Hashimoto, Takashi Onishi, and Yoshimasa Tsuruoka. Dropout Q-functions for doubly efficient reinforcement learning. 2021.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- Good empirical results in MuJoCo. The tested baselines, REDQ and DroQ, are recent and strong baselines yet CrossQ still performs equally or better in the tested environments.\n- Removes the target networks of SAC. This is a pain point in terms of both learning speed and implementation, and this is the first time (that I know of) that it has been removed without hurting performance. The authors provide an interesting and sensible hypothesis about the interaction between batch normalization and target networks in Section 3.2 to support their empirical observations. It would be exciting if this generalizes to other algorithms as well.\n- The total computation and wall-clock execution time of the proposed method is significantly less than baseline methods, yet sample efficiency is not sacrificed. Along with its simplicity, the new method is very appealing.\n- The paper is well written and the discussion of recent related papers is excellent.", "weaknesses": "- The paper claims it “show[s] the first successful application of BatchNorm in off-policy Deep RL.” However, it appears that DDPG [1] also used batch normalization 7 years ago in the exact same setting: off-policy deep RL for continuous control in MuJoCo. This means the titular contribution of the paper is not original.\n- There is a lot of hyperparameter tuning that might be unfairly benefiting CrossQ. For instance, the network width for CrossQ is increased from 256 to 2048 units, an 8x increase. Adam’s momentum parameter was also reduced from 0.9 to 0.5. As far as I can tell, these hyperparameters were not tested for the baselines, so CrossQ has been over-optimized for MuJoCo. A fairer comparison would be testing the methods with the same network sizes and then selecting the best Adam hyperparameters for each of them.\n- I feel that the claim of “state-of-the-art” performance is a bit strong given that only 6 MuJoCo environments were tested, and CrossQ is a clear improvement only in 2 of them. The method has not been evaluated in other continuous-control problems nor against other potential baselines, so I would not necessarily agree that it is state-of-the-art. Nevertheless, the empirical results are still great.\n\n**Minor Edits**\n- Inconsistent use of hyphens should be fixed. For example, “Q-function” and “Q function” both appear in the paper.\n- Possible typo in Figure 2’s caption: “a **single double** Q function network”?\n- As Equation 1 is currently written, it is not correct to refer to the Polyak averaging parameter $\\tau$ as “momentum” since $\\tau=1$ corresponds to a moving average with no inertia.\n- References are not consistently formatted, e.g., some conference names are capitalized but others are not.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cXmyMDTrU2", "reviewer": "Reviewer_7EH3", "summary": "The authors present CrossQ, a variation of SAC that removes target networks and leverages Batch Normalization to improve learning efficiency while avoiding high update-to-data ratios. CrossQ presents a straightforward modification of SAC and is experimentally validated on a selection of tasks from the OpenAI Gym.", "questions": "- How does CrossQ compare to REDQ and DroQ under the experiments mentioned above?\n- Have you tried removing the double Q functions on top of removing the target networks?\n- Have you tried keeping the target network and normalize with batch norm parameters of the regular network?\n- Do you have an intuition for why CrossQ (black line) performs worse on HalfCheetah in Figure 7?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "-\tThe changes over SAC are straight-forward and easy to implement, simplicity being a great asset in RL\n-\tThe approach removes target network dynamics as a step towards decluttering RL\n-\tImproved learning efficiency over the baselines on the selected tasks\n-\tSeveral interesting ablation studies", "weaknesses": "- The evaluation looks promising but some experiments are missing to yield a well-rounded study. For example, CrossQ updates the Adam momentum and critic width compared to REDQ and DroQ. How do REDQ and DroQ compare to CrossQ when we set their (1) Adam momentum to 0.5, (2) critic width to 2048, (3) momentum to 0.5 and width to 2048? To reduce compute requirements, this could be studied for e.g., 5 seeds on Humanoid + HumanoidStandup + 1 other task\n- The concept should be sufficiently general to combine with other baseline algorithms than SAC, so evaluation across multiple algorithms would further broaden applicability [i.e.are the results specific to SAC variations?]\n- Generally, a more diverse set of environments would further strengthen the evaluation – this could include other domains (DeepMind Control Suite / MetaWorld / …) or visual control tasks [i.e.are the results specific to OpenAI Gym domains?]", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)