Q-Ensemble for Offline RL: Don't Scale the Ensemble, Scale the Batch Size
Authors: Alexander Nikulin
Vladislav Kurenkov
Denis Tarasov
Dmitry Akimov
Sergey Kolesnikov
Venue: Submitted to ICLR 2023 (2023)
Keywords: Offline Reinforcement Learning
Q-Ensemble
Large Batch Optimization
Ensemble Based Reinforcement Learning

Abstract: Training large neural networks is known to be time-consuming, where learning duration may stretch up to days or weeks. To address this problem, the approach of large-batch optimization was introduced, demonstrating that scaling mini-batch sizes with appropriate learning rate adjustments may speed up the training process by orders of magnitude. While long training time was not typically a major issue for model-free deep offline RL algorithms, recently introduced Q-ensemble methods achieving state-of-the-art performance made this issue more relevant, notably extending the training duration. In this work, we demonstrate how large-batch optimization, typically overlooked in deep offline RL community, can benefit this class of methods. We show that simply scaling the mini-batch size and naively adjusting the learning rate allows for (1) a reduced size of the Q-ensemble, (2) stronger penalization of out-of-distribution actions, and (3) improved convergence time, effectively shortening training durations by 2.5x times on average.

Under review as a conference paper at ICLR 2023
Q- ENSEMBLE FOR OFFLINE RL: DON'T SCALE THE ENSEMBLE, SCALE THE BATCH SIZE
Anonymous authors Paper under double-blind review
ABSTRACT
Training large neural networks is known to be time-consuming, with the learning duration taking days or even weeks. To address this problem, large-batch optimiza- tion was introduced. This approach demonstrated that scaling mini-batch sizes with appropriate learning rate adjustments can speed up the training process by orders of magnitude. While long training time was not typically a major issue for model-free deep offline RL algorithms, recently introduced Q-ensemble methods achieving state-of-the the-thhertt performance made this issue more relevant, notably extending the training duration. In this work, we demonstrate how this class of methods can benefit from large-batch optimization, which is commonly overlooked by the deep offline RI. community. We show that scaling the mini-batch size and naively adjusting the learning rate allows for (1) reduced size of the Q-ensemble, (2) stronger penalization of out-of-distribution actions, and (3) improved convergence time, effectively shortening training duration by 3-4x times average.
INTRODUCTION
Offline Reinforcement Learning (ORL) provides a data-driven perspective learning decision- making policies by using previously collected data without any additional online interaction during the training process (Lange al.) [2012] Levine et al.) 22220) Despite its recent development (Fujimoto
ett al. 2019 Nair et al. 2020; An et al.
Zhou et al. 2021 Kumar et al. (0200) and application
progress (Zhan al. 2022 Apostolopoulos et al. 2021] Soares et 2021). one of the current challenges in ORL remains algorithms extrapolation error, which is an inability to correctly estimate the values of unseen actions (Fujimoto et al. 2019). Numerous algorithms were designed to address this issue. For example, Kostrikov et al. (2021) (IQL) avoids estimation for out-of- actions entirely. Similarly, Kumar et al. (2020) (CQL) penalizes -ustt--distribution actions such that their
values are lower-bounded. Other methods explicitly make the learned policy closer to the behavioral one (Fujimoto & Gu 2021 Nair et al. 2020 Wang al. 2020).
In contrast to prior studies, recent works (An et al. 2021 demonstrated that simply increasing
the number of value estimates in the Soft Actor-Critic (SAC) (Haarnoja et al.) 2018) algorithm is enough to advance state-of-the-art performance consistently across various datasets in the D4RL benchmark (Fu et al. 2020). Furthermore, An et al. (2021) showed that the double-clip trick actually serves umcertainty-quaaiiictttn quantification mechanism providing the lower bound of the estimate, and simply increasing the number of critics can result in a sufficient penalization for -otto-ddistributionn actions. Despite its state-of-the-are results, the performance gain for some datasets requires significant computation time or optimization of an additional term, leading to extended training duration (Figure 2). In this paper, inspired by parallel works on reducing the training time of large models in other areas of deep learning YYuu al., 2019 2017) (commonly referred to as large batch optimization), we study the overlooked use of large batches in the deep ORL setting. We demonstrate that, instead of increasing the number of crities or introducing an additional optimization term SAC-N (An et al. 2021 algorithm, simple batch scaling and naive adjustment of the learning rate can (1) provide a sufficient penalty on -ut-of-distributiin actions and (2) match
performance on
'Offline RI. is often referred to as Batch RI., here, extensively use the hatch term denote mini-batch, the dataset size.

---

Under review as a conference paper at ICLR 2023
the D4RL benchmark. Moreover, this large batch optimization approach significantly reduces the convergence time, making possible to train models 4x faster on a single-GPe setup. To the best of knowledge, this is the first study that examines large batch optimization in the ORL setup.
2 Q-ENSEMBLE FOR OFFLINE RL
LB-SAC
Q-Ensemble, 50
+- Ensembbe Update, Vanilla
Mini-Batch, B 10K
Q
Upcate each Q-laaezin gradiers doceet
(5,a,'', (s,0,8',r)
1 Q [J]
Σ Janriee
Qu (6.0) -v(rx))
N
SAC-N
Q-Ensemble, 500
Q-Enc Ensemble Update, Vanilla
Q1
Updar Q-traction Qo, with descent
(8, a, 80, .r. (s,a,8,,))
QG (6.0) -v(re))
N
EDAC
F.Esembble, - 50
Q-Ensemble Update, Diversifled
Q1
Updan cack Qolanation Qo with gradicet descers usite
(8,a, 8', (s,a,8'r))
Q
Velpil
parkied
Σ
(a..)-s(rra))) W-
E
(A
Figure 1: The difference between recently introduced SAC-N, EDAC, and the proposed LB-SAC. The introduced approach does not require auxiliary optimization term while making it possible to effectively reduce the number critics in the Q-ensemble by switching to the large-batch optimization setting.
Ensembles have long history of applications in the reinforcement learning community. They are employed in the model-based approaches to combat compounding error and model exploitation (Kurutach et al. 2018 Chua et al. 2018 Lai et al., 2020 Janner et al. (209)), in model-free to greatly
increase sample efficiency (Chen al. 2021 Hiraoka et al. 2021 Liang al. 2022) and in general
to boost exploration in online RI. (Osband et al. 2016 Chen et al.] 2017 | Re et al. 2021 Ciosek
[
ctt al. 2019). In offline RL, ensembles were mostly utilized to model epistemic uncertainty in value
function estimation (Agarwal et al. 2020 Bai al. 2022 Ghasemipour et al. 2022), introducing uncertainty aware conservatism.
Recently, An al. 12021) investigated isolated effect of clipped Q-learning value overestimation in offline RL, increasing the number of crities in the Soft Actor Critic (Haarnoja et al.) 2018 algorithm from N. Surprisingly, with tuned N SAC-N outperformed previous state-of-the-an algorithms on D4RL. benchmark (Fu et al. 2020) by large margin, although requiring up to 500 critics on some datasets. To reduce the ensemble size, An al. (2021) proposed EDAC which adds auxiliary loss to diversify the ensemble, allowing to greatly reduce N (Figure I).
Such pessimistic Q-ensemble can be interpreted as utilizing the Lower Confidence Bound (LCB) of the Q-value predictions. Assuming that ((s,, a) follows Gaussian distribution with mean m, standard deviation and (Q, a)) are realizations of Q(s, a), we can approximate expected
minimum (An et al. 2021 Royston 1982)
N
E
min
(5(,,a)) m(ss Φ-
(sssa)
(1)
J=I...,N
F
2

---

Under review as a conference paper at ICLR 2023
where & is the CDF of the standard Gaussian distribution. In practice, OOD actions have higher variance on Q-value estimates compared to ID (Figure Sap. Thus, with increased N we strengthen value penalization for OOD actions, inducing conservatism. As will show in Section 5.2 this effect can be amplified by scaling batch instead of ensemble size.
3 How LONG DOES IT TAKE FOR Q-ENSEMBLE TO CONVERGE?
Hours
EDAC (Ensemble) Score = 93.4 2.72 Hours
SAC-N (Ensemble) Score 92.6 2.63 Hours
COL Score 78.1 0.69 Hours
IQL Score 79.7 0.48 Hours
TD3+BC Score 76.4 0.21 Hours
Algorithm
Figure 2: The convergence time of popular deep offline RL algorithms 9 different D4RL locomo- tion datasets (Fu et al., 2020). We consider algorithm convergence similar to Reid et al. (2022). i.c., marking convergence the point of achieving results similar to the ones reported in performance tables. White boxes denote mean values (which are also illustrated on the x-axis). Black diamonds denote samples. Note that the convergence time of Qeensemble based methods is significantly higher. Raw values can be found in the Appendix A.6
To demonstrate the inflated training duration of Q-ensemble methods, we start with an analysis of their convergence time compared to other deep offline RL algorithms. Here, show that while these methods consistently achieve state-of-the art performance across many datasets, the time it takes to obtain such results is significantly longer when compared their closest competitors.
While not a common practice to report training times or convergence speed, some authors carefully analyzed these characteristics of their newly proposed algorithms (Fujimoto & Gu Reid et al. 2022). There are two prevailing approaches for this analysis. First is to study the total training time for fixed number of training steps or the speed per training step (Fujimoto & Gu 2021 An et al. 202T Second approach, such as the one taken by Reid et al. (2022), is to measure the convergence time using a relative wall-clock time to achieve the results reported in the performance tables.
202T
Measuring the total training time for fixed number of steps can be considered straightforward approach. However, it does not take into account that some algorithms may actually require smaller (or bigger) number of learning steps. Therefore, in this paper, we choose the second approach, which involves measuring the relative clock-time until convergence. Similar to Reid et al. (2022), we deem an algorithm to converge when its evaluation (across multiple seeds) achieves a normalized score within two points or higher of the one reported for the corresponding method in the performance table.
We carefully reimplemented all the baselines in same codebase and re-run the experiments to make sure that they are executed on the same hardware (Tesla 100) in order for the relative-clock time to be comparable. The results depicted in Figure 2 One efficient compared to their competitors, and their convergence times do not exceed two hours even in
that IQL. and TD3+BC highly
We derive this value as mean standard deviation typically observed when evaluating checkpoints across random seeds, e.g., Table

---

Under review as a conference paper at ICLR 2023
the worst case scenarios. This efficiency comes from the fact that these methods do not take long for one training step (Fujimoto & Gul [202]] Kostrikov et al.] (021). However, while being fast to train, they severely underperform when compared to the Q-ensemble based methods (SAC-N, EDAC).
Unfortunately, this improvement comes at a cost. For example, SAC-N median convergence time is under two hours (2-3x times longer than for IQL or TD3+BC), but some datasets may require up to nine hours of training. This is due to its usage of a high number of critics, which requires more time for both forward and backward passes. Notably, EDAC was specifically designed to avoid using large number of critics, reporting smaller computational costs per training step (An et al., 2021) and lower memory consumption. However, as can be seen in Figure 2 its convergence time is still on par with the SAC-N algorithm. As we discussed earlier, some algorithms might require a higher number of training iterations to converge, which is exactly the case for the EDAC algorithm.
LARGE BATCH SOFT ACTOR-CRITIC
One common approach for reducing training time of deep learning models is to large-batch optimization (Hoffer et al. 2017 You et al., 2017]] 2019). In this section, we investigate a similar line of reasoning for deep ORL and propose several adjustments to the SAC-N algorithm (An al. 2021) in order to leverage large mini-batch sizes:
1. Scale mini-batch Instead of using significantly larger number of crities, is done in SAC-N, we instead greatly increase the batch size from the typically used 256 state-action-reward tuples to 10k. Note that in the case of commonly employed D4RL datasets and actor-critic architectures, this increase does not require using more GPU devices and can be executed on the single-GPU setup. While higher batch sizes are also viable, we will demonstrate in Section 5 that this value sufficient, and does not result in over-conservativeness.
Learning rate square root scaling In other research areas, it was often observed that a simple batch size increase can be harmful to the performance, and other additions are needed (Hoffer et al.) 2017). The learning rate adjustment one such modification. Here, we use the Adam optimizer (Kingma & Ba 2014), fixing the learning rate derived with formula similar to Krizhevsky (014) also known as "square root scaling":
Batch Size
learning rale = base learning rale V Base Batch Size
where we take both base learning rale and base batch size to be equal to the values used in the SAC-N algorithm. Note that they are the same across all datasets. The specific values can be found in the Appendix A.5 Unlike Hoffer et al. (2017), we do not use warm-up stage.
We refer to the described modifications as Large-Bater Batch SAC (LB-SAC Overall, the resulting approach is equivalent to the SAC-N algorithm, but with carefully adjusted values which are typically considered hyperparameters. FFiuure Tsmmmarizes the distinctive characteristics of LB-SAC and the recently proposed deep offline Q-ensemble based algorithms.
EXPERIMENTS
In this section, we present the empirical results of comparing LB-SAC with other Q-ensemble methods. We demonstrate that LB-SAC significantly improves convergence time while matching the final performance of other methods, and then analyze what contributes to such performance.
5.1 EVALUATION ON THE D4RL MuJoCo GYM TASKS
We run our experiments on the commonly used MuloCo locomotion subset of the D4RL benchmark (Fu et al. 2020): Hopper, HalfCheetah, and Walker2D. Similar to the experiments conducted in Section
we use the same GPU device and codebase for all the comparisons.
'One may rightfully point out that, while we work in single- GPU setup, we rely on modern computational devices such as A100 with large memory capabilities. However, as we will demonstrate in the next section, the memory consumption D4RI. datasets does not exceed 5GB of video memory in the worst-case scenarios, i.e., making it possible utilize devices with less computation power.
single-GPU
4

---

Under review as a conference paper at ICLR 2023
LB-SAC Normalized Scores First, we report the final scores of the introduced approach. The results are illustrated in Table [] We see that the resulting scores outperform the original SAC-N algorithm and match the EDAC scores. Moreover, when comparing the whole suit of locomotion datasets, the average final performance is above both SAC-N and EDACC results (see Table 3 in the Appendix). We highlight this result, it is common observation in large-hatch optimization that naive learning rate adjustments typically lead to learning process degradation, making adaptive approaches or warm-up stage necessary (Hoffer al. 2017 You 2017 2019). Evidently, this kind of treatment can be bypassed in the ORL setting for the SAC-N algorithm.
Table I: Final normalized scores on D4RL Gym tasks, averaged over 4 random seeds. LB-SAC out- performs SAC-N and attains a similar performance to the EDAC algorithm while being considerably faster to converge as depicted in Figure For ensembles, we provide additional results on more datasets in the Appendix A.4
Task Name
TD3+BC
IQL
CQL
SAC-N
EDAC
LB-SAC
srrrchetaa--mddinn
48.1
48.3
47.0
67.5+1.2
65.9+0.6
11.5+1.2
halfcheetah- hanireettt--iiium-----
mwdi-m--dxppppeeeeee expert
90.7
94.5
95.9
107.1+2.0
106.3+0.6
109.1+2.6
44.8
43.5
45.1
63.9
61.3
64.3+0.7
hopper-medium
60.3
62.7
64.9
100.3:10.3
101.6:0.66
100.7:15.2
repper-mediuumeeppet peapper-ediumm-rppeee
101.1
106.2
93.8
110.1
110.7.0.11
110.9110.4
64.4
84.5
87.6
101.800.5
101.0
101.6+1.0
walker2d-medium
82.7
84.0
80.3
87.9+0.2
92.5++0.8
9..1+1.4
walker2d-medduu--eppet
110.0
111.6
109.6
116.7+00.4
114.7+00.9
113.443.4
walker-dd-mcdiuu--relay
85.6
82.5
79.2
78.7+00.7
87.1+2.33
79.9+0.44
Average
76.4
79.7
78.1
92.6
93.4
93.4
LB-SAC Convergence Time Second, we study the convergence time a similar fashion to Section 3 The results are depicted in Figure 3 One can that the training times of LB-SAC are less than those of both EDAC and SAC-N, outperforming them in terms average and the worst convergence times. Moreover, LB-SAC
outperforms CQL in the worst-case scenario, achieving commensurate average convergence time. This improvement comes from both (1) the usage of low number of critics, similar to EDAC (see Figure 4) and (2) improved ensemble convergence rates, which demonstrate further in Section 5.2 Additionally, we report convergence times for different return thresholds on all D4RL Gym datasets the Appendix A.4 FFiuure 13) confirming faster convergence each of them.
LB-SAC Memory Consumption Memory consumption an obvious caveat of using large batch sizes. Here, we report the worst-case memory requirements for all of the Q-ensemble algorithms. Table 2) shows that utiliza- tion of large batches can indeed lead to considerable increase in memory usage. However, this still makes it possible to employ LB-SAC even using single-GPU setups on devices with moderate computational power, requiring around 5GB of memory in the worst-case scenarios. Ad- ditionally, we report memory usage for high dimensional Ant task in the Appendix A.4 Table 5).
Table 2: Worst-case number of critics and memory consumption for D4RL lo- comotion datasets.
Crities (Quantity)
GPU Mem.
(GB)
SAC-N
500
5.1
EDAC
50
1.8
LB-SAC
50
5.4
5.2 BIGGER BATCH SIZES PENALIZE OOD ACTIONS FASTER
As argued in An et al. 2022 the major factor contributing to the success of Q-ensemble based methods in deep ORL setups is the penalization of actions based on prediction confidences. This achieved by optimizing the lower bound the value function. Both increasing the number of critics and diversifying their outputs enhanced performance (An et al., 202T). On the other hand, unlike SAC-N, the proposed LB-SAC method does not rely large amount of crities (see Figure 4), additional optimization
improve prediction confidence, which in turn leads to
5

---

Under review as a conference paper at ICLR 2023
Hours
EDAC (Ensemble) Score 93.4 2.72 Hours
SAC-N (Ensemble) Soore 92.6 2.63 Hours
COL Score 78.1 0.69 Hours
LB-SAC (Ensemble)
IQL Score 79.7 0:48 Hours
TD3+BC Score 78.4 0,21 Hour
Score 93.4 0.68 Hours
Algorithm
Figure 3: The convergence time of LB-SAC in comparison to other deep offline RI. algorithms different D4RL locomotion datasets (Fu et al. 2020). We consider an algorithm to converge similar (Reid al. 2022), i.e., marking convergence the point of achieving results similar those reported in performance tables. White boxes denote mean values (which are also illustrated on the x-axis). Black diamonds denote samples. LB-SAC shows significant improvement distribution-vin in comparison to both SAC-N and EDAC, notably performing better on the worst-cass dataset even when compared the ensemble-free CQL algorithm. Raw values can be found in the Appendix A.6
HarCheetan
Welker0
10
AMD
Agarlive
AAG
EDAC
reck
Figure 4: The number of Q-ensembles (N) used to achieve the performance reported in Table and the convergence times in Figure 3 LB-SAC allows to use a smaller number of members without the need introduce additional optimization term. Note that, for both SAC-N and EDAC, we used the number of critics described in the appendix of the original paper (An et al. 202T).
term for diversification as required by EDAC, while matching their performance with an improved convergence rate.
To explain the empirical success of LB-SAC, we vary the batch size and analyze the learning dynamics in a similar fashion An al. (202T) both in terms of out-of-distribution penalization
and the resulting conservativeness, which is currently known to be the main observable property of successful ORI. algorithms (Kumar al. 2020 Rezaeifar et al. 2022). To do so, we keep track of two values. First, we calculate the standard deviation of the q-values for both random and behavioral policies and record relation throughout the learning process. The former is often utilized as the
producing out-of distiibbbtttto actions (Kumar et al. 2020 An al. 2021). Second, we estimate the distance between the actions chosen by the learned policy to the in mn-datsee ones for tracking how conservative the resulting policies are.
The learning curves are depicted in Figure 3 faster growth of the standard deviation relation between out-of-distribution and in-dataset actions,
We observe that increasing the batch size results in

---

Under review as a conference paper at ICLR 2023
which can produce stronger penalization uut-of-distribuut actions earlier in the training process. This, in turn, leads to an elevated level of conservativeness, as demonstrated in Figure 5b)
000 10 7
Su
Trarning Stege
Thurzny Stete
Transing Stepe
(a) Standard Deviation Relation (b) Distance In-Dataset Actions
(c) Normalized Score
Figure 5: Effects of increasing the batch size. (a) Depicts the relation of standard deviation for random actions dataset actions (b) Plots the distance the dataset actions from the learned policy (c) Shows averaged normalized score. The number of critics is fixed to 10 for all the batch sizes. Result averaged over seeds. For the brevity of presentation, the results are reported for the hopper-medium-v2 dataset, however, similar behavior occurs all other datasets as well.
Despite the fact that Figure 5) deppccs the effect increasing batch size on standard deviation ratio between random and dataset actions, the true nature of this outcome remains not completely clear. One possible explanation can be based on a simple empirical observation confirmed by our practice (Section 2 and An al. (202I) analysis: with further training Q-ensemble becomes more and more conservative (Figure Sa). While standard deviation for dataset actions stabilizes at some value, for random actions continues to grow for very long time. Hence, one might hypothesize that for different ensemble sizes N > N', both will achieve some prespecified level of conservatism, but for a smaller one will take a lot more training time. Therefore, since by increasing the batch we also increase the convergence rate, we should observe that with equal ensemble size LB-SAC will reach same penalization values but sooner.
To test the proposed hypothesis, conduct experiment two datasets, fixing number of critics, only scaling batch size and learning rate as described in Section 4 and training SAC-N for 10 million instead of million training steps. One can see (Fiuuee 6) that larger batches indeed only increase convergence speed, as SAC-N can achieve same standard deviation values but much further during training. Thus, most tasks, convergence compensates for the reduction in diversity, allowing us to remain at the same or higher level of penalization growth SAC-N with larger ensemble.
reduce the ensemble size for LB-SAC, since accelerated
1R Axd
1.8 3R 30
1045803
LB353G4
Ix
(a) halfcheetah-mediun--22
(b)
Figure 6: Both LB-SAC and SAC-N with ensemble size achieve similar standard deviation ratio between random and dataset actions, but with larger batches it happens a lot faster. This allows us to reduce the size of the ensemble for most environments while leaving the growth rate comparable to the SAC-N. Results averaged over seeds. All hyperparameters except batch size and learning rate fixed for fair comparison. Notice that we train SAC-N for 10 million steps.

---

Under review as a conference paper at ICLR 2023
ABLATIONS
6.1
How LARGE SHOULD THE BATCH BE?
1983
140
AVRESS Nomallan SCOTE
0000 1D Sid Raii
1010
2012
Training Steps
Trabting Stops
Traininn Stepa
(a) Standard Deviation Relation (b) Distance to In-Dataset Actions
(c) Normalized Score
Figure 7: Scaling batch size further leads diminishing returns. (a) Depicts the relation of standard deviation for random actions to dataset actions (b) Plots the distance between dataset actions and the learned policy (c) Shows the averaged normalized score. The number of critics is fixed for all batch sizes. Results were averaged over 4 seeds. For presentation brevity, the results are reported for the
dataset. However, similar behavior occurs on all other datasets as well.
Observing that the increased mini-batch size improves the convergence, natural questions to ask is whether should we scale it even bigger. To answer this, we run LB-SAC with batch-sizes up to 40k.
We analyze the learning dynamics in a similar way Section 5.2 The learning curves can be found in the (iguue 7 Overall, we see that further increase leads to diminishing returns on both penalization and the resulting score. For example. example,Figure 7a shows that increasing the batch further leads an improved penalization of the out-of out-of-distributte di-bstrribution actions. However, this improvement becomes less pronounced as we go from 20k 40k. It may interesting to note that the normalized scores for typically utilized batch size of 256 start to rise after certain level of penalization, and conservatism (MSE is around 0.3) is achieved. Evidently, the utilization of larger batch sizes helps achieve this level significantly earlier in the learning process.
6.2 Is JUST THE LEARNING RATE?
To illustrate that the attained convergence rates benefit from an increased batch size and not merely from an elevated learning rate, conduct ablation, where we leave size of the batch equivalent to the one used in the SAC-N algorithm (B 256), but scale the learning rate. The results are depicted in Figure S
Loaming Scaling
140
1.20
100
Normalized
LB-SAC 1B-14, LI--0/011)
0.0033
130000
300000
320000
400000
Training Steps
Figure 8: The average normalized score. The improvement comes from both large batch size and adjusted learning rate. Fixing batch size the values typically used for SAC-N and scaling the learning rate does not help. The results are averaged over 4 seeds. For presentation brevity, the results reported for the walker2d deeedddmmiiim-eeeeeee dataset.
8

---

Under review as a conference paper at ICLR 2023
Unsurprisingly, scaling learning rate without doing the same for batch size does not result in effective policies and stagnates at some point during the training process. This can be considered expected, as the hyperparameters for the base SAC algorithm were extensively tuned by extensive number of works on off- ool-policy RI. and seem to adequately transfer to the offline setting, as described in An et al. (2021)
6.3
Do LAYER-ADAPTIN OPTIMIZERS HELP?
Average Normelized Score
OOD to ID Sid Ratio
LARS
50
60
2
30
LAMB
20
100000
2000000
300000
600030
500000
100000
200003
500000
402000
500000
Training Steps
Training Steps
(a) Standard Deviation Relation
(h) Normalized Score
Figure 9: Using layer-adaptive optimizers leads either to a similar performance or degrades the learning process. (a) Depicts the relation of standard deviation for random actions dataset actions (b) Shows the averaged normalized score. The number of critics fixed for all optimizers. The results are averaged over 4 seeds. For presentation brevity, the results reported for the halfcheetah- medium-v2 dataset.
While we settled naive approach for trgge-bacch optimization in the setting of deep offline RL, one may wonder whether using more established and sophisticated optimizers may work as well. Here, we take look at commonly employed LARS (You et al. 2017) and LAMB (You al. 2019) optimizers reporting learning dynamics similar to the previous sections. The resulting curves can be found in Figure
We see that the LAMB optimizer could not succeed in the training process diverging with more learning steps. On the other hand, LARS behaves very similar to the proposed adjustments. While by no means suggest that adaptive learning rates are not useful for offline RL, the straight adoption of established large-batch optimizers for typically used neural network architecture (linear layers with activations) in the context of locomotion tasks does not bring much benefits. Seemingly, these methods may require more hyperparameter tuning, different learning rate scaling rules or warm-up schedule as opposed to the simple "square root scaling".
CONCLUSION
In this work, we demonstrated how the overlooked use of large-batch optimization can be leveraged in the setting of deep offline RL. We showed that naive adjustment of the learning rate in the large-batch optimization setting is sufficient to significantly reduce the training times of methods based on Q-ensembles (4x average) without the need use adaptive learning rates.
Morcover, we empirically illustrated that the use of large batch sizes leads to an increased penalization of -iitrii-iitiiiioonnn actions, making estimates an ensemble or an additional optimization term. We hope that this work may serve as a starting point for further investigation of large-batch optimization in the setting of deep offline RL.
efficient replacement for increased number of q- v-value
REFERENCES
Igor Adamski, Robert Adamski, Tomasz Grel, Adam Jędrych, Kamil Kaczmarek, and Henryk Michalewski. Distributed deep reinforcement learning: Learn how to play atari games in 21 minutes. In International conference on high performance computing, pp. 370-388. Springer, 2018.

---

Under review as a conference paper at ICLR 2023
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective offline reinforcement learning. In International Conference on Machine Learning, pp. 104-114 114. PMLR, 2020.
Arthur Allshire, Roberto Martin-Martín, Charles Lin, Shawn Manuel, Silvio Savarese, and Animesh Garg. Laser: Learning a latent action space for efficient reinforcement learning. In 202/ IEEE International Conference on Robotics and Automation (ICRA), pp. 6650-6656. IEEE, 2021.
Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-base based offline reinforcement learning with diversified q-ensemble. Advances in neural information processing systems, 34:7436-7447, 2021.
Pavlos Athanasios Apostolopoulos, Zehui Wang, Hanson Wang, Chad Zhou, Kittipat Virochsiri, Norm Zhou, and Igor L Markov. Personalization for web-based services using offline reinforcement learning. arXiv preprint arXiv:2102.05612, 2021.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 167.0:160..064450 2016.
Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhihong Deng, Animesh Garg, Peng Liu, and Zhaoran Wang. Pessimistic bootstrapping for incertainty-driven offline reinforcement learning. arXiv preprint arXiv:2202.1 /566, 2022.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680 2019.
Richard Y Chen, Szymon Sidor, Pieter Abbeel, and John Schulman. Ucb exploration via q-ensembles. arXiv preprint arXiv: /700.0000002222 2017.
Xinyue Chen, Che Wang, Zijian Zhou, and Keith Ross. Randomized ensembled double q-learning: Learning fast without a model. arXiv preprint arXiv:2101.05982, 2021.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in handful of trials using probabilistic dynamics models, Advances in neural information processing systems, 31, 2018
Kamil Ciosek, Quan Vuong, Robert Loftin, and Katja Hofmann. Better exploration with optimistic actor critic. Advances in Neural Information Processing Systems, 32, 2019.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint M0XY::2004.07219, 2020.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach offline reinforcement learning. Advances in neural information processing systems, 00:00122-20145 2021.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. International conference on machine learning, pp. 2052-2062. PMLR, 2019.
Seyed Kamyar Seyed Ghasemipour, Shixiang Shane Gu, and Ofir Nachum. Why so pessimistic? estimating uncertainties for offline rl through ensembles, and why their independence matters. arXiv preprint arXiv:2205. 13703, 2022.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learming. pp. 1861-1870. PMLR, 2018.
Takuya Hiraoka, Takahisa Imagawa, Taisei Hashimoto, Takashi Onishi, and Yoshimasa Tsuruoka. Dropout q-functions for doubly efficient reinforcement learning. arXiv preprint a 10.000334 2021.
Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generaliza- tion gap in large batch training of neural networks. Advances in neural information processing systems, 30, 2017.
10

---

Under review as a conference paper at ICLR 2023
Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado Van Hasselt, and David Silver. Distributed prioritized experience replay. arXiv preprint 1r03:v1803.000 2018.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When trust your model: Model-basee policy optimization. Advances in Neural Information Processing Systems, 32, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations, 2021.
Alex Krizhevsky, One weird trick for parallelizing convolutional neural networks. arXiv preprint arXiv:1404.5997, 2014.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1179-1109 2020.
Vladislav Kurenkov and Sergey Kolesnikov. Showing your offline reinforcement learning work: Online evaluation budget matters. International Conference on Machine Learning, pp. 11729- 11752. PMLR, 2022.
Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-ensemble trust-region policy optimization. arXiv preprint arXiv:1802.10S9) 2018.
Hang Lai, Jian Shen, Weinan Zhang, and Yong Yu. Bidirectional model-based policy optimization. In International Conference on Machine Learning, pp. 5618-5627. PMLR, 2020.
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement leaanonn In Reinforcement learning, pp. 45-73. Springer, 2012.
Kimin Lee, Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning. In International Conference Machine Learning, pp. 6131-6141, PMLR, 2021.
Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble. In Conference on Robot Learning, pp. 1702-1712. PMLR, 2022.
Nir Levine, Tom Zahavy, Daniel J Mankowitz, Aviv Tamar, and Shie Mannor. Shallow updates for deep reinforcement learning. Advances in Neural Information Processing Systems, 30, 2017.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives open problems. arXiv preprint (xXi:2005.00644) 2020.
Litian Liang, Yaosheng Xu, Stephen McAleer, Dailin Hu, Alexander Ihler, Pieter Abbeel, and Roy Fox. Reducing variance in temporal-difference value estimation via ensemble of deep networks. In International Conference on Machine Learning, pp.
PMLR, 2022.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization.
arXiv preprint
171:0:171..05101, 2017.
Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAl Dota Team. An empirical model of large-batch training, arXiv preprint 18X2:0/81..0616... 2018.
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets, arXiv preprint vrXiv:2006.09359, 2020.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. Advances neural information processing systems, 29, 2016.
Ian Osband, John Aslanides, and Albin Cassirer. Randomized prior functions for deep reinforcement learning. Advances in Neural Information Processing Systems, 31, 2018.
11

---

Under review as a conference paper at ICLR 2023
Machel Reid, Yutaro Yamada, and Shixiang Shane Gu. Can wikipedia help offline reinforcement learning? arXiv preprint arXiv:2201.12122, 2022.
Shideh Rezaeifar, Robert Dadashi, Nino Vieillard, Léonard Hussenot, Olivier Bachem, Olivier Pietquin, and Matthieu Geist. Offline reinforcement learning as anti-exploration. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 8 8106-81 14, 2022.
J. P. Royston. Algorithm 177: Expected normal order statistics (exact and approximate). Journal of the Royal Statistical Sociery. Series C (Applied Statisticcs), 31(2):161-163 1982. ISSN 00359254, 14679876. URL
jstor. wttp:/orrggssaasaeoeeee/....
Takuma Seno and Michita Imai. d3rlpy: An offline deep reinforcement learning library. arXiv preprint /tti::2//11.070000 2021.
Hassam Sheikh, Kizza Frisbee, and Mariano Phielipp. DNS: Determinantal point process based neural network sampler for ensemble reinforcement learning. Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 19731-19746. PMLR, 17-23 Jul 2022. URL https: ptttps::/prroceedingg.. mlr press/v162 wttpd///hhiihh22a..mmmm
Douglas W Soares, Acordo Certo, Telma Lima, and Deep Learning Brazil. Pulserl: Enabling offline reinforcement learning for digital marketing systems via conservative q-learning. Advances in Neural Information Processing Systems, 2nd Workshop on Offline Reinforcement Learning, 2021.
Adam Stooke and Pieter Abbeel. Accelerated methods for deep reinforcement learning, arXiv
preprint arXiv: 000.00881,.,11,,, 2018.
Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenherg. Scott Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, et al. Critic regularized regression. Advances in Neural Information Processing Systems, 33:7768-7773 2020.
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint arXiv: /708.03888. 2017.
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsich. Large batch optimization for deep learning: Training bert in 76 minutes. In International Conference on Learning Representations, 2019.
Xianyuan Zhan, Haoran Xu, Yue Zhang, Xiangyu Zhu, Honglei Yin, and Yu Zheng. Deepthermal: Combustion optimization for thermal power generating units using offline reinforcement learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pp. 4680-4688, 2022.
Wenxuan Zhou, Sujay Bajracharya, and David Held. Plas: Latent action space for offline reinforce- ment learning. In Conference on Robot Learning, pp. 1719-1735. PMLR, 2021.
12

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 RELATED WORK
Model-Free Offline RL A large portion of recently proposed deep offline RL algorithms focuses on addressing the extrapolation issue, trying to impose a certain degree of conservatism limiting the deviation of the final policy from the behavioral one. Researchers approached this problem from multiple angles. For example, Kumar et al. (2020) proposed to directly penalize duttof-distribution
actions, while Kostrikov al. (2021) avoids estimating values for out-of-sample actions completely, Others (Fujimoto & Gu 2021; Nair et al., 2020; Wang et al., (2020) put explicit constraints to stay closer to the behavioral policy either by directly making models closer to the behavioral one (Fujimoto & Gu (02T), or by re- weightiing behavioral policy actions with the estimated advantages. In addition, there are works that construct a latent policy action space and optimize models within (Zhou et al. 2021 Allshire et al.] (02T). In our work, we explore different class of methods based on mode frree uncertainty quantification (An et al. [02T], and demonstrate that it can be achieved with sufficiently large mini-batch sizes and a handful number of value estimates.
Q-Ensembles RL There has been a body of work on leveraging ensembles deep reinforcement learning (Sheikh et al., 2022; An et al. [02T Lee et al. 202T; 2022 Osband et al. 2018). For offline RL.[An An et al. (2021) demonstrated that using the soft-actor critic with large amount of value estimates leads to state-of-th the art results on D4RL benchmark. As the number of crities grows, so do the computational time, in order to remedy this issue, [An et al. (2021) further proposed to diversify the ensemble by disaligning the critics. In our work, we largely build upon these findings. However, we demonstrate that it is possible to alleviate a large number of crities by simply using large mini-batch sizes, which stabilizes the training process and speeds up convergence both in terms of training cycles and wall time.
Large Batch Optimization Unlike deep offline RL, the effect of larger mini- sizes was exten- sively studied in other areas of deep learning. This is typically referred as large batch optimization (Hoffer et al. 2017). For example, You et al. (0177) proposed to use layer-wise adaptive learning rates to train big vision models in minutes. In the Natural Language Processing field, You et al. (2019) proposed another adaptive learning rates mechanism train attention- bbased models such as BERT, making it possible to train big language models within hours instead of days. In our work, we investigate a similar line of reasoning, and demonstrate that simply increasing the mini-batch size and naively adjusting the learning rate is sufficient for obtaining a faster convergence speed while matching state-of-the the-art performance on the D4RL benchmark.
Large Batch in RL While large batches not commonly employed in online RL, their benefits were already extensively demonstrated and analyzed. For example, |Brner et al. (2019) describes large-scale setting and uses
million timesteps per update to train Dota-agent. On the other hand, in simple Atari environments, batch sizes of thousands were also found to be effective for online RL algorithms Levine et al. (2017): Horgan et al. (2018): Adamski et al. (2018); Stooke & Abbeel (2018). Notably, McCandlish et al. (2018) proposed a statistic called "ggaddent noise scaling" to predict the largest useful batch size for both supervised and online reinforcement learning problems. As for offline RL, to the best of our knowledge, we are the first to explore the effects of large mini-batch sizes and demonstrate how it can benefit the learning process of Q-ensemble methods.
A.2 LIMITATIONS AND FUTURE WORK
Multiple Evaluations In our study we extensively use the notion of convergence time by evaluating multiple checkpoints during the training process. This contradicts the penultimate limitation of pure offline RI., where one should be allowed to evaluate exactly one policy. However, in this work, we were specifically interested in studying the convergence properties, therefore requiring multiple evaluations for such an analysis. Still, the practice of multiple evaluations common in the community (Kurenkov & Kolesnikov] 2022 Seno & Imail 2021) since the proper approach to compare offline RL methods is still an open question.
Overfitting It was noted in Seno & Imai (2021) that the best checkpoint across the training process significantly outperforms the final performance typically reported in deep offline RL papers. This is either attributed to the noisy learning process or some form of overfitting, as it can be often observed
13

---

Under review as a conference paper at ICLR 2023
that the performance starts to deteriorate with more learning steps (Seno & Imail 2021 We observed
a similar behavior for large-batch optimization on some datasets (see Section 6.1 Figure 7 for 40k
batch size). We believe this is interesting direction for future work (including large-batch settings), since the methodological apparatus for studying the properties of overfitting is yet to be developed for the offline RL setting.
LBO for Other Offline RL Algorithms Although the sole focus of our work was on studying large-batch optimization for Q-ensemble methods demonstrate the benefits it can bring (e.g. reduced number of critics or improved convergence time), an intriguing line of investigation would be the application of big batch sizes to other deep offline RL algorithms which could potentially also lead to improved convergence.
Layeradaptive Optimizers for Offline RL While we conducted a preliminary of experiments on layer-adaptive optimizers You et al., 2017 2019) in the Appendix 6.3] more investigation is certainly needed in this direction. Since we run our experiments on D4RL benchmark and locomotion environments, we believe these optimizers may find their use in the context of different tasks typically employing other types of architecture (e.g. convolutional neural networks, recurrent networks, and transformers).
A.3 ON LAYER NORMALIZATION
While scaling batch size significantly improves convergence speed and penalization of OOD actions, there were some difficult tasks where such naive approach could not help much. As showed in I scaling has its limits and can lead to diminishing returns, where at some point (around 10k batch size) the std ratio (as well penalization) will stop growing. From this point, increasing batches further does not always provide much benefit, since without sufficient level of penalization for given task even with faster convergence rate the agent may not converge or the training can be unstable. With a default MLP critic architecture, the only option left is to make the ensemble bigger use diversity loss from EDAC. In our preliminary experiments, we found that increasing the ensemble size could remedy this issue but led to longer training times.
After more experimentation, we found that the use of Layer Normalization (Ba et al. 2016) completely reduces diminishing returns all batch sizes we considered (Figure 10). One can see in Figure 10 the scaling tendency is preserved in opposition to the results we observed without layer normalization (Figure
(a) Standard Deviation Relation (b) Distance to In-Dataset Actions
(c) Normalized Score
Figure 10: Layer normalization reduces the effects of diminishing returns. (a) Depicts the relation of standard deviation for random actions to dataset actions (b) Plots the distance to the dataset actions from the learned policy (c) Shows the averaged normalized score. The number of critics fixed for all batch sizes. Results were averaged over 4 seeds. malfcheetah-meddim--y is the dataset used.
The most illustrative example the hopper-medium--2 dataset, which can be considered the hardest task among all others. To solve it, SAC-N uses 500 critics while EDAC uses 50 (An et al. 2021). In our experiments, we were unable to get competitive results with bigger batches and 50 critics, and increasing ensemble size was not an option due to efficiency reasons. However, with additional layer normalization for critics (after each layer and before nonlinearity). we were able to get strong results using only 25 critics due to increased penalization. Note that more simple tasks (Figure 10). such a boost can lead to overconservative policies and lower scores as a result (almost 10 points worse on halfcheetah-mediume
14

---

Under review as a conference paper at ICLR 2023
A.4 RESULTS ON ADDITIONAL DATASETS
Table 3:
Final normalized scores all D4RL Gym tasks, averaged over 4 random seeds. LB- SAC attains a better average performance over both the SAC-N and EDAC algorithm while being considerably faster to converge depicted in Figure 11
Task Name
SAC-N
EDAC
LB-SAC
halfchhetaah-aadde
28.0=:..
28.41:.
31.11:8
calfcheetahh-medine
6..5=1.2
65.9
71,51:2
hallichettah-exper
105.2-2.6
106.813.4
109.0:2.8
halfcheetah-ne
halffchectah-mcdiu--nyyy
medium-expert
107.1=2.0
106.3+0.6
109.1
haffchoctah-mmcii--rpppp medium-replay
halfcheetah-
63.9--0.8
61,3++19
64.3+0.7
halfchcetah-fuie -epllrl
84.5+112
84.6+0.9
86.6+0.5
hopper-random
31.3.1..0
25.311004
31.4.0.0
hopper-medium
100.311.33
101.610.6
100.715.2
hopper-expert
110.3=0.3
110.1
110.0#0..1
hopper-medium- expert
110.1=0.3
110.7#0.1
110.9
hopper-mediumg replay
101.8=0.5
101.0
101.6110
hopper-full-repla
102.9+0.3
105.4
108
walker2d-rande random
21.7=0.0
16.647.0
21.610.1
87,9=0.2
92.5
1..1
walker2d-expert
107.4=2.4
115.1
107.61:0.4
malkkerdd-medium-exppp
116.7+00.4
114.740.9
113.4+3.4
78.7+00.7
87.1+2.3
79.9+
malker2d-full-replhe
94.6+t0.5
99.8+00.7
109.1
Average
84.5
85.2
86.4
Table 4: Normalized maximum scores on all D4RL Gym tasks, averaged over 4 random seeds.
LB-SAC attains better average performance over both the SAC-N and EDAC algorithms.
Task Name
SAC-N
EDAC
LB-SAC
halfchectah-random
29.7+11.
29.941..6
34.1+:144
halfcheetah-mmedimm
71.0
68.242.3
7..0+
halfcheetah-eexee
108.5=1.0
108.240.7
110.2+1.1
halfcheetah- medium- expert
109.1-2.1
107.011..
110.6:2..7
halfcheetah-meddiue medium--rppaay
65.8-1..9
65.0112.1
66.010.7
halfcheetah- fuffihhcttah-flllrrlaayy full-replay
85.9+0.4
85.6+0..5
87.6+0088
hopper-random
32.0.1.6
32,910.2
31.410.0
hopper-medium
101.011.66
102.710.2
103.810.00
hopper-expert
110.610.5
111.10000
110.110.1
hopper-medium expert
110.4=0.4
111.4002
110.9+0.4
hopper-medium--ee replay
102.9=0.8
102.640.6
104.
hopper-full-reples
107.9+0.3
107.940.4
109.3+0.3
walker2d-random
21.940.00
22.0.10.3
21.8--0.0
walker2d-medium
88.7-0.5
94.6.-1.5
11.
walker2d-exppe -e-peett
110.610.6
116.740.7
10,,810,9
walker2d-medium-ee expert
116.0=0.8
115.3+0.2
115.5+1.1
walker2d-medius medium-repllay
83.3+22.9
87.812.3
88.1
malker22d-ull-rrplaa
97.8+0088
100.8+00.5
110.4+5.00
Average
86.2
87.2
88.3
15

---

Under review as a conference paper at ICLR 2023
12
11
10
9
Hours
2
LB-SAC Score 86.4 0.82 Hours
SAC-N Score = 64.5 3.07 Hours
EDAC Score 86.2 2.77 Hours
Algorithm
Figure 11: The convergence time LB-SAC in comparison to SAC-N EDAC on all D4RL Gym datasets (Fu al. 2020). We consider algorithm converge similar (Reid et al. 2022). i.e., marking convergence at the point of achieving results similar to those reported in performance tables. White boxes denote mean values (which are also illustrated on the x-axis). Black diamonds denote samples. LB-SAC shows significant improvement distribution-wsn wiie in comparison to both SAC-N and EDAC, notably performing better on the worst-case dataset.
SACA
Agartive
193
Agorttr
RIMG
Datasel
Datase!
Datasel
Figure 12: The number of Q-ensembles (N) used to achieve the performance reported Table and the convergence times in Figure 11 LB-SAC makes it possible to use a smaller number of members without the need introduce an additional optimization term. Note that, for both SAC-N and EDAC, used the number of critics described in the appendix of the original paper (An et al. 2021.
Table 5: Q-ensemble methods evaluation on ant- mitt-meddumv v2 with large action and observation spaces (119 dimensions total). Results are averaged over seeds. As one can see, LB-SAC can successfully scale to high dimensional problem achieving higher score faster. Note that the EDAC's memory consumption more than the one required by the SAC-N with higher number of critics. This is due to the need to compute the gradient for each ensemble member with respect to the action space.
Method
Score
Critics
Time (Min)
GPU Mem. (GB)
SAC-N
121.5+1.11
50
116
2.47
EDAC
120.4+1..5
10
122
2.85
LB-SAC
126.1:1.0
21
3.27
16

---

Under review as a conference paper at ICLR 2023
1.0
Percentage of the final score
0.9
0.8
0.7
0.6
0.5
0.4
0.3
LB-SAC
0.2
SAC-N
EDAC
0.1
25
50
75
100
125
150
175
Minutes
Figure 13: Relative wall-clock time in minutes to achieve specified percentage of final score for ensemble algorithms. Results are averaged over all D4RL Gym datasets (Fu et al. 2020) with 4 seeds each. As one see, LB-SAC converges faster any given return threshold. Note that LB LB-SAC achieves competitive average final score (see Table 3).
A.5 HYPERPARAMETERS
Table 6: SAC-N, LB-SAC, EDAC shared hyperparameters
Parameter
Value
optimizer
AdamW (oshchilov & Hutter 2017
tau (7)
5e-3 (5e-4 on walker2ddexpert-222
hidden dim (all networks)
256
hidden layers (all networks)
3
target entropy
-action dim
gamma (?)
0.99
nonlinearity
ReLU
Table 7: Algorithm specific hyperparameters. For LB-SAC algorithm on all environments used SAC-N batch size and learning rate as base values for learning rate scaling (precise formula described Equation 2).
Algorithm
Batch size
Learning rate
SAC-N
256
3e-4
EDAC
256
3e-4
LB-SAC
10000
1.8e-3
17

---

Under review as a conference paper at ICLR 2023
Table Environment specific hyperparameters.
Task Name
SAC-N (N)
LB-SAC (N, LayerNorm)
EDAC (N, 7))
halfcheetab-random
10
2, False
10, 0.0
halfcheetah-medium
10
4, False
10, 1.0
halfcheetah-expeee
10
False
10, 1.0
medeut-expeett
10
False
10, 5.0
10
4, False
10, 1.0
halfcheetah- rupl-p
10
4. False
10. 1.0
hopper-random
500
25, Falsc
50, 1.0
hopper-medium
500
25, Truc
50, 1.0
hopper-expert
500
50, Falsc
50, 1.0
copper--mccum-eepeet
200
40, False
50, 1.0
hupper-medium-replyy
200
False
50,
hopper-full-replay
200
25, False
50, 1.0
walker2d-random
20
15, False
10, 1.0
walker2d-medium
20
10, False
10, 1.0
2alker2d-expert
100
30, False
10, 5.0
walker2dd-mciium-erry
20
10, False
1.,
malker2d-modium-rrqpee
10, False
10, 1.0
walker2d -epleray
20
4. False
10. 1.0
ant-medium
50
10, True
10, 5.0
A.6
CONVERGENCE TIME
Table 9: Convergence times in minutes for each D4RL Gym dataset.
Task Name
TD3+BC
IQL
CQL
SAC-N
EDAC
LB-SAC
halfchectah-randee
18.0
18.0
halfcheetah-mmdiium
6.5
20.6
84.0
halfchectah-expertt
435.0
563.0
50.0
89.5
13.0
289.7
336.0
537.0
60.0
balfcheetah-mediin--eee
89.5
3.2
24.4
17.0
23,0
2.0
Salfchheeah-full-reeee
164.0
hopper-random
7.0
3.0
19,0
hopper-medium
10.7
50,5
3.7
508.0
105.0
115.0
hopper-expert
853.0
276.0
276.0
hopper-meedium-expert
10.2
51.0
58.3
219.0
408.0
93.0
hopper--eeduum-rellay
11.1
72.2
112.9
$4.0
44,0
16.0
hopper-full-reee replay
60.0
38.0
16.0
walker2d -aaedd
132.0
73.0
26.0
walker2d-medium
16.1
11.4
24.4
39.0
115.0
15.0
walker2d-expert
267.0
330.0
116.0
walker24-mediuu-eexpe
9.3
19.5
48.9
216.0
125.0
48.0
10.2
10.3
26.3
6.0
33,0
4,0
walker2d-full full-r
41.0
58.0
26.0
Average
12.6
26.4
41.6
166.5
184.2
49.5
18


Reviews:
[{"review_id": "zYoc_3aTc0T", "reviewer": "Reviewer_Z3P6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies if large batch size and learning rate could improve the convergence speed of ensemble-based offline RL methods. Based on SAC-N, the authors found that both large batch size and large learning rate contribute to a faster training. ", "summary_of_the_review": "I'm inclined to reject the paper because the it seems to be a straightforward adoption of well-established and generic training speedup approaches for training ensemble-based offline RL methods. I'm not convinced that this paper solves a problem unique to offline RL or generates any novel insights on how to better design efficient offline RL methods.", "strength_and_weaknesses": "### Strength\nResults show that increasing the batch size and the learning rate can lead to a faster training.\n\n### Weaknesses\n* The novelty of this paper seems limited. \n* Clarity needs improvement. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "gx5ru2cf_kz", "reviewer": "Reviewer_D2xN", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies how increasing batch size and adjusting learning in SAC- N [1] can improve the training time in terms of \"convergence\" time. To adjust learning rate, they use square root scaling formula which is mostly based on previous works, To evaluate their method, they only consider a small subset of environments (i.e halfcheetah-medium-*, hopper-medium-*, and walker2d-medium-*). Finally, their studies show that their proposed approach helps with \"convergence\" time but doesn't improve final performance.  \n\n\n\n[1] Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-based offline reinforcement learning with diversified q-ensemble. Advances in neural information processing systems, 34:7436–7447, 2021", "summary_of_the_review": "I appreciate the authors for studying the effect of batch size and learning rate in batch reinforcement learning.That being said,the scope of this paper is very limited and only applicable to a particular method (SAC-N). In addition, they use a limited number of environments to study their proposed approaches. Finally, the proposed method does't lead to significant improvement in the results and method like EDAC already shows better results in the similar setting. Hence, the paper doesn't meet ICLR acceptance bar. ", "strength_and_weaknesses": "While studying different supervised learning training techniques in RL is an interesting direction and can bring in better training practices into reinforcement learning, there are various concerns with this work.\n\nStrength:\n- This method helps to decrease training time of SAC-N.\n- This work applies multiple supervised learning techniques to SAC-N.\n\nWeaknesses:\n- This paper only focuses on SAC-N and doesn't study any other offline methods. Thus this work is tightly coupled to a very specific method and it is hard to draw general conclusions from these results.\n\n- The definition of convergence time seems arbitrary to me. This is an important issue as the main claim of this paper is improving training time based on the \"convergence\" time. I understand that this paper follows [2] to define this metric. But the setting of this paper seems different from them as [2] works with a large language model and having a metric like that might make sense [?].\n\n- This paper only considers a small subset of D4RL tasks which doesn't help the paper either. Also, halfcheetah, walker2d, and hopper all have very small state and action spaces whereas environments like FrankaKitchen and Adroit in D4RL have larger state and action space. Hence increasing batch sizes like in MuJoCo experiment might not be possible in those other environments as the network can't fit in a single gpu.\n   \n- The fact that results does't change that much by reducing the number of Qs in SAC-N from 500 to 50 tells that Qs are not that diverse. I believe that is what EDAC studies. Plotting means and standard deviation of Qs during training can help to better understand this. \n\n- EDAC results are better in both terms of training time and final results than SAC-N. Also it has better memory consumption per table 2 results in this paper. Thus I am not sure why one should still consider SAC-N. \n\nminor comments:\nDiamond and white squares are not defined in figure 1. \n\n\n[2] Can Wikipedia Help Offline Reinforcement Learning? Machel Reid, Yutaro Yamada and Shixiang Shane Gu\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "QcK7RaQzt3U", "reviewer": "Reviewer_Fv4L", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors investigate the effect of large batch size in training the SAC algorithm. The result shows that a large batch size allows the algorithm to converge faster and introduces a form of conservatism (shown through the standard deviation ratio between ID vs. OOD state/action). This effect cannot be achieved by layer-wise optimization or learning rate alone.", "summary_of_the_review": "The paper shows that large batch size empirically helps with conservatism but offers no explanation of why this phenomenon occurs, and how it impacts the RL algorithm (except for some simple statistics that are used to describe the phenomenon). All the observations/experiments are superficial. If the authors fail to offer more significant insight and cannot design an experiment that reveals why larger batch size leads to more conservatism, this paper does not offer enough contribution to the research community.", "strength_and_weaknesses": "Strength:\nThe overall quality of the paper is high. The paper is well-written, and there are some ablation experiments to support the main argument of the paper.\n\nWeakness:\nMy main criticism of this paper, which ultimately led me to a borderline reject decision, is that this paper treats RL as if it were supervised learning. If I replace SAC with ResNet in every part of this paper, and D4RL with ImageNet or CIFAR-10, only section 4.2.1 (Bigger Batch Size Penalize OOD Actions) would not make sense, but all the rest would still be fine. For example, 4.2.2 (batch size ablation), 4.2.3. (learning rate), 4.3 (a better optimizer) all offer no explanation from the RL perspective.\n\nSection 4.2.1 offers an explanation of why the large batch size would help, and it says:\n> One possible explanation for why earlier penalization might be helpful is for prevention of the iterative error exploitation (which is prominent in iterative algorithms and extensively discussed in Brandfonbrener et al. (2021)), as it allows to soften the overestimation bias early in the training process making its propagation less severe.\n\nIn the TD3/SAC algorithm, critics are updated using the bellman-backup operator:\n\n$$L = \\mathbb{E}\\_{\\tau \\sim D}[(r\\_{t+1} + \\gamma \\min_j Q_{\\theta_j}(s_{t+1}, \\tilde a + \\epsilon) - Q_{\\theta_i}(s_t, a_t))^2]$$\n\nThis is a moving target, which is influenced by how well the Q-function is trained.\nOne can argue, with larger batch size, bellman value propagation is more random at the start -- because you essentially fit $r$ with a random noise outputted by an untrained Q function. So it's possible that large batch size helps reduce the \"early commitment\" issue.\nTherefore, a simple experiment to understand why large batch size helps is to just add Gaussian noise to the critics' target during early training and anneal this noise to see if you can recreate this effect of large batch size.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "DjyZaMKmqPW", "reviewer": "Reviewer_mkHX", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tries to use a large batch size to benefit Offline RL training. The authors try to scale up batch size and square root scaling for the learning rate to accelerate the Offline RL training. The experimental results illustrate that Large-Batch Training can efficiently accelerate offline rl training and achieve about 2x speed up. ", "summary_of_the_review": "NA", "strength_and_weaknesses": "Strength:\n\nThis paper focuses on an interesting area. Offline RL has achieved great attention in the last few years. \nThis paper verifies the performance of proposed method in various tasks, such as the subsets of D4RL.  \n\n\nWeakness:\n\nThe proposed method is not novel. Actually, this paper does not propose a novel method. The main contribution is to scale up the batch size to accelerate the offline SAC training. And I think the paper is more like a technical report.\nIn my opinion, the comparison is not fair for many experiments. For example, the x-axis represents steps in Figure 5, 6, 7. That means the number of epochs is increased for the same number of steps when you scale up the batch size. When the agent can be trained with more data, the convergence will be faster. So I think the s-axis should be the number of epochs. \nThis paper mainly focuses on the SAC-based method and proposes LR-SAC. To verify the generality of your proposed method, maybe you can provide more experimental results about other methods. \nSensitivity analysis about hyper-parameters, such as learning rate. We know RL is more sensitive to hyper-parameters. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject