Revisiting the Minimalist Approach to Offline Reinforcement Learning
Authors: Denis Tarasov
Vladislav Kurenkov
Alexander Nikulin
Sergey Kolesnikov
Venue: NeurIPS 2023 poster (2023)
Keywords: Offline Reinforcement Learning
TL;DR: We modify TD3+BC with [layer norm, deeper critic, higher discount factor, decoupled penalization] and find it can outperform SOTA ensemble-free offline RL algorithms

Abstract: Recent years have witnessed significant advancements in offline reinforcement learning (RL), resulting in the development of numerous algorithms with varying degrees of complexity. While these algorithms have led to noteworthy improvements, many incorporate seemingly minor design choices that impact their effectiveness beyond core algorithmic advances. However, the effect of these design choices on established baselines remains understudied. In this work, we aim to bridge this gap by conducting a retrospective analysis of recent works in offline RL and propose ReBRAC, a minimalistic algorithm that integrates such design elements built on top of the TD3+BC method. We evaluate ReBRAC on 51 datasets with both proprioceptive and visual state spaces using D4RL and V-D4RL benchmarks, demonstrating its state-of-the-art performance among ensemble-free methods in both offline and offline-to-online settings. To further illustrate the efficacy of these design choices, we perform a large-scale ablation study and hyperparameter sensitivity analysis on the scale of thousands of experiments.

Revisiting the Minimalist Approach to Offline Reinforcement Learning
Denis Tarasov Vladislav Kurenkov Alexander Nikulin Sergey Kolesnikov Tinkoff
{den. tarasov, V. V.kurenkv,, a. a.p.nikulin, s.s. o.s.
Abstract
Recent years have witnessed significant advancements in offline reinforcement learning (RL), resulting in the development of numerous algorithms with varying de- grees of complexity. While these algorithms have led noteworthy improvements, many incorporate seemingly minor design choices that impact their effectiveness beyond core algorithmic advances. However, the effect these design choices established baselines remains understudied. In this work, we aim to bridge this gap by conducting a retrospective analysis of recent works offline RL and propose ReBRAC, minimalistic algorithm that integrates such design elements built on top the TD3+BC method. We evaluate ReBRAC on 51 datasets with both proprioceptive and visual state spaces using D4RL and V-D4RL benchmarks, demonstrating its stateoof-the-ar performance among ensemble-frre methods both offline and affline-to-onling settings. To further illustrate the efficacy of these design choices, we perform a large-scale ablation study and hyperparameter sensitivity analysis the scale of thousands of experiments
Introduction
Interest of the reinforcement learning (RL) community in the offline setting has led myriad of new algorithms specifically tailored to learning highly performant policies without the ability to interact with an enviromment (Levine et al. 2020; Prudencio et al. 2022). Yet, similar to advances in online RL (Engstrom et al., 2020: Henderson et al.] 20188) many of those algorithms come with an added complexity design and implementation choices beyond core algorithmic innovations, requiring delicate effort in reproduction, hyperparameter tuning, and causal attribution of performance gains.
Indeed, the issue of complexity was already raised in the offline RL community by Fujimoto Gu (202T); the authors highlighted veiled design and implementation-Ien adjustments (e.g.. different architectures or actor pre-training) and then demonstrated how simple behavioral cloning regularization added the TD3 (Fujimoto al. 2018) constitutes strong baseline in the offline setting. This minimalistic and uncluttered algorithm, TD3+BC, has become de-facto standard baseline to be compared against. Indeed, most new algorithms juxtapose against it and claim significant gains over (Akimov et al., 2022 An et al., [02T]: Nikulin et al. 2023 Wu et al. 2022 Chen et al. 2022b Ghasemipour et al. 2022). However, application newly emerged design and implementation choices to this bascline is still missing.
In this work, we build upon Fujimoto & Gu 2021 line of research and ask: what the extent to which newly emerged minor design choices can advance the minimalistic offline RL algorithm? The answer illustrated in Figure we propose an extension to TD3+BC, ReBRAC (Section (3), that simply
adds on recently appeared design decisions upon it. We test our algorithm on both proprioceptive and
Our implementation is available wtt.ttp::/ggcccu..com///////////
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

POR
1.000
(TESHE))
TC3
8.75
IOL
Algorithm X
Algorithm
Biter critie (TSO: (AE)
moodie here ek:
TD3+BC
(9g)
ANRRAC
0.50
ReSRAC
SAC-IND
Larger beсohes ILU-3M4 946)
Adjusted (SPOT)
ReSRAC
IOL
30 15
103
130
0.5
0.6
0.7
D4RL Scor
POX
(c)
Figure 1: (a) The schema of our approach ReBRAC (b) Performance profiles (c) Probability of improvement. The curves (Agarwal al. 2021) are for D4RL benchmark spanning all
Gym-MuJoCo, AntMaze, and Adroit datasets (Fu et al. 2020).
visual state space problems using D4RL (Fu et al. 2020) and V-D4RL (Lu et al. 2022) benchmarks (Section 4 demonstrating its state-of-the ofe--o--thtrrrrr art performance across ensemble-tres methods. Moreover, our approach demonstrates state-of of the-the-rt performance in offlnneto-offling setup on D4RL datasets (Section 4.3) while not being specifically designed for this setup. further highlight the efficacy of the proposed modifications, we then conduct large-scale ablation study (Section 4.4). We hope the described approach can serve as strong baseline under different hyperparameter search budgets (Section 4.6). further accentuating the importance of seemingly minor design choices introduced along with core algorithmic innovations.
2 Preliminaries
2.1 Offline Reinforcement Learning
standard Reinforcement Learning problem defined Markov Decision Process (MDP) with the tuple {S, A, P, R2 7). where C IR" is the state space, A C R is the action space, P: 5 x A - S the transition function, R S A E is the reward function, and 7 E (0, 1) is the discount factor. The ultimate objective is to find a policy π (als) that maximizes the cumulative discounted return E, Er R(s,, a.). This policy improves by interacting with the environment, observing states, and taking actions that provide rewards.
In offline RL, policies cannot interact with the environment and only access static transaction dataset D collected by one or more other policies. This setting presents new challenges, such as estimating values for state-action pairs not included in the dataset while exploration unavailable (Levine et al. 2020
2.2 Behavior Regularized Actor-Critic
Behavior Regularized Actor-Critic (BRAC) is an offline RL framework introduced in Wu et al. (2019) The core idea behind BRAC is that actor-critic algorithms can be penalized in two ways to solve offline RL tasks: actor penalization and critic penalization. represented in Equation (T), and the critic objective in Equation (2). where F is divergence function between dataset actions and policy actions distributions. The differences from a vanilla actor-critic are highlighted in blue.
this framework, the actor objective is
argmax (e,,)) -
0 = argmin E
a... anD (r
F(d', (())))
d'mm)(s')
In the original work, various choices F were evaluated when used as the regularization term for the actor or critic. The authors tested KI divergence, Kernel MMD, and Wasserstein distance but did not observe any consistent advantage. Finally, is essential to note that, originally, both regularizations coefficients had the same weight.

---

Subsequently, TD3+BC (Fujimoto & Gu 2021) was introduced, utilizing Mean Squared Error (MSE) as the regularization term for the actor. TD3+BC considered be the minimalist approach to offline RI as it modifies existing RI. algorithms by simply adding behavior cloning term into actor loss which is easy to implement and does not bring any significant computational overhead.
3 ReBRAC: Distilling Key Design Choices
In this section, we describe the proposed method along with the discussion of the new design choices met in the offline RL literature (Table TT. Our approach is more general version of BRAC built on top of the TD3+BC (Fujimoto & Gu 2021) algorithm with different modifications in design while keeping it simple (Figure We refer to our method as Revisited BRAC (ReBRAC).
Table I:
Adoption of implementation and design choices beyond algorithmic advancements in some recently introduced algorithms.
Modification
TD3+BC
COL
EDAC
MSG
CNF
LB-SAC
SAC-RND
Deeper networks
X
J
Larger batches
X
X
X
X
Layer Normalization
X
X
X
Decoupled penalization
X
X
X
Adjusted discount factor
X
X
X
X
Deeper Networks The use of deeper neural networks has been a critical factor in the success of many Deep Learning models, with model quality generally increasing as depth increases, provided there is enough data to support this scaling (Kaplan et al. 2020). Similarly, recent studies in RL (Neumann Gros, 2022, Sinha et al., 2020) and offline RI. (Lee et al. 2022 Kumar et 2022) have demonstrated the importance of depth in achieving high performance. Although most offline RL. algorithms are based on SAC (Haarnoja et al. 2018 or TD3 (Fujimoto et al. 2018), which by default employ two hidden layers, recent work (Kumar et al. 2020) uses three hidden layers for SAC instead, which appears to be an important change (Fujimoto & Gu 202T). The change network size has been adopted by later works (An et al. 2021 Yang et al. 2002 Zhuang et al. 2023 Nikulin et al. 2023) and may be one of the critical modifications that improve final performance.
The original BRAC and TD3+BC algorithms used only two hidden layers for their actor and critic networks, while most state-of-the-an solutions use deeper networks. Specifically, three hidden layers have become a common choice for recent offline RI. algorithms. In ReBRAC, we follow this trend and use three hidden layers for the actor and critic networks. Additionally, we provide an ablation study in Section 4.5 to investigate the effect of the number of layers ReBRAC's performance.
LayerNorm LayerNorm (Ba et al. 2016 is a widely used technique in deep learning that helps improve network convergence. In Hiraoka al. (202T), authors add dropout and LayerNorm to different RI. algorithms, notably boosting their performance. This technique also applied in Smith
al. (2022), and it appears that boost is achieved primarily because of LayerNorm. Specifically. in offline RL, various studies have tested the effect of normalizations between layers (Bhatt et al. 2019
Kumar al. 2022) Nikulin et al. 2022 2023). A parallel study by Ball al. (023) empirically shows that LayerNorm helps to prevent catastrophic value extrapolation for the Q Fuuction when
using offline datasets in online RL. Following these works, in our approach, we also apply LayerNorm between each layer of the critic networks.
Larger Batches Another technique to accelerate neural network convergence is large batch opti- mization You et al. 2017 2019). While studying batch sizes larger than 256 is limited, some prior works have used them. For instance, the convergence SAC-N was accelerated in Nikulin et al. (2022. More recently proposed algorithms also use larger batches for training, although without providing detailed analyses (Akimov et al. 2022 Nikulin et al. 2023).
The usage of large batches in offline RL is still understudied, and its benefits and limitations are not fully understood. Our experiments show that in some domains, using large batches can lead to significant performance improvements, while in others, it might not have notable impact or
3

---

even drop the performance (see Table 8). We increased the batch size to 1024 samples and scaled the learning rate for D4RL Gym-MuloCo tasks, following the approach proposed by Nikulin et al.) (2022).
Actor and critic penalty decoupling The original BRAC framework proposed penalizing the actor and the critic with the same magnitude. Most of the previous algorithms restrict only actor (Fujimoto
& Gu 2021 Wu al., 2022) or only critic (Kumar al. 2020 An et al. 202T, Ghasemipour et al. 2022). TD3-CVAE (Rezaeifar et al. (022) penalizes both using the same coefficients, while
another study, SAC-RND (Nikulin al. al.]| [023)) shows that decoupling the penalization in offline RL
is beneficial for algorithm performance, although ablations on using only one of the penalties are
missing.
Our method allows simultaneous penalization of actor and critic with decoupled parameters. Inspired by TD3+BC (Fujimoto & Gu 202T), Mean Squared Error (MSE) is used as divergence function F, which we found simple and effective. The actor objective is shown in Equation (3), and the critic objective is shown in Equation (4). Differences from the original actor-critic are highlighted in red. Following TD3+BC (Fujimoto & Gu 202T). the Q function is normalized to make the algorithm less sensitive regularization parameters. Nonetheless, we forego the utilization of state normalization, as initially suggested in TD3 + BC, driven by our intention to execute the algorithm online and the observation that this adjustment typically results in negligible impact. Since our approach principally builds upon TD3+BC, the differences in their performances should be considered the most important ones and do not appear only because of the additional hyperparameters search.
argmax ((,D))D (()(s, (T(S) a)²]
(s.a a AND [ (r (r 7 (QT(s' d') B2 (a' - 22)))))))
0 argmin E
n' (m))s')
Discountt factor y value change The choice of discount factor is an important aspect in solving RL problems (Jiang et al. 2013). A recent study (Hu et al. 2022) suggests that decreasing the default value of y from 0.99 may lead to better results in offline RL settings. In contrast, in SPOT (Wu let al. 2022), the authors increased the value of 7 up to 0.995 when fine-tuning AntMaze tasks with sparse rewards, which resulted in mate-of-the-art solutions. Similarly, in the offline setting SAC-RND (Nikulin et al. (023)., increasing also achieved high performance on the same domain. The choice of increased y for AntMaze tasks was motivated by the sparse reward, i.e., a low 7 value may not propagate the training signal well. However, further ablations are needed to understand if the change in the parameter directly responsible for the improved performance. In our experiments, we also find that increasing from the default value of 0.99 to 0.999 is vital for improved performance on this set of tasks (see Table 8).
4 Experiments
4.1 Evaluation on offfine D4RL
We evaluate the proposed approach three sets of D4RL tasks: Gym-MuJoCo, AntMaze, and Adroit. For each domain, we consider all of the available datasets. We compare our results to several ensemble-free baselines, including TD3+BC (Fujimoto & Gu 202T). IQL. (Kostrikov al. (021).
CQL Kumar et al. 2020) and SAC-RND (Nikulin et al. 2023).
The majority of the hyperparameters are adopted from TD3+BC, while 31 and B3 parameters from Equation (3) and Equation ( are tuned. We examine the sensitivity to these parameters in Section 4.6
For complete overview of the experimental setup and details, see Appendix A
Following Wu et al.) (2022), we tune hyperparameters over four seeds (referred to as training seeds) and evaluate the best parameters over ten new seeds (referred to as unseen training seeds), reporting the average performance of the last checkpoints for D4RL tasks. On V-D4RL, we use two and five seeds, respectively. This helps to avoid overfitting during hyperparameters search and outputs more just and reproducible results. For a fair comparison, we tune TD3+BC and IQI. following the same protocol. We also rerun SAC-RND Gym- and AntMaze tasks and tune it for the Adroit

---

Table 2: Average normalized score the final evaluation and ten unseen training seeds Gym- MuloCo tasks. CQL scores were taken from An et al. (202T) The symbol + represents the standard deviation across the seeds. To make fair comparison against TD3+BC and IQL, we extensively tuned their hyperparameters.
Task Name
TD3+BC
IQL
CQL
SAC-RND
ReBRAC, our
halfcheetah-rannde
30.9 1 0.4
0.8
31,1 3.5
27.6 2.1
29.5 + 1.5
seachheeta--mediine
0.9
5..0
46,9
1..4
65.6 + 1.0
mulfcheetah-exper
0.44
2.1
9
102.6 4.2
105.9 + 1.7
halfcheetah medium expert
5.6
92
9.
1.5.1
101.
45.0
4..1
45.3
51.2
51.0 + 0.8
halfcheetah- -ulleal
2.5
7.7
7.99
81.2
82.1 + 1.1
hopper-random
0.7
5.9
0.6
19.6 = 12.4
8.1 + 2.4
hopper-medium
6.6
6.22
6..9
91. 10.1
102.0 + 1.0
hopper-expert
3.7
108.8
1.6.5
109.8
100.1 1 8.3
87.8 = 10.5
=
96.9
109.8 0.6
107.0 + 6.4
$
89.6 13.2
8.3
9..2
98.1 + 5.3
Sooppe--fullrepleee
97.9 17.5
104.4 =
1.6
1.8.4
107.1 + 0.4
walker2d- randomm
2.6
11.37..
5.1
18.7 6.9
18.4 + 4.5
walker2d-medium
+
80.7 3.4
79.5
92.7 1.2
82.5 + 3.6
walker2d-exper expert
110.0
+ 32.3
109.3
104.5 22.8
112.3
110.4
112.1 0.5
1.2.
104.6 = 11.2
111.6 + 0.3
walker2d-medinee meduuuu---m
68.0 + 19.2
75.4 9.3
76.8 + 10.0
89.4 3.8
77.3 + 7.9
walker2d walker2d-full-repla full-replay
90.3 + 5.4
97.5 + 1.4
94.2 = 1.9
105.3 3.2
102.2 + 1.7
Average
70.3
72.9
73.6
82.6
81.2
Table 3: Average normalized score over the final evaluation and ten unseen training seeds AntMaze tasks, CQL scores were taken from Ghasemipour et al. (2022). The symbol represents the standard deviation across the seeds. To make fair comparison against TD3+BC and IQL, we extensively tuned their hyperparameters.
Task Name
TD3+BC
IQL
CQL
SAC-RND
ReBRAC, our
antmaze-umaze
66.3
83.3
74.0
97.0 1
97.8 1.0
antmaze-umaze-diveer
53.8 1 8.5
70.6 3.7
84.0
66.0
1 25.0
88.3 1 13.0
antmaae-mediim-ppee play
26.5 1 18.4
64.6149
61.2
38.5
29.4
84.0 4.2
25.9 15.3
61.7
53.7
74.7
10.7
76.3 = 13.5
hatmaze-large-plat
0.0
6.5
15.8
43.9
29.2
60.4 26.1
antmaze-late laaiazzellree-diveeee diverse
0.0 0.0
2.86
14.9
45.7
28.5
54.4 = 25.1
Average
28.7
58.3
50.6
60.9
76.8
domain. In other cases, we report results from previous works, meaning that scores for other methods can be lower if evaluated under protocol.
The results of our tests on D4RL's Gym-MuJoCo, AntMaze, and Adroit tasks are available in Table 2 T Table 4 respectively. The mean-win wise best results among algorithms are highlighted with bold. and the second best performance underlined. Our approach, ReBRAC, achieves state-of-the-art results on Gym-MuloCo, AntMaze, and Adroit tasks outperforming all baselines on average, except SAC-RND on Gym-MuJoCo tasks, which slightly better. Performance profiles and probability of improvement (Agarwal et al.) 2021 in Figure Tb and Figure acoo demonstrate that ReBRAC is competitive when compared to algorithms that we contrast against. Our method is also comparable to ensemble-based approaches (see Appendix C for additional comparisons).
4.2 Evaluation on offline V-D4RL
In addition to testing ReBRAC on D4RL, we evaluated its performance V-D4RL. benchmark LLu al. 2022). Our motivation for doing so was the fact that scores on D4RI. Gym-MuloCo tasks have saturated in recent years, and even after the introduction of ensemble-bard offline RL methods

---

Table Average normalized score over the final evaluation and ten unseen training seeds on Adroit tasks. BC and CQL scores were taken from Yang et al. (022). The symbol + represents the standard deviation across the seeds. To make fair comparison against TD3+BC and IQL., we extensively tuned their hyperparameters.
Task Name
BC
TD3+BC
IQL
CQL
SAC-RND
ReBRAC, our
pen-human
34.4
81.8 + 14.9
81.5 + 17.5
37.5
5.6 + 5.8
103.5 + 14.1
pen-cloned
56.9
61,4 + 19.3
77.2 + 17.7
39.2
2.5 6.1
91.8 + 21.7
pen-expert
85.1
146.0 7.3
133.6
16.0
107.0
45.4 22.9
154.1 +: 5.4
door-human
0.5
-0.1 = 0.0
3.1 2.0
9.9
0.1
0.0 + 0.0
door-cloned
-0.1
0.1 0.6
0.8 1.0
0.4
0.2
1.1 + 2.6
door-expert
34.9
84.6 - 44.5
105.3 - 2.8
101.5
73.6 26.7
104.6 + 2.4
hammer-human
0.4 =
+
4.4
-0.1 0.110..
0.2
hammer-cloned
0.8
0.8 = 0.7
1.1 0.5
2.1
0.1 0.4
6.7 1: 3.7
hammer-expert
125.6
117.0 + 30.9
129.6 + 0.5
86.7
24.8 + 39.4
133,8 + 0.7
relocate-human
0.0
-0.2 = 0.0
0.1 + 0.1
0.2
0.0 4.0
0.0 + 0.0
relocate-cloned
-0.1
-0.1 0.1
0.2 + 0.4
-0.1
0.0
0.9 + 1.6
relocate-cxpert
101.3
107.3
1.6
106.5 + 2.5
95.0
3.4
106.6 + 3.2
Average w/o expert
11.7
18.0
20.8
11.7
1.0
25.5
Average
36.7
49.9
53.4
40.3
12.9
58.6
Table Average normalized score over the final evaluation and five unseen training seeds on V-D4RL tasks. The score is mapped from range of [0, 1000] to [0, 100). The symbol + represents the standard deviation across the seeds.
Environment
Ofline DV2
DrQ+BC
CQL
BC
LOMPO
ReBRAC, our
random
28.7 =13.0
5.5 +0.9
14.4 +12.4
2.0 +0.2
21.9=8.1
15.9 2.3
mixed
56.5 118.1
28.7 16.9
11.4 112.4
16.5 14.3
34.7
416
walker walk
medium
34.1 119.7
46.8 12.3
14.8 116.1
40.9 13.1
43.4
52.5 3.2
medexp
43.9 4334.4
86.4 45.66
56.4 438844
47.7 43.9
39.2
92.7 1.3
expert
40.6
67.5
89.6
93.9
577
81.4 10.0
random
31.7
1:0.6
5.94
10.0
11.45) =311
12.9 = 2.2
mixed
61.61:1.0
44.8 1.3.6
10.7 1:12.8
25.0 13.6
46.3-133.6
46.8 0.7
cheetah-run
medium
17.243..5
+3.0
40,9 +:5.1
51.6
16.4 8.3
39.0
medexp
10.4 #3.55
50.6 +8.2
20,9 45.5
$7,5 16.3
1199199
$8.3 2 11.7
expert
13.22
34.5
67.4 16.8
35.6
random
0.1 +0.0
0.1 +0.0
0.2
+0.0
0.1 =0.0
0.1 = 0.0
mixed
0.2:10.1
13.9
0:00
18.8 1442
0,2 20.0
16.0
humanoid walk
medium
0.240..1
6.2
40.0
13.5 14.1
0.0
9.0 2.3
medexp
0.1 +0.0
7.0
0.1 40.0
17.2 44.7
0,2 =0.00.0
7.8 2.4
expert
0.21
2..9
1..5
13.77
=0.0
2.9
Average
20.0
30.4
21.9
30.3
15.6
35.5
by An al. (2021), there has been no notable progress on these tasks. On the other hand, V-D4RL provides a similar set of problems, with datasets collected in the same way as in D4RL but with the agent's observations now being images from the environment.
We tested our algorithm on all available tingle-task datasets without distractors and compared it to the baselines from the original V-D4RL work (Lu al., 2022). The results reported in Table 5 Our results on most of the tasks,
proposed approach achieves state-of-the-art or Mlose-to-stateoof and it the only method that, on average, performs notably better than naive Behavioral Cloning.
4.3 Evaluation on offline-to-online D4RL
The evaluation of offline-to-onlin performance is pivotal aspect for reinforcement learning (RL) algorithms, partieularly in light of recent developments. In this context, we conducted additional tests on ReBRAC, it stands out promising algorithm for several compelling reasons.

---

First and foremost, ReBRAC demonstrates a remarkable proficiency following offline pre-training. Secondly, our algorithm shares notable similarities with TD3+BC, method that has exhibited effectiveness in online fine-tuning as observed by Beeson et al. (Beeson & Montana 2022)
For the sake of simplicity, we opted disable critic penalization during the online fine-tuning. Furthermore, we linearly decay the actor's penalty to half of its initial value, following the approach described by Beeson & Montana (0022). Notably, no hyperparameter tuning was performed this process.
To evaluate approach in the offline-to-onling setting, we followed the methodology outlined by Tarasov et al. (2022). In our comparative analysis. we consider the following algorithms: TD3+BC (Fujimoto & Gu 2021 D. IQL (Kostrikov et al. 202T), CQL (Kumar al. 2020), SPOT (Wu et al.) 2022). and COL- (Nakamoto et al., 2023). The scores after the offline stage and online tuning,
reported in Table 6 We also provide finetuning cumulative regret proposed by Nakamoto et al. (2023) in Table 7
Table 6: Normalized performance after offiin pretraining and online finetuning on D4RL. Baselines scores except TD3+BC taken from Tarasov al. (2022). ReBRAC and TD3+BC scores averaged over ten random seeds, and all others are averaged over four as in Tarasov et al. (2022).
IQL
SPOT
Cal-QL
ReBRAC,
antmone-e-v2
668
77.00 -
91.00 99.90
76.75
99.75
97.8 99.8
unmuue d v2
59.1 - 48.4
19.50 - 63.75
36.25
95.00
32.00
98.50
85.7
9%.
59. 94.8
71.75 -2 89:75
67:25 97.22
71.75-+ 98.75
78.4
97.7
unimare-mm m-d-y2
62.6 -
64.25 -+ 92.25
73.75 -a 94,50
62:00 - 96.
78.6
98.5
aamuae à pp-2
21.5 -a 0.1
38,50 64.50
31.50 -7.00
31.75 - 97.25
47.0-> 39.5
untmose-bd-12
9.5 -4 0.4
36.75 - 64.25
17.50 $1.00
44.00 91.50
66.7 77.6
AntMaze
46.4 (18.4
56.29 (+22212))
52.88
53,04 97.33 (+24.29)
75.7 85.2 (+9.5)
pon-c-yl
84.19 02
43.63
-2.66
152.0
300-+3.4
0.33
hummas-ooy]
3.97 3.73
0.17
41
telocatc-ovvl
.0.3
0.04 -4 0.32
-0.34
10.31
-1.331
Adroit Ang
22.1 -9 (493))
21.09 44.99 (423.3))
2,43 -2 11.81 (19.30)
-0.76 -4 -0.99 (220))))
24.0 -> (476.1)
Total avg
|
36.7 - (+..)
42.45 -+ (+2266))
32,70 -+ (+2744))
31.52 - 58.08 (+25.56)
55.0 (-36.1)
Table 7: Cumulative regret of online finetuning calculated as 1 average success rate. Baselines scores except TD3+BC are taken from Tarasov et al. (0022). ReBRAC and TD3+BC regrets are averaged over ten random seeds, and all others are averaged over four as in Tarasov et al. (2022).
Task Name
TD3 BC
CQL
IQL
SPOT
Cal-QL
ReBRAC, our
Latmaze-umazee umaze-v2
0.09 = 0.08
0.02 0.00
0.07 2 0.00
0.02 + 0.00
0.01 = 0.00
0.00 1 0.00
animaze umaze diverse-v2
0.47 1 0.16
0.09 1.01
043 0.11
0.22 1: 0.07
0,05 1 0.01
0.06 0.13
animaze-mett -wedimddddiiimee play-
0,12 : 0.05
0.08 0.01
0.09 0.01
0.06 1 0.00
0.04 : 0.01
0.03 0.01
animaze medium peedii--diversee diverse-x2
0.09 0.02
0.08 0.00
0.10 : 0.01
0.05 4: 0.01
0.04 4: 0.01
0.02 0.00
smtnaze-larg--plyy--2
0.99
0.21 0.02
0.34
0,29 0.07
0.13
0.36
seimaze-lrge-ddiveeee
0.99 0.01
0.21 +: 0.03
0,1 4 0.03
0.23 + 0.08
0.13 4: 0.02
0.10 0.07
avg
0.45
pen cloned-vi door-eloned-vl
0:30::0.10
0.97 1: 0.00
0.37 :: 0.01
0.58 1: 0.02
0.98 4: 0.01
0.08 0.00
0.97 0.03
0.00
0.3 0.03
0.99
0.01
1.00
+ 0.00
0.26
hammer--lonod-vi
0.92 0.14
1,00 0.00
0.65
= 0.10
0.98
: 0.01
1.00
: 0.00
0.13 0.02
relocate-cloned-vl
0.99 0:01
1,00
+ 0.00
1.00 :: 0.00
1,00 + 0.00
1.00 :: 0.00
0.85 0.07
Adrolt avg
0.79
0.99
0.71
0.99
Total avg
0.59
0.47
0.43
0.44
0.44
0.18
ReBRAC exhibits competitive performance, surpassing four out of six AntMaze datasets and achiev- ing tfte-of-the-art results in terms of final scores on Adroit tasks. On average, ReBRAC outperforms problem in
its closest competitor, Cal-CQL, which was specifically designed for the the concurrent work (Nakamoto al. 2023).
Regarding regret, ReBRAC outperforms all other algorithms, with Cal-QL. being the sole exception, showing slightly better results average only in the AntMaze domain.
4.4 Ablating Design Choices
To better understand the source of improved performance, we conducted an ablation study on the modifications made to the algorithm. Results can be found in Table 8 Additional ablation studies
7

---

for all datasets can be found in Appendix G One modification at time was disabled, while all other changes were retained, including layer normalization in the critic network, additional linear layers in the actor and critic networks, adding an MSE penalty to the critic and actor loss. In the case of AntMaze, we also attempted to use the default y value instead of the increased one. To further demonstrate the efficacy of our modifications, we also ran our implementation as equívalent to the original TD3+BC, with all changes disabled and hyperparameters were taken from the original paper. This experiment serves to show that the improved scores are due to the proposed changes in the algorithm and not just different implementations. Furthermore, we searched for the regularization parameter for our implementation of TD3+BC to demonstrate that tuning this parameter is not the sole source of improvement. Moreover, we tested the TD3 + BC by adding each of the ReBRAC's modifications independently to demonstrate that each individual modifications is not sufficient for achieving performance of ReBRAC where the combination of modification appear.
Additionally, we run ablation which to validate the importance of decoupling by disabling it and searching the best penalty parameter value from the set of all previously used values listed in the Appendix B
Table 8: ReBRAC's design choices ablations: each modification was disabled while keeping all the others. For brevity, report the mean of average normalized scores over four unseen training seeds across domains. We also include tuned results for TD3+BC to highlight that the improvement does not come from hyperparameter search. For dataset-specific results, please refer to Appendix G
Ablation
Gym-MuloCo
AntMaze
Adroit
All
TD3+BC, paper TD3+BC, our
27.3
0.0
63.4
18.5
52.3
52.2
TD3+BC, tuned
71.8 (-10.9%)
27.9 (-62.9%)
53.5 (-25.9%)
58.3 (-19.2%)
TD3+BC w/ y change
17.5 (-76.7%)
TD3+BC w/ LN
71.4 (-11.4%)
35.6 (-52.7%)
55.6 (-4.1%)
60.2 (-16.6%)
TD3+BC w/ large batch
14.4 (-82.1%)
0.0 (-100.0%)
1.6 (-97.2%)
7.9 (-89.0%)
TD3+BC w/ layer
71.2 (-11.6%)
44.1 (-41.4%)
56.4 (-2.7%)
61.9 (-14.2%)
ReBRAC w/o large butch
75.9 75.9 (-5.8%)
ReBRAC w large batch
41.0 (-45.6%)
55.4 (-4.6%)
ReBRAC w/o y change
21.0 (-72.1%)
ReBRAC w/o LN
59.2 (-26.5%)
0.0 (-100.0%)
25.1 (-56.7%)
38.0 (-47.3%)
ReBRAC w/o layer
78.5 (-2.6%)
18.1 (-75.9%)
59.0 (+1.7%)
61.9 (-14.2%)
ReBRAC w/o actor penalty
22.8 (-71.7%)
0.1 (-99.8%)
0.0 (-100.0%)
11.4 (-84.2%)
ReBRAC w/o critic penalty
81.1 (+0.6%)
72.2 (-4.1%)
56.9 (-1.8%)
71.5 (-0.9%)
ReBRAC w/o decoupling
79.8 (-0.9%)
76.9 (+2.1%)
56.7 (-2.2%)
71.6 (-0.8%)
ReBRAC
80.6
75.3
58.0
72.2
The ablation results show that ReBRAC outperforms TD3+BC not because of different implementa- tions or actor regularization parameter choice. All domains suffer when the LayerNorm is disabled, leading to the halved average performance overall. Removing additional layers leads to a notable drop in AntMaze tasks, while on Gym-MuJoCo decrease is small, and on Adroit tasks, we can see a slight boost. The algorithm fails to learn most tasks when the actor penalty is disabled. Notably, the critic penalty plays a minor role in improving the performance on most of the problems as well as the decoupling penalties. Using standard batch size on Gym MMy-MouoCoo tasks significantly decreases final scores, while using the increased discount factor for AntMaze is crucial for obtaining state-of-the-art performance.
Based on the conducted ablations studies, the proposed configuration of design choices leads the best performance on average. Note that tuning these choices for each task independently makes it possible get even higher scores than we report in Section 4.1 We also can change the number of hidden layers in the networks, leading to better performance, see Section 4.5) But in real life, algorithm evaluation might be costly, so limit ourselves to tuning only regularization parameters.
4.5 Stacking Even More Layers
As ablations show, the depth of the network plays an important role when solving AntMaze and HalfCheetah tasks (see Appendix G). We conduct additional experiments to check how the perfor-
8

---

mance depends on the network depth in more detail. Ball et al. (2023) used networks of depth four when solving AntMaze tasks. Our goal to find the point where the performance saturates. For this, we run our algorithm on AntMaze tasks increasing the number of layers to six while keeping other parameters unchanged. We attempt to increase actor and critic separately and once. We also decreased each network's size by one layer similarly. Results can be found in Figure
Several conclusions can be drawn from the results. First, further increments in the number of layers can lead to better results on the AntMaze domain when layers are scaled up to five. For six layers, performance drops or does not improve. Second, decreasing the critic's size leads the worst performance on most datasets. Lastly, there is no clear pattern on how the performance changes even within single domain. The drop on six layers is the only common feature that can be seen. On average, four critic layers and three actor layers were the best. Changing only the actor's 's network is more stable on average.
med um dlay
broe bay
52-
Nomna feed scare
Uffeze diers
1:30
ITuB 14C)
[TABRAC Acerage
Number of hidden layers
Figure 2: Impact of networks' depth on the final performance for the AntMaze tasks. Scores are averaged over four unseen training seeds. Shaded areas represent one standard deviation across seeds. These graphics demonstrate that one can achieve marginally better scores by tuning the number of layers for certain tasks.
4.6 Penalization Sensitivity Analysis
Following Kurenkov & Kolesnikov (2022). we demonstrate the sensitivity of ReBRAC the choice of 31 and 32 hyperparameters under uniform policy selection on D4RL tasks using Expected Online Performance (EOP) and comparing it to the TD3+BC and IQL. EOP shows the best performance expected depending on the number of policies that can deployed for online evaluation. Results demonstrated in Table As one can see, approximately ten policies are required for ReBRAC to attain ensemble-frre state-of-the-art performance. ReBRAC's EOP is higher for any number of online policies when compared to TD3+BC and better than IQL all domains when the evaluation budget is larger than two policies. See Appendix F for EOP separated by tasks.
Table 9: Expected Online Performance (Kurenkov & Kolesnikov) 2022) under uniform policy selection aggregated over D4RL domains across four training seeds. This demonstrates the sensitivity to the choice of hyperparameters given certain budget for online evaluation. For dataset-specific results, please see Appendix F
Domain
Algorithm
1 policy
policies
policies
policies
10 policies
15 policies
20 policies
TD3+BC
19.3
61.0 = 14.5
65.3 9.3
67.8 =: 3.9
Gym-MuloCo
IQL
65.0 9.1
69.9 #: 5.6
3:55
72,9 = 1.7
73.6 :: 0.8 75.6
73.8 1:
74,0 0.6
ReBRAC
73.3
7588
TD3+BC
6.9 7.0
10.7 1 6.8
13.0
15.5 1:
AntMaze
JO
20.8 + 15.5
38.0 15.4
43.1 13.8
48.7 10.2
$3.2 + 4.4
54.3 + 2.1
$4.7 + 1.2
ReBRAC
67.9 10.0
73.6 7.4
7..14:5.5
78.3 3.4
79.9 à 1.7
80.4 à: 1.1
TD3-BC
23.6
34.6 17.7
40.6 14.5
46.4 9.8
Adroit
IOL
53,1
53.5 0.6
$3.7 1: 0.5
53.9 - 0.3
$1.1 1: 0.2
54.2 1: 0.2
51.2 4 0.1 59.1 0.1::00 0.6
ReBRAC
11.1
53.2 10.9
56.1 6.1
$7.8 = 2.3
58.6 2:
58.9 0.7

---

Related Work
Ensemble-free offline RL methods. recent years, many offline reinforcement learning algorithms were developed. TD3+BC (Fujimoto & Gul 2021) represents a minimalist approach to offline RI,, which incorporates Behavioral Cloning component into the actor loss, enabling online actor-critic algorithms to operate in an offline setting. CQL (Kumar et al. 2020) drives the critic network to assign lower values to out-of-distribttion state-action pairs and higher values to in-distribution pairs. IQL (Kostrikov et al. 2021) proposes method for learning policy without sampling out-of-distribution actions.
Despite this, more sophisticated methods may be necessary to achieve state- ensemble-fre free setup. For instance, Chen et al. (2022b); Akimov et al. (2022) pre-train different forms of encoders for actions, then optimize the actor to predict actions in the latent space. SPOT (Wu et al. 2022) pre-trains Variational Autoencoder and uses its uncertainty to penalize actor for sampling OOD actions while SAC-RND (Nikulin et al. 2023) applies Random Network Distillation and penalizes actor and critic.
ceate--o--the of afrth results in an
Ensemble-basse offline RL methods. A significant number of works in offline reinforcement learning have also leveraged ensemble methods for uncertainty estimation. The recently introduced SAC-N (An et al.) (021) algorithm outperformed all previous approaches on D4RL Gyym-MuJCC tasks; however, necessitated large ensembles for some tasks, such as the hopper task, which required an ensemble size of 500 and imposed a significant computational burden. To mitigate this, the EDAC algorithm was introduced in the same work, which utilized ensemble diversification reduce the ensemble size from 500 to 50. Despite the reduction, the ensemble size remains substantial compared to ensemble-free free alternatives. is worth mentioning that neither SAC-N nor EDAC is capable of solving the complex AntMaze tasks (Tarasov et al.) 2022).
Another state-of -hhe--rrtt algorithm in the Gym-MuJoCo tasks RORL (Yang et al. 2022), which is modification of SAC-N that makes the Q function more robust and smooth by perturbing state-action pairs with the use of out-of-distribution actions. RORL also requires an ensemble size of up 20. On the other hand, MSG (Ghasemipour et al. 2022) utilizes independent targets for each ensemble member and achieves good performance on the Gym-Mut MyM-MuooCo tasks with an ensemble size of four but requires 64 ensemble members to achieve state-of-the of-the-art performance on the AntMaze tasks.
Design choices ablations. Ablations of different design choices established baselines very limited, especially in offline RL. It only shown by Fujimoto & Gu (202T) that CQL performs poorly proposed non-algorithmic differences are eliminated. A parallel study (Ball al. (202)) shows that some of the considered modifications (LayerNorm and networks depth) are important when used in online RL with offline data setting, which is different from pure offline.
6 Conlusion, Limitations, and Future work
In this work, we revisit recent advancements in the offline RL field over the last two years and incorporate modest improvements previously established minimalistic TD3+BC baseline. Our experiments demonstrate that despite these limited updates, we can achieve more than competitive results on offline and affline-to-onlin D4RL and offline V-D4RL benchmarks under different hyperparameter budgets.
Despite the noteworthy results, work is limited to approach and subset of possible design changes. It is imperative to explore the potential impact of these modifications on other offline RL methods (e.g., IQI., CQL, MSG) and to investigate other design choices used in offline RI., e.g., learning rate schedules, dropout (like in IQL), wider networks, or selection between stochastic and deterministic policies.
References
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G Bellemare. Deep reinforcement learning the edge of the statistical precipice. Advances in Neural Information Processing Systems, 2021.
10

---

Dmitriy Akimov, Vladislav Kurenkov, Alexander Nikulin, Denis Tarasov, and Sergey Kolesnikov. Let offline rl flow: Training conservative agents the latent space of normalizing flows. arXiv preprint arXiv:221 I 1096, 2022.
Gaon An, Seungyong Moon, lung-Hyun Kim, and Hyun Oh Song. Uncertainty-basee based offline reinforcement learning with diversified q-ensemble. Advances in neural information processing systems, 34:7436-745 7447, 2021.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607006450, 2016.
Philip Ball, Laura Smith, Nly Kostrikov, and Sergey Levine. Efficient online reinforcement learning with offline data. arXiv preprint (rXiv:2302.02945, 2023.
Alex Beeson and Giovanni Montana. Improving td3-be: Relaxed policy constraint for offline learning and stable online fine-tuming ane-ttuiig. arXiv preprint arXiv:2211.11800 2022.
Aditya Bhatt, Max Argus, Artemij Amiranashvili, and Thomas Brox. Crossnorm: Normalization for off-policy td reinforcement learning. arXiv preprint arXiv:1902.0560) 2019.
Xi Chen, Ali Ghadirzadeh, Tianhe Yu, Yuan Gao, Jianhao Wang, Wenzhe Li, Bin Liang, Chelsea Finn, and Chongjie Zhang. natent-variable advantage-weighhee policy optimization for offline rl. arXiv preprint arXiv:2203.089. 2022a.
Xi Chen, Ali Ghadirzadeh, Tianhe Yu, Jianhao Wang, Alex Yuan Gao, Wenzhe Li, Liang Bin, Chelsea Finn, and Chongjie Zhang. Lapo: Latent-variable advantage-weighet policy optimization for offline reinforcement learning. Advances in Neural Information Processing Systems, 35:
36902-3699 36913, 2022b.
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters deep policy gradients: case study on ppo and trpo. arXiv preprint arXiv:2005. 2020.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 2020.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach offline reinforcement learning. Advances in neural information processing systems, 14:20132-201455 2021.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference on machine learning, pp. 1587-1596. PMLR, 2018.
Seyed Kamyar Seyed Ghasemipour, Shixiang Shane Gu, and Ofir Nachum. Why so pessimistic? estimating uncertainties for offline rl through ensembles, and why their independence matters. arXiv preprint arXiv:2205.1 13703, 2022.
Tuomas Haarnjja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pp. 1861-1870. PMLR, 2018.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau. Doina Precup, and David Meger. Deep reinforcement learning that matters. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Takuya Hiraoka, Takahisa Imagawa, Taisei Hashimoto, Takashi Onishi, and Yoshimasa Tsuruoka. Dropout q-functions for doubly efficient reinforcement learning, arXiv preprint 1rXiv22110.02034, 2021.
Hao Hu, Yiqin Yang, Qianchuan Zhao, and Chongjie Zhang, On the role of discount factor in offline reinforcement learning. In International Conference on Machine Learning, pp. 9072-9098. PMLR, 2022.
Nan Jiang, Alex Kulesza, Satinder Singh, and Richard Lewis. The dependence of effective planning horizon on model accuracy. Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems, pp. 1181-1189, 2015.

---

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0880 2020.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprini arXiv: 1412.6990, 2014.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169, 2021.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances Neural Information Processing Systems, 33:1 179-1199,, 2020.
Aviral Kumar, Rishabh Agarwal, Xinyang Geng, George Tucker, and Sergey Levine. Offline q- learning on diverse multi-task data both scales and generalizes. arXiv preprint 4rX1::2211.15144, 2022.
Vladislav Kurenkov and Sergey Kolesnikov. Showing your offline reinforcement learning work: Online evaluation budget matters. In International Conference on Machine Learning, pp. 11729- 11752. PMLR, 2022.
Kuang-Huei Lee, Ofir Nachum, Mengjiao Yang, Lisa Lee, Daniel Freeman, Winnie Xu, Sergio Guadarrama, lan Fischer, Eric Jang, Henryk Michalewski, et al. Multi-game decision transformers. arXiv preprint arXiv:2205. /55.222,55/521,,, 2022.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.0 01643, 2020.
Cong Lu, Philip J Ball, Tim GJ Rudner, Jack Parker- Holder, Michael A Osborne, and Yee Whye Teh. Challenges and opportunities in offline reinforcement learning from visual observations. arXiv preprint arXiv:2206.0477 2022.
Mitsuhiko Nakamoto, Yuexiang Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning. arXiv preprint arX14:2303.0547, 2023.
Oren Neumann and Claudius Gros. Scaling laws for multi-agent reinforcement learning model. arXiv preprint arXhv:2210.00849) 2022.
Alexander Nikulin, Vladislav Kurenkov, Denis Tarasov, Dmitry Akimov, and Sergey Kolesnikov. Q-ensemble for offline rl: Don't scale the ensemble, scale the batch size. arXiv preprint arXiv:2211.1 2022.
Alexander Nikulin, Vladislav Kurenkov, Denis Tarasov, and Sergey Kolesnikov. Anti-exploration by random network distillation. arXiv preprint 133 2023.
Rafael Figueiredo Prudencio, Marcos ROA Maximo, and Esther Luna Colombini. A survey on offline reinforcement learning: Taxonomy, review, and open problems. arXiv preprint drX1v:2203.01887, 2022.
Shideh Rezaeifar, Robert Dadashi, Nino Vieillard, Léonard Hussenot, Olivier Bachem, Olivier Pietquin, and Matthieu Geist. Offline reinforcement learning as anti-exploration. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 8106 81 11,,,11111001 2022.
Samarth Sinha, Homanga Bharadhwaj. Aravind Srinivas, and Animesh Garg, D2rl: Deep dense architectures in reinforcement learning. arXiv preprint arXiv:2010.09163: 2020.
Laura Smith, Ilya Kostrikov, and Sergey Levine. A walk in the park: Learning to walk in 20 minutes with model-free reinforcement learning. arXiv preprint arXiv:2208.07866 2022.
Denis Tarasov, Alexander Nikulin, Dmitry Akimov, Vladislav Kurenkov, and Sergey Kolesnikov. Corl: Research-orienter deep offline reinforcement learning library. arXiv preprint 10Xiv:2210.071005 2022.
12

---

Jialong Wu, Haixu Wu, Zihan Qiu, Jianmin Wang, and Mingsheng Long. Supported policy optimiza- tion for offline reinforcement learning. arXiv preprint arXiv:2202.062239, 2022.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv: 1911. 1361, 2019.
Rui Yang. Chenjia Bai, Xiaoteng Ma, Zhaoran Wang, Chongjie Zhang, and Lei Han. Rorl: Robust offline reinforcement learning via conservative smoothing. arXiv preprint arXiv:2206.22829, 2022.
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv
preprint arXiv: 03888, 2017.
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsich. Large batch optimization for deep learning: Training bert in 76 minutes. arXiv preprint arXiv: 1904.00962, 2019.
Zifeng Zhuang, Kun LEI, Jinxin Liu, Donglin Wang, and Yilang Guo. Behavior proximal policy optimization. In International Conference on Learning Representations, 2023. URL https htwpaannoooo..tttoooommmmdddeeee
13

---

A Experimental Details
In order generate the results presented in Table Table and Table + we conducted hyperparam- eter search and selected the best results from the final evaluations for each dataset. Our algorithm was implemented using JAX for the D4RL benchmark, For V-D4RL, we implement our approach using Py Torch adopting the TD3+BC implementation from Clean Offline RL (Tarasov et al. 2022). The experiments were conducted on V100 and A100 GPUs.
Gym-Mu.JoCo and Adroit tasks. Our study utilized the latest version of the datasets v2 for Gym-MuloCo and v1 for Adroit. The agents were trained for one million steps and evaluated over ten episodes.
For ReBRAC, fine-tuned the B1 parameter for the actor, which was selected from 0.001, 0.01, 0.05, 0.1. Similarly, the 82 parameter for the critic was selected from a range of 0, 0.001, 0.01, 0.1, The selected best parameters for each dataset are reported in Table I1
For TD3+BC here and the AntMaze domain, we use the same grid used in ReBRAC for actor regularization parameter and add the default value of 0.4.
For IQL. here and in the AntMaze domain, we selected / value from a range of 0.5, 1,3, 6, 10 and IQL value from range of 0.5, 0.7, 0.9, 0.95. We used the implementation from Clean Offline RL (Tarasov al. 2022) and kept other parameters unchanged.
For SAC-RND in Adroit domain we tune 31 (actor parameter) in the range of 0.5, 1.0, 2.5, 5.0, 10.0 and 82 (critic parameter) in the range of 0.01, 0.1, 1.0, 5.0, 10.0.
Ant. tasks. In our work, we utilized v2 of the datasets. It's worth noting that previous studies have reported results using vθ datasets, which were found contain numerous issues" Each agent was trained for million steps and evaluated over 100 episodes. Following Chen et al. 2022a). we modified the reward function by multiplying by 100.
For ReBRAC, the 31 (actor) and 82 (critic) hyperparameters were carefully selected from the respec- tive ranges of 0.0005, 0.001, 0.002, 0.003 and 0, 0.0001, 0.0005, 0.001. In addition, the actor and critic learning rates were optimized from 0.0001, 0.0002, 0.0003, 0.0005 and 0.0003, 0.0005, 0.001, respectively. The optimal hyperparameters for each dataset are presented in Table 11
We also modified the ? value for ReBRAC when addressing these tasks, driven by the following motivation. The length of the episodes in AntMaze can be as long 1000 steps, while the reward is sparse and can only be obtained at the end the episode. As result, the discount for the reward with the default y can be low as 0.99 = 4 - 10 -3, which is extremely low for signal propagation, even when multiplying the reward by 100. By increasing y to 0.999, the minimum discount value becomes 0.999 1000 0.36, which is more favorable for signal propagation.
V-D4RL.
We used single-task datasets without distraction with a resolution of 84 x 84 pixels. For ReBRAC 31 (actor) parameter selected from the range of {0.03, 0.1, 0.3, 1.0} and [3 (critic) parameter from the range of (0.0, 0.001, 0.005, 0.01, 0.13.
Offiine-tooofline. We used the same parameters for the affline-to-online setup with the only difference of setting 32 to zero and lineary decaying 31 to half of it's initial value during the online stage.
14

---

B Hyperparameters
B.1 ReBRAC
Table 10: ReBRAC's general hyperparameters.
Parameter
Value
optimizer
Adam Kingma Ba (2014)
batch size
1024 on Gym-MuJoCo, 256 on other
Icarning rate (all networks)
lc-3 on Gym-Mulod 3 on Adroit and V-D4RL, 1c-4 on Antmaze
tau (7)
5e-3
hidden dim (all networks)
256
num hidden layers (all networks)
3
ganma (7) nonlinearity
0.999 on AntMaze, 0.99 on other
ReL.U
Table 11: ReBRAC's best hyperparameters used in D4RL benchmark.
Task Name
31 (actor)
B2 (critic)
halfcheetah-randee random
0.001
0.1
srrcheetah-mediiue
0.001
0.01
halfcheetah-expe expeet
0.01
halfcheetah- halfcheetah-medium-eee
medium-experrt
halfcheetah-meedee sattheetaa--mdiiiu-rllllll
0.01
0.001
halfcheetah-full-rephhe
0.001
0.1
hopper-random
0.001
0.01
hopper-medium
0.01
0.001
hopper-expert
0.1
0.001
hopper-mediuge
medium-exper
0.1
0.01
hopper-medium-rpppe
0.05
0.5
hopper-full-replay
0.01
0.01
milkerdd-randon
0.01
0.0
walker2d-medine
0.05
0.1
walker2d-expert
0.01
0.5
mallker2t-medium-xxxeee
0.01
walker24--medim--rrppy
0.05
0.01
wallepydd-ulll-rppaay
0.01
0.01
antmaze-une umaze
0,003
0.002
caltaaz--umzzeddivee
0.003
0.001
antmaze-medium-plaa
0.001
0.0005
antmaze-mediun-diivnss
0.001
0.0
antmaze-large-play
0.002
0.001
antmaze-large-divers.
0.002
0.002
pen-human
0.1
0.5
pen-cloned
0.05
0.5
pen-expert
0.01
0.01
door-human
0.1
0.1
door-cloned
0.01
door-expert
0.05
0.01
hammer-human
0.5
hammer-cloned
0.1
0.5
hammer-expert
0.01
0.01
relocate-hune human
0.1
0.01
relocate-clonee
0.1
0.01
relocate-cxpert
0.05
0.01
15

---

Table 12: ReBRAC's best hyperparameters used in V-D4RL benchmark.
Task Name
B1 (actor)
Sa (critic)
walker-wall-random
0.03
0.1
walker- malker-wwalk-nee mell-mediim
0.03
0.005
walker-walk walk-expert
0.1
0.01
weneer-wll-mmedinne
0.3
0.005
walker- saar--alkkmeen medium-repplay
0.3
0.01
Mnretaa-run-randon
0.1
0.01
cheetah-run-meddee nun-medium
0.1
0.1
cheetab-run-expert
0.01
0.01
1.0
0.001
chectah-run
Munetta-ru--medium-aa medium- replay
0.03
0.1
ralk-ro
1.0
0.01
humanoid-wer waetaaoiddwwwll-mme walk-medium
1.0
0.005
humanoid-we walk-eerrr
1.0
humanoid-wall walk-medium-expeee
1.0
0.005
humanoid-wal medium- replay
1.0
0.001
16

---

B.2
IQL
Table 13: IQL's best hyperparameters used in D4RL benchmark.
Task Name
IQL
naffchectah-randeee
0.95
halfchectah-medde
3.0
0.95
halfcheetah-exxe
6.0
0.9
malicheetah-medium-eee
3.0
0.7
sallicheetahmmddiu--rree
0,95
halicheetah-full-reehhh
1.0
0.7
hopper-random
1.0
0,95
hopper-medium
3.0
0.7
hopper-expert
3.0
0.5
hopper-medium-expee
6.0
0.7
hopper-medium-replay
hopper-full-replas
10.0
0.9
walker2d-random
walker2d-medium
6.0
0.5
walker2d-expert
6.0
0.9
wallker2d-mediun--eppeee
1.0
0.5
wallerr2d-meeduu---elayy
walker2d-ffll-reelaay
1.0
0.7
antmaze-uman
10.0
0.7
antmaze-umazzedlivrss
10.0
0.95
antmaze-medium-phe -lay
6.0
0.9
caimaaze-mdiumm-iieeree
6.0
0.9
sutmaze-large-laa
10.0
untmaze-large-divee divered
6.0
0.9
pen-human
0.95
pen-cloned
10.0
0.9
pen-expert
10.0
0.8
door-human
0.5
0.9
door-cloned
6.0
0.7
door-expert
0.5
0.7
hammer-human
3.0
0.9
hammer-cloned
hammer-expert
0.5
0.95
relocate-human
1.0
0,95
relocate-cloned
6.0
0.9
relocate-expert
10.0
0.9
17

---

B.3 TD3+BC
Table 14: TD3+BC's best hyperparameters used in D4RL benchmark.
Task Name
halfcheetah mandom
0.001
halfcheetab-mediumm
0.01
halfcheetah-expert
0.4
halfcheetah-mediing -ed-imm-xxduerrr
0.1
halfchectah medium repplay
0.05
halfcheetah- faul-fflllllll
0.01
hopper-random
0.4
hopper-medium
0.05
hopper-expert
mepper-medium--xpert
0.1
hopper-meddiun-rellyy
0.4
hopper-full-replay
0.01
walker2d-random
0.001
walker2d-medium
0.4
walker2d-expert
0.05
waller224-medium-xxxer
0.1
walker2d medium- replay
0.1
Bilker2d-fullreplaar
0.1
antmaze-umaze
0.4
antmaze-uaazeddiverse
0.4
antmaze-meddummppla
0.003
autraaze-mediuu-ddveesee
0.003
antmaze-large-plee
0.003
antmaaze-arge-divvrre
0.003
pen-human
0.1
pen-cloned
0.4
pen-expert
door-human
door-cloned
0.4
door-expert
0.1
hammer-human
0.4
hammer-cloned
0.4
cammer-expert
0.4
relocate-human
0.1
relocate-cloned
0.1
relocate-expert
18

---

B.4 SAC-RND
Table 15: SAC-RND's best hyperparameters used in D4RL Adroit tasks.
Task Name
81 (actor)
B2 (critic)
pen-human
1.0
10.0
pen-cloned
0.01
pen-expert
10.0
5.0
door-human
5.0
0.01
door-cloned
5,0
1.0
door-expert
10.0
10.0
hammer-human
10.0
0,01
hammer-cloned
1.0
1.0
hammer-expert
10.0
relocate-human
5.0
0.01
relocate-cloned
1.0
relocate-expert
10.0
10.0
C Comparison to Ensemble-based Methods
Comparison of ReBRAC with the ensemble-based methods is presented in Table T6. Table 17 and
Table 18 We add the following ensemble- based methods: RORL for each domain Yang et al. (2022) VEDAC (An al. 2021 for the Gym-MuJoCo and Adroit t and MSG (Ghasemipour et al. 2022 for AntMaze tasks") The mean-wise best results among algorithms are highlighted with bold, the second- best performance is underlined. Our approach, ReBRAC, shows competitive results on the Gym-Mu.l Myy-MuuloCo datasets. On AntMaze tasks, ReBRAC achieves state-oftthe-ee results among ensemble-fre algorithms and good score compared to ensemble-bar based algorithms. And on Adroit tasks, our approach outperforms both families of algorithms.
Table 16: ReBRAC evaluation on the Gym domain. We report the final normalized score averaged over 10 unseen training seeds on v2 datasets. CQL, SAC-N and EDAC scores are taken from An et al. (202T). RORL scores are taken from Yang et al. 2022).
Ememmblofax
Fracmbii-brscd
Tank Name
THS-BC
IQr.
COL
SAC-RND
SAC-N
EDAC
RORI.
RaBRAC,
hullcheeth-anddon
31.9 a: U.4
195 US
31.1 1.1:::15
27.:2:211
25.0 4 119
28.3 2:
25.5 UN
29.5
anllchetab--roiiu
54.7 1.0.
500
469 2 0.4
66.4
67.5 2:
65.9 1.0.5
66.5 0.7
65.6 1.0
hullcheetub expert
93.4 0.4
95.5 2.1
973 1.1
102.6:42
1053 2: 2.6
1063 3.4
105.2
105.9 1.7
hulfcheetab mediun replay
bupper
bopermmlim
60.9
65-2-1-422
61.9 1.944.64
91.1 10.1
100.3 1.0.
101. 0.0.6
104.5
100.0 8: 1.0
hoppe: expert
3.5.5
106.5-29911
1095 0.5
1103 115
1161 0.1
112.5 2: 022
100. 4:33
heppesmmediin-ssge
87.8 10.5
83.5 1:332:30 20.7
96.9-1 15.1
1098 0.6
110.1
4 0.3
180. 6.0.1
112.7
107.0
hopper medium ceplay
31.7
8816 13.2
86.3 7.3
97.2.2990
103.8 1: 11.
101 0.05
102.8
SAL1 4:53
hoppos full-rppay
0.0.7
107.1
randons
wallerdi-mstiee
walkerdd
110. 4
66.9.:32.3
106.5
101.5 22.4
10T.4 2.4
112.5
walkerdd madiun-csper
110.4 4 0.6
112.1:.055
109.1 0.2
104.5 11.2
1167 4 044
1147 2.0.9
121.2 2
111.6
walterld-mannn mederrt--dmdi---pppp
6810 8: 19.2
75.4 08 9.5
76.8 10.0
$9.4
7K. 4 117
N7.1 8
98.4 0.5
77.3 88
90.34 9.4
07.5 1 1.4
94.2 1.9
188.3
94.6 8 DS
99.8 0.7
100.2 9 1.7
Averge wi tull-eplay
7011
70.1
79.5
$2.4
82.9
85.7
7810
Ascrnge
729
SAA-- and EDAC score on medium and large AntMaze tasks Tarasov et 2022). MSS numerical results available for Gym- tasks and Adroit tasks not benchmarked.
19

---

Table 17: ReBRAC evaluation on AntMaze domain. We report the final normalized score averaged over 10 unseen training seeds on v2 datasets, CQL scores are taken from Ghasemipour et al. (2022). RORL scores are taken from Yang et al. (022))
Ensemble free
Ensemhie-bused
Tisk Name
TD3+BC
IQL
CQL
SAC-RND
RORL
MSG
ReiRRA,, our
66.3 +:
83.3 à
74.0
97.0
97.7 :: 1.9
97.9 1.3
97.8
ontmaze-mmire- amare-diverse
antmare medium- play
26.5 = 18.4
64.6 +
61.2
38.5
294
76.1
85.9 +
84.0
animaze medium- medium-diverse diverse
25.9 = 15.3
61. 4 6.1
53.7
74.7 = 10.7
69.3 + 3.3
84.6
2.22
26.1
13.5
larg-pplay
42.5
15.8
64.4
26.1
anmaae laree-lar-eeeee diveree
Average
28.7
58,3
30.6
60.9
65.2
80.5
76.8
Table 18: ReBRAC evaluation Adroit domain. We report the final normalized score averaged 10 unseen training seeds on v1 datasets. BC, CQL, EDAC and RORL scores are taken from Yang al. (2022).
Ensemble
Eusemble
Taok
TD3+CC
SAC-RND
RORT.
EDAC
pen pan-cloned
72-2
21.7
pen expen
85.1
186.0 7.3
133.6
107.0
45.4 22.9
130.3
122.8 14.1
154.1
dooe buman
0.5
0.1 0.0
3.1 2.0
9.9
0.0 0.0
3,7 1 0.7
10.7
6,8
0.0 0.1
dox--elaned
41.1
0.1 11::0. 016
0.3 1.0
0.4
0.2 0.8
.1.1 4 1.1
9.6
= X.5
L.1
doorsespent
34.9
84.6 44.5
105.3 2.8
101.5
73.6 + 26.7
104.9 0.9
-0.3 - 0.1
104.6 2.4
0.44
4.4
hammer-cleod
TT
1.7
expert
relocate human
02 0.0
0.1 0.1
0.2
0.0
0.1 1.1.
0.0
celocat-cloned
.0.1
0.1
01 01.4
-1.1
0.0 0.0
[10
B10 0.01
0,9 1.6
relocat--ngeet
101.3
107.3 1.6
106.5 2.5
95.0
3,4 4.5
47.8 =: 13.5
-0.3 0.0
106.6 3.2
Average wio
11.7
18.0
20.8
11.7
1.0
9.6
17.4
25.5
Average
41.0
20

---

D
Feature Normalization
Table 19: Average normalized score over the final evaluation and ten unseen training seeds on D4RL tasks for types of normalization: LayerNorm (LN) and Feature Norm (FN). The symbol + represents the standard deviation across the seeds. For both variatns tune hyperparameters using the same grid.
Task Name
ReBRAC + LN
ReBRAC + FN
malfcheetah-randoen
29.5 + 1.5
31.4 + 2.7
malfcheetah-medingg
65.6 1 1.0
66.1 +: 1.2
halfchectah-exee exppet
105.9 + 1.7
104.1 3.7
halfchectah-mmediee expprt
101.1
100.9 + 4.7
halfcheetah medium-rephas
51.0 0.8
54.7 + 1.0
halfcheetah-full-reppe
82.1 + 1.1
81.5 + 1.6
hopper-random
8.1 2.4
8.2 + 2.2
hopper-medium
102.0 1.0
102.4 0.2
hopper-expert
100.1 = 8.3 8.3
99.7 11.7
hopper-mediummexpper
107.0 6.4
107.7 + 6.4
hopper-meedimm-replay
+
91.0 = 15.3
hopper-full-replay
107.1 + 0.4
106.7 + 0.5
walker2d-random
18.4 + 4.5
0.0 + 0.7
walker2d-medium
82.5 + 3.6
81.8 + 4.7
walker2d-expert
112.3 + 0.2
110.7 3.2
walleerad-meddum--xpprr
111.6 + 0.3
100.6 34.3
walkerrdd-mediun-rrpla
77.3 + 7.9
80.2 + 8.1
walker2d-full-replay
102.2 1 1.7
101.2 + 2.7
Gym-Mr MuloCo average
81.2
79.3
antmaze-umaze
:: 1.0
96.8 1.6
antmaze-umazz-dliverse
88.3 + 13.0
88.5 7.5
antmaze-mediim-pkle
84.0 4.2
84.1 + 10.1
cemmaze-mediumdiin
76.3 13.5
75.6 = 13.7
antmaze-large play
60.4 26.1
55.0 30.0
untmaze-large-dieees diverse
54.4 + 25.1
66.4 + 7.4
AntMaze average
76.8
77.7
pen-human
103.5 14.1
107.0 13.8
pen-cloned
91.8 +: 21.7
84.9 : 20.1
pen-expert
154.1 + 5.4
151.6 + 4.7
door-human door-cloned
0.0 + 0.1
0.0 + 0.0
1.1 1 2.6
0.1 1 0.1
door-expert
104.6 1: 2.4
105.1 1. 1.4
hammer-human
0.2 + 0.2
0.2 + 0.1
hammer-cloned
6.7 + 3.7
10.1 + 9.5
hammer-expert
133.8 +:
133.4 + 1.5
relocate-human
+
0.0
relocate-cloned
0,9 + 1.6
1.2 + 1.9
relocate-expert
106.6 + 3.2
108.7 + 3.0
Adroit average w/o expert
25.5
25.4
Adroit average
58.6
58.5
21

---

E
Computational costs
Table 20: Computational costs for algorithms in Table 2
Algorithm
Number of runs
Approximate hours per run
TD3+BC, tuning
360
0.3
IQL, tuning
1440
1.8
ReBRAC, tuning
0.4
TD3+BC, eval
180
0.2
IQL, eval
180
SAC-RND, eval
180
1.8
ReBRAC, eval
180
0.4
Sum
3960
4032.0
Table 21: Computational costs for algorithms in Table 3 and Table
17
Algorithm
Number of
Approximate hours per run
TD3+BC, tuning
96
0.5
IQL, tuning ReBRAC, tuning
480
2.1
384
0.6
TD3+BC, eval
0.5
IQL, eval
60
2.0
SAC-RND, eval
60
2.9
MSG. eval
60
5.1
ReBRAC,
0.4
Sum
1260
1940.4
Table 22: Computational costs for algorithms in Table 4
Algorithm
Number of runs
Approximate hours per run
TD3+BC, tuning
240
0.3
IQL, tuning
960
1.8
SAC-RND, tuning
1200
1.1
ReBRAC, tuning
960
0.3
TD3+BC, eval
120
0.2
IQL, eval
120
1.9
SAC-RND, eval
120
1.1
ReBRAC, eval
120
0.3
Sum
3840
3828.0
Table 23: Computational costs for algorithms in Table
Algorithm
Number of runs
Approximate hours per run
ReBRAC, tuning
600
10.6
ReBRAC, eval
75
10.5
Sum
675
7147.5
22

---

Table 24: Computational costs for algorithms in Table and Figure
Algorithm
Number of
Approximate hours per run
ReBRAC, ablations eval
1104
1.4
Sum
1104
1545.6
Expected Online Performance
Table 25: TD3+BC, IQL and ReBRAC Expected Online Performance under uniform policy selection on HalfCheetah tasks.
10mm
(a) strcheetah-randdon EOP.
(b) halfcheetah mediim EOP.
(c) halfcheetah expert BOP.
Noces www.eteee
(d) wettrcetah-mdiium--- expert BOP.
(c) medium-replayy EOP.
(f) halfcheetah- full-replay EOP.
suffihoctah-full-repla
Figure 3: TD3+BC, IQL and ReBRAC visualised Expected Online Performance under uniform policy selection on HalfCheetah tasks.
Table 26: TD3+BC, IQL and ReBRAC Expected Online Performance under uniform policy selection on Hopper tasks.
23

---

(a) hopper-random EOP.
(b) hopper-mediun EOP.
(c)
hopper-expert EOP.
Soppontal splent
(d) hopper-meddum-opprt EOP.
(c) hopper-mcciuumrrpllyy EOP.
(f) hopper-full-replay EOP.
Figure 4: TD3+BC, IQL and ReBRAC visualised Expected Online Performance under uniform
policy selection Hopper tasks.
Table 27: TD3+BC, IQL and ReBRAC Expected Online Performance under uniform policy selection on Walker2d tasks.
10 1 3
100 68
walker2d-random EOP.
(b) walker2d m-ddium EOP.
(c) walker2d- expert EOP.
101
(d) madluerdd EOP. (c) walker2d--eddium-rpply EOP.
(f) 2ulk EOP.
Figure TD3+BC, IQL and ReBRAC visualised Expected Online Performance under uniform policy selection on Walker2d tasks.

---

Table 28: TD3+BC, IQL and ReBRAC Expected Online Performance under uniform policy selection on AntMaze tasks.
044 244
11
10-6
(a) antmaze- umzize EOP.
(b) antmaze- antmaze-medium- EOP.
(c) antmaze- lareeellrrr-ppppp EOP.
12243 23
12375 3
(d) saamaazeumazz-diveee EOP.
(e) suumaaze-edium-divee EOP.
(l) antmaze- large-iv-ree EOP.
Figure 6: TD3+BC, IQL and ReBRAC visualised Expected Online Performance under uniform policy selection on AntMaze tasks.
Table 29: TD3+BC, IQL and ReBRAC Expected Online Performance under uniform policy selection on Pen tasks.
cinted
Polictes
TD3-IC
IQL
ReBRAC
TD3+BC
ReURAC
TD]+BC
ReBRAC
42.0 32.0
81.1 4.1
60.9 28.2
33.6 35.2
73.8 5.8
65.8 + 12.7
73.9 +5.5
130.1 +:28
136.9 25.4
60.4 25.4
89.4 2.9
55.8
20.7
44.9 25.4
76.7 4.8
83.6 +: 21.8
108.9 52.9
131.7 2:1
149.0 :: 13.6
68.7 13.9
00.3 2.5
92. 14.5
52.6 25.2
75.0 4.9
98.8 :: 15.4
125.6 4: 36.7
132.4 2: 1.
152.4 7.0
730 14.2
M39 2.2
9.. 18,9
55.1 a 26.1
7911
94.6 1: 12.2
154.1 à: 27.8
152.5
1.6
153.5 A8
75.6 11.2
V1 4 2.0
V8.4 8.5
62.5 8 22.6
797
97.0 1: 10.4
138. 8 2010
155.1
15
156.1 122
01
99.7 6.5
803
96.7 0.3
1394
154.1 1.5
92.0 1.7
100.5
5.2
500 5.6
1000
8.4
133.6
1545 1.1
92 1.6
101.1
$1.4 5.7
1011
7.6
1337
1547 0.9
9
101.6
81
161.9 7.0
1338 1.1
1548 088
6.4
0.6
43
$3.0 0.9
84.3 5.8
1343
930
86.6
165.1 A8
154.-000 0.0
044
91.12 0.5
HOK2 1.6
848-8-57
168.1 35
14A-200
1831 04
91.1 0.0
HOL1 1.4
85.0---.. 5.6
1683 3
1444-05
1581 0.4
93.2 0.3
HOX3
8..3..5.5
1657 2
1345 0.
155.1 0.4
25

---

(a) pen-human EOP.
(b) pen-cloned EOP.
(c) pen-expert EOP.
Figure 7: TD3+BC, IQL and ReBRAC visualised Expected Online Performance under uniform policy selection on Pen tasks.
Table 30: TD3+BC, IQL and ReBRAC Expected Online Performance under uniform policy selection on Door tasks.
haman
cloned
Policies
ReBRAC
0.1
0.0
0. 0.0
0.1
2:3
1.4
$3.2
103. 1.3
102.3
13.5
0.1 +00.1
5.6 0.9
40= 8.0
012 0.3
244 0.4
1.0 1.6
94.9
22.9
185. -.N
104
0.1 0.1
58 0.9
400 8.0
012 +
2.5 01
1.7
98.6
17.2
1858
105.
5.9 + 0.9
00-00 0.0
2.5 0.3
133 1.8
185.
0.5
105 2.4
6.0 1 0,9
00 0.0
26 0.3
1.5 1.8
106.0 0.5
1053 1.5
8.8
00= 0.0
266 113
14 1.9
106.1
1059 1.0
6.2 8M
=0=00 0.0
266 112
18
106.
1059 0.7
10
0.8
00-00 0.0
257 0.2
1.9
1061
106. -
11
0.8
0.0
2.7
1
12
15
186
17
00
28
27
106.3 0.3
106 11
18
00
28
106
19
67 0.6
00 0.0
28
28 199
106.3 0.3
106 21 01
20
67 + 0.5
00 00
28 0.3
- 1.9
106.3 0.2
106 +
101-
(a) door-human EOP.
(b) door-cloned EOP.
(c) door-expert FOP.
Figure TD3+BC, IQL and ReBRAC visualised Expected Online Performance under uniform policy selection on Door tasks.
26

---

Table 31: TD3+BC, IQL and ReBRAC Expected Online Performance under uniform policy selection on Hammer tasks.
human
cloned
expent
Policies
TD3-BC
IQI.
ReBRAC
TD3+BC
IQI.
ReBRAC
TD3+BC
IQL.
ReBRAC
0.2 0.0
1,6 0.4
0.3 1 0.2
1. 0.8
19 1.1
4,5 1. 5.5
60.2 1. 55.4
125.7
101.9
89.5
0.3 1.00
1,8 0.44
0.3 = 0.2
16 0.8
255 1.1
7.3+-611 6.1
9000 + 48.4
129.3 +
124.3
3 26.8
0.3 0.0
0,4
0.2
1. 0.7
28
9.1..622
105.7
1 18.1
129.
0.7
130.1
136
0.0
0:05
0.0
2.1
0.2
0.9
1.0
132.6
+.3
11
2.3
0.1
135
1.1
133.
12
2,3 0.3
0.1
160
0:1
1338
13
2.4 0.3
06 0.1
39
16.3 5.3
1300 0.1.1
1339
08
14
244 0.3
06 = 0.1
19 DA
16.7 5.2
1500 +.011
133. 0.7
15
244 0.3
06 0.1
391 DA
17.0 1 5.1
1300 0.1.1
134,9 0.6
16
2.4 0.3
06 0.1
4.0 DA
17. 4.9
130. 0.0.1
134. 0.6
17
244 0.2
06
40 UA
17.5 1 48
1300 1
134, 0.5
18
2.4-00
06
40 0.3
173 + 4.7
1500 +:00
134. 0.5
19
2.5
4010.
180
130.
134.1
20
255
0.110.1
401.03
183 1
13000.0..0
134.1
1.40
12522 5 5.033 6
(a) hammer- human EOP.
hammer-cloner EOP.
(c) hammer-expert EOP.
Figure 9: TD3+BC, IQL and ReBRAC visualised Expected Online Performance under uniform policy selection on Hammer tasks.
Table 32: TD3+BC, IQL and ReBRAC Expected Online Performance under uniform policy selection on tasks.
expert
Policies
TD3+BC
IQL
ReBRAC
TD3+C
IQL
ReBRAC
TD3-BC
IQL
ReBRAC
-0.2 0.1
0.2 :: 0.2
.0.1 0.1
.02 + 0.1
-0.0
0.1
0,5 0.8
21.4 43.2
106.0
1.4
73.5
0.2 3: 0.1
0.2 3: 0.2
0.1 0.1
0.2 1: 0.1
0.0
0.1
0,9 1 0.9
38.5 2: 51.9
106.8
1.0
96.0
27.4
0.1 1. 0.1
0.3 0.2
0.0
0.2 1 0.1
0.1
0.1
1.2 0.9
52.6 1. 51.0
107.1
0.8
100.7
15.9
-0.1 = U.1
0.3 0.2
-00 0.0
-1.1 + 0.1
0.1
1.4 0.9
63.7 = 53.
107.4 IL7
106.7
9.4
00.1 2 0.1
0.4 0.2
-00-00
1
0.1
16 2: 0.9
72.5 50.7
107
0.6
107.9 5.8
0.2
-00-00
0.1 0.1
1.7 + 0.8
107
10K.6 AA
0.0 0.0
0.1
0.5
1089
0.0
14
08- 0.0
0.1
9079
18
107
16
08 0.0
2,3
108.0 0.4
1099
17
0.6 0.2
040 0.0
0.2
2 IL5
10800 014
1099 08
18
0.6 2: 0.2
040 0,0
0.2 1.1
24 U.5
108.0
1099 - 03
19
0.6 + 0.2
00-
0.2 0.200.1
24 0.5
108
1100 +
20
0.6 0.1
00-
0.2
25 0.4
108.0
1100 1
27

---

2115
(a) relocate-human EOP.
(b) relocate-cloned EOP.
(c) relocate-expert EOP.
Figure 10: TD3+BC, IQL and ReBRAC visualised Expected Online Performance under uniform policy selection on Relocate tasks.
G D4RL tasks ablation
Table 33: ReBRAC ablations for halfcheetah tasks. We report final normalized score averaged over unseen training seeds.
Abluiin
raskes
media
EqUT
rediaz-oplo
TAlcopho
Auaage
TES+-C
110:1 22
963:2
002
468 2:33
Till-R" teed
55
643.8
580 H422)
RuDIRAC beyr
(-.0G)
164.1
24.59
66.1 8.8 (6.3))
5044 (4.55
KUBKAC acuaccally
11..2.4.11 4.1
845 0-00
7L8-2 (1:300099)
96. 27.1122u0))
2033 41-33-1111 096
RATRRAC ans lore buck
2100
153 761
60.0 07 (%)
13.6. 21311-91755
65.3
1.3:17788)
507 (uuS))
6130115955
Table 34: ReBRAC ablations for hopper tasks. We report final normalized score averaged over unseen training seeds.
Abaaiia
soken
paper
TUSADC TOG-IC sead
13. 18
35.
66.5 24.9
203
K0A4
193
180451
1W07 (110111
1962 23/4099)
RARRAC wiLL
123-2 20.
1D 06 (999))
112. 8-0 140.990
S1. 25.0-38 59
10:20 2.2 retudey
lape
(413999)
070
RABRAC
L... (4839))
1.3.8
(88.95)
27.7 23.4 (76.19)
100 (-75510)
74.55 1.
(054 B.IEEESI
1112
K5.1 1-31-11 (1g
100.5 (4977))
15
30
015.775
Table 35: ReBRAC ablations for walker2d tasks. We report final normalized score averaged over
unseen training seeds.
mypes
replas
metla
TDU+BBC paier
16:11.1
237.122
1803 =
11831 ULS
$18.2.5
TOSS-BC rand
191559)
1102 (7:1139)
WR4 (3300
15BC-1995
523
RUSMAA lowr
114-2:119 (46.2%)
MA2 08 6R1
(425))
S4.3
KANRAN mine peadly
1::09 689
993
66.3-
BeRlANN wis lagg hunds
56-5-022-70.00
64.8 1000
115.3 6-0 6-33--9991
1104 1104:02:0000
71.756-0
9:
Table 36: ReBRAC ablations for AntMaze tasks. We report final normalized score averaged over unseen training seeds.
Ablatos
dhare
medaraple:
Grens
chame
pagpe TDS-BC.
42444224
a... 8,5
TOO-NC sead
(4409)
19:300005
229 (42279)
RUBRACC whr
08 100-
LOS 00 100000500 510
02.8883 199379
153.8 Halida
13.0 100---00 (0107l
2188 H722
KABRAC
(1009)
00 (000-00000
(0100000))
2. 30r-3009)
1185-15511 1466
12. B8))
Belle whuuin poulty
10--1.14
1068))
1001)
607-:31
14.14
685 W0)
48468)
14 (59-6-0500
14 (1.99)
on 6.8 30300/6
28

---

Table 37: ReBRAC ablations for pen tasks. We report final normalized score averaged over 4 unseen training seeds.
Ablation
human
cloned
expert
Average
TD3+BC, paper
0.0
0.0
0.3
0.0
TD3+BC, TD3+BC, tuned
65.9 4: 24.6
78.1 5.7
144,9 7.5
96.3
77.6
18.5 (-23.9%)
78.1 1: 5.7 (3855%
144.9 1: 7.5 (-10.3%)
100.2 (-12.5%)
ReBRAC
(-22.9%)
(-7..%)
(-44.68))
(-45.8%)
ReBRAC w/o layer
89.1 = 14.7 (-12.6%)
106.7 = (+24.99)
147.2 5.7 (-6.0%)
114.3 (-0.3%)
actor penally
(-100.5))
(-99.3%)
(-100.08)
(-0000%)
ReBRAC w/o critic penalty
99.9 = 6.1 (-2.1%)
75.0 16.7 (-12.2%)
154.6 = 1.8 (-1.3%)
109 8 (-4.2%)
ReBRAC large batch
83.2 23.3 (-2.6%)
155.0 6.8
(-11.2%)
ReBRAC
102.0 10.8
85.4 4: 24.2
156.6
114.6
Table 38: ReBRAC ablations for door tasks. We report final normalized score averaged over 4 unseen training seeds.
Ablation
human
cloned
expert
Average
TD3+BC, paper TD3+BC, our
0.0
0.0
0.0
0.0
0.0 + 0.1
0.4 + 1.0
102.5 + 2.9
34.3
TD3+BC, tuned
0.0 (-)
0.4 + 1.0 (+100.00)
105.8 (+0.8%)
35.4 (+1.1%0
ReBRAC w/o LN
-0.1 -0.1 0.0
-0.3 1:3::000 0.0 (-250.0%)
106.0 = 0.8 (+1.0%)
35.1 (+0.3%)
ReBRAC layer
0.0
(-50.0%)
104.4 + 2.3 (-0.5%)
34.8 (-0.6%%)
ReBRAC w/o actor penalty
-0.1 + 0.1
0.0 (-100.06))
0.0 0.2 (-100.0%)
0.0 (-100.0%%
ReBRAC w/o critic penalty
0.0 + 0.0 (-)
0.1 #: 0.0 (-50.06%)
106.1 0.3 (+1.11%)
35.4 (+1.19)
ReBRAC large batch
- 0.1
0.3 (-50.0%)
106.1 (+1.1%)
35.3 (+0.9%)
ReBRAC
0.0 1
0.2 = 0.3
104.9 + 2.2
35.0
Table 39: ReBRAC ablations for hammer tasks. We report final normalized score averaged over unseen training seeds.
Ablation
human
cloned
expert
Avernge
TD$+BC, paper
0.0
ID3+BC,
0.3 + 0.4
1.1
127.0 0.4
42.8
TD3+BC, luned
0.3 1 0.4 (+50.0%)
1.1 L 1.1 (-80.0%)
127.0 1 0.4 (-5.3%)
42.8 (-8.1%)
ReBRAC
+ 0.0 (0.0%)
9.9 19.1
(-92.390
ReBRAC w/o layer
0.1 + 0.0 (-50.0%)
21.3 = 19.7 (+287.3%)
133.1 0.5 (-0.8%)
51.5 (+10.5%)
ReBRAC actor penalty
0.0 0.0 (-100.0%)
0.0 0.1 (-100.09)
0.0 (-100.0%)
0.0 (-100.0%)
ReBRAC critic penalty
(.50.0%)
1.9 (-65.5%)
134.1 1. 0.2 (-0.1%)
45.3 (.2.8%)
ReBRAC large batch
0.3 1 0.8 (+50.0%)
10.6 14.0 (+92.79)
133.4 1 0.5 (-0.6%)
48.1 (+3.2%
ReBRAC
- 0.2
5.5
134.2
46.6
Table 40: ReBRAC ablations for relocate tasks, We report final normalized score averaged over unseen training seeds.
Ablation
human
cloned
expert
Average
TD3+BC, paper TD3+BC,
0.0
-0.1
107.9 177.9-::00 0.6
35,9 35.9 (-0.5%)
TD3+BC, tuned
0.0 + (-)
-0.1 # 0.0 (-10..3)
107.9 = 0.6 (+1.22%)
ReBRAC w/o LN
+0.2
0.0 = 0.3 (-100.0%)
-0.1 + 0.0 (-100.1%)
-0.1 (-100.3%)
ReBRAC w/o layer
0.1 0.3
1.7 2 2.1 (-10.5%)
105.0 0 3.1
35.6 (-1.4%)
ReBRAC actor penalty
-0.1 (-)
0.0 = 0.0 (-1000%))
0.1 (-100.15)
(-100.0))
ReBRAC w/o critic penalty
0.1
1.9 1.9 (0.0%)
109.6
1.2 (+2.8%)
37.1 (+2.8%)
ReBRAC w large batch
0.0
0.1 = 0.2 (-94.7%)
109.6 0.9 (+2.8%)
36.5 (+1.1%)
ReBRAC
0.0 + 0.0
1.9 + 2,3
106.6 +: 3.1
36.1
29


Reviews:
[{"review_id": "jyKl2GeHyo", "reviewer": "Reviewer_7C4n", "summary": "In this work, a number of recent advancements are added to the minimalist TD3+BC baseline, and the authors found that the resulting algorithm leads to a new SOTA performance on D4RL benchmark with raw state and visual input. Extensive empirical results and ablations are provided and the authors show a dedication to fairness and reliability of their comparisons. ", "questions": "- The authors mentioned that to be fair, the competing methods have been extensively tuned, I wonder how much performance gain you are able to achieved for these competing methods compared to their reported results in their original papers? \n- when you tune for d4rl, do you fine-tune a different set of hyperparamter for each task? Is this done for all algorithms in the comparisons? ", "limitations": "Authors discussed the limitations of the work. ", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "**originality**\n- the paper studies existing methods, and the novelty of the work is main on its empirical findings and ablations. However, due to the extensive experiments and attention to fairness and details, these results can be considered novel findings. \n\n**quality**\n- quality is great, paper is well structured, important things are highlighted and performance changes are quantified. \n- technical details are given, things like computational speed discussed, code provided, overall good. \n- authors even tuned and tested on different seeds. \n\n**clarity**\n- paper is very clear and easy to read. \n\n**significance**\n- Finding a new sota baseline for offline RL setting, with a full set of ablations understanding the impact of each of its components can be a good and significant contribution. \n- Although no new algorithm is introduced, these results can be very helpful to researcher who want to really understand what is helping the performance (whether it's an algorithmic component or just a code hack or hyperpararmeter choice). ", "weaknesses": "**originality**\n- the paper focuses on an empirical study of existing methods, which will reduce the novelty a bit. \n\nOther than this, I think within its own scope, the paper is very good. No major concern. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review. Regarding your questions: \n\n> when you tune for d4rl, do you fine-tune a different set of hyperparamter for each task? Is this done for all algorithms in the comparisons?\n\nYes, we tune hyperparameters per-dataset for each method.\n\n> The authors mentioned that to be fair, the competing methods have been extensively tuned, I wonder how much performance gain you are able to achieved for these competing methods compared to their reported results in their original papers?\n\nHere are the scores over four training seeds. The \"paper\" label stands for hyperparameters taken from the original papers and the \"tuned\" label stands for the best hyperparameters we found. As you can see below, the gains from per-dataset hyperparameter tuning are significant. This is often done for new methods and missed for the baselines considered. We believe this is expected, as different datasets probably benefit from varied levels of penalization.\n\n| **Dataset**               | **IQL, paper** | **IQL, tuned** | **TD3 + BC, paper** | **TD3 + BC, tuned** |\n|---------------------------|----------------|----------------|---------------------|---------------------|\n| halfcheetah-random        | 9.4            | 19.6           | 2.2                 | 30.1                |\n| halfcheetah-medium        | 48.3           | 50.2           | 44.6                | 55.4                |\n| halfcheetah-expert        | 96.4           | 96.6           | 93.8                | 95.5                |\n| halfcheetah-medium-expert | 94.7           | 94.7           | 91.9                | 91.9                |\n| halfcheetah-medium-replay | 44.4           | 45.0           | 40.5                | 45.1                |\n| halfcheetah-full-replay   | 74.9           | 75.6           | 69.3                | 74.1                |\n| hopper-random             | 7.5            | 11.8           | 10.3                | 10.3                |\n| hopper-medium             | 67.5           | 67.5           | 53.2                | 57.6                |\n| hopper-expert             | 100.0          | 112.7          | 108.7               | 110.7               |\n| hopper-medium-expert      | 80.7           | 112.4          | 75.8                | 106.2               |\n| hopper-medium-replay      | 97.4           | 97.6           | 64.5                | 64.5                |\n| hopper-full-replay        | 104.4          | 108.4          | 49.9                | 106.2               |\n| walker2d-random           | 4.0            | 12.0           | 4.5                 | 4.5                 |\n| walker2d-medium           | 80.9           | 82.5           | 77.1                | 77.1                |\n| walker2d-expert           | 112.8          | 113.8          | 109.1               | 110.1               |\n| walker2d-medium-expert    | 111.7          | 112.4          | 108.9               | 110.2               |\n| walker2d-medium-replay    | 82.1           | 83.0           | 50.9                | 58.8                |\n| walker2d-full-replay      | 97.7           | 98.2           | 86.7                | 89.4                |\n| **Gym-MuJoCo avg**        | 73.0           | 77.4 __(+6%)__           | 63.4                | 72.0 __(+13%)__               |\n| antmaze-umaze             | 76.0           | 84.2           | 62.0                | 62.0                |\n| antmaze-umaze-diverse     | 59.5           | 75.0           | 48.0                | 48.0                |\n| antmaze-medium-play       | 69.7           | 70.2           | 0.0                 | 39.0                |\n| antmaze-medium-diverse    | 63.0           | 64.7           | 0.5                 | 18.5                |\n| antmaze-large-play        | 41.5           | 41.5           | 0.0                 | 0.2                 |\n| antmaze-large-diverse     | 22.5           | 29.5           | 0.5                 | 0.5                 |\n| **Antmaze avg**           | 55.3           | 60.8 __(+10%)__           | 18.5                | 28.0 __(+80%)__                |\n| pen-human                 | 87.1           | 93.6           | 65.9                | 77.6                || pen-cloned                | 73.2           | 89.4           | 78.1                | 78.1                |\n| pen-expert                | 130.9          | 134.6          | 144.9               | 144.9               |\n| door-human                | 3.5            | 7.0            | 0.0                 | 0.0                 |\n| door-cloned               | 1.0            | 2.8            | 0.4                 | 0.4                 |\n| door-expert               | 106.0          | 106.5          | 102.5               | 105.8               |\n| hammer-human              | 1.5            | 2.5            | 0.3                 | 0.3                 |\n| hammer-cloned             | 1.4            | 4.1            | 1.1                 | 1.1                 |\n| hammer-expert             | 127.7          | 130.0          | 127.0               | 127.0               |\n| relocate-human            | 0.0            | 0.6            | 0.0                 | 0.0                 |\n| relocate-cloned           | 0.0            | 0.2            | -0.1                | -0.1                |\n| relocate-expert           | 106.0          | 108.2          | 107.9               | 107.9               |\n| **Adroit avg**            | 53.1           | 56.6 __(+6%)__           | 52.3                | 53.5 __(+2%)__                |\n----------"}, {"review_id": "0opXgpG3cE", "reviewer": "Reviewer_HdeU", "summary": "This paper revisits several minor design choices in recent offline RL literature and equips a standard method TD3+BC (or more generally BRAC) with these designs to attain a strong baseline for offline RL with state-of-the-art performance on both D4RL and V-D4RL benchmarks. These critical designs include deeper networks, LayerNorm, larger batches, decoupled actor-critic penalty, and adjusted discount factor. Extensive benchmarking experiments and ablation studies are conducted across a range of domains.", "questions": "1. If I understand correctly, BRAC has adopted actor penalization and critic penalization at the same time. Indeed, BRAC-v in the original paper adds a penalty to both actor and critic learning objectives (Eq. 6 and 7 in the BRAC paper). Thus, some assertions in this paper (Lines 65 and 111) are incorrect and should be revised. Nevertheless, the idea of decoupling actor and critic penalty has not been explored to my knowledge.\n2. Following the above question, an ablation study comparing decoupled actor-critic penalties and coupled ones (using only one hyperparameter) is missing in Table 6. Can authors present these results?\n3. What do you mean by \"Since our approach principally builds upon TD3+BC, the differences in their performances should be considered the most important ones.\" in line 124?\n4. In line 185, the authors claim, \"ReBRAC outperforms TD3+BC not because of different implementations or actor regularization parameter choice\". However, in my opinion, the differences between  ReBRAC and TD3+BC are indeed implementation choices rather than algorithmic innovations. I understand that the efforts of building ReBRAC upon TD3+BC are non-trivial, but the authors should consider revising this sentence to make it much clearer.\n5. Implementation details for vision-based V-D4RL tasks, including CNN architecture to encode visual observations, are missing in both the main paper and the appendix. Although I have found the implementations in supplementary source code, I recommend clarifying these details in a future revision of the paper.\n6. TD3+BC uses state normalization on Gym-MuJoCo tasks. Did the authors include this implementation detail in their experiments for both TD3+BC and ReBRAC? Since in the TD3+BC paper, state normalization does not impact significantly, in my opinion, it is okay not to include it, but I recommend the authors clarify this in the paper. ", "limitations": "This work has discussed its limitations and future work in the conclusion. There does not seem to be any negative social impact of this paper that should be discussed. ", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. Careful look into detailed design choices from a large amount of literature, which naturally motivates the proposed solution\n2. Extensive experiments across D4RL’s Gym-MuJoCo, AntMaze, and Adroit tasks, and even vision-based V-D4RL tasks\n3. Great efforts for a fair comparison with baseline methods, including hyperparameter tuning for baselines and measuring Expected Online Performance\n4. Well written\n\nOverall, I really appreciate the contribution to the offline RL community by revisiting a strong yet overlooked baseline method, BRAC.", "weaknesses": "1. A probably incorrect assertion to a technical detail of BRAC (see Question 1 below)\n2. Limited technical novelty (but I think it doesn't matters for this paper focusing on a retrospective analysis)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank your for the review and identified weaknesses, we address them and your questions as follows:\n\n> If I understand correctly, BRAC has adopted actor penalization and critic penalization at the same time. Indeed, BRAC-v in the original paper adds a penalty to both actor and critic learning objectives (Eq. 6 and 7 in the BRAC paper). Thus, some assertions in this paper (Lines 65 and 111) are incorrect and should be revised. Nevertheless, the idea of decoupling actor and critic penalty has not been explored to my knowledge.\n\nYou're correct in your assessment, we will revise our statements in the final version of the paper.\n\n> Following the above question, an ablation study comparing decoupled actor-critic penalties and coupled ones (using only one hyperparameter) is missing in Table 6. Can authors present these results?\n\nYes, please, see the attached .pdf file in the official comment. Notably, performance with coupled penalties drops by a small margin when compared to decoupled ones. We will make sure to include this information in the appendix, as this significantly reduces the hyperparameter space search and should improve the expected online performance on smaller budgets when compared to other methods.\n\n> What do you mean by \"Since our approach principally builds upon TD3+BC, the differences in their performances should be considered the most important ones.\" in line 124?\n\nThis was meant to highlight that ReBRAC improvements do not simply come from the actor-regularization hyperparameter search for TD3+BC. Hopefully, this clarifies what we meant, we will make sure to update the corresponding text. \n\nMoreover, we also included results for experiments where we add one design choice at a time to the TD3+BC to further demonstrate the preference of the found combination of choices.\n\n> Implementation details for vision-based V-D4RL tasks, including CNN architecture to encode visual observations, are missing in both the main paper and the appendix. Although I have found the implementations in supplementary source code, I recommend clarifying these details in a future revision of the paper.\n\nSure, we will provide a clear reference to the original architecture appearence and will elaborate on it in the appendix.\n\n> TD3+BC uses state normalization on Gym-MuJoCo tasks. Did the authors include this implementation detail in their experiments for both TD3+BC and ReBRAC? Since in the TD3+BC paper, state normalization does not impact significantly, in my opinion, it is okay not to include it, but I recommend the authors clarify this in the paper.\n\nAs you rightfully noted, state normalization does not impact performance much (this is what we also observed in the preliminary experiments), therefore we decided not to use it -- keeping in mind that this may complicate further application in the offline-to-online setting which we did not cover in the original submission. To further illustrate the use, we conducted a set of experiments in the offline-to-online setting and found ReBRAC to be a strong competitior, especially on the Adroit domain (for results, again, see the attached .pdf)."}, {"review_id": "aVrGewIpK2", "reviewer": "Reviewer_8F2Q", "summary": "This paper revisited recent methods in the offline RL area and discussed how different design choice impact offline RL methods' performance. In particular, the authors focus on four hyper-parameter choices (i) number of network layers, (ii) using LayerNorm, (iii) batch size and (iv) discounting factor $\\gamma$, as well as one algorithmic choice (v) using actor/critic regularization, and conducted comprehensive experiments to show the importance of different factors. Their method, ReBRAC, achieved strong empirical performance on 51 datasets.", "questions": "I have no further question as it was clearly written.", "limitations": "See above.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents comprehensive experiments.\n\n2. The proposed method demonstrates strong empirical performance in the conducted experiments.\n\n3. The paper is clearly written.\n\n4. This work shows great engineering values and has the potential to offer valuable guidance in design choices for the offline RL community.", "weaknesses": "1. A combination of existing techniques/tricks. In particular, most of the design choices (i) number of network layers, (ii) using LayerNorm, (iii) batch size and (iv) tuning $\\gamma$ are hyper-parameters or standard deep NN techniques. \n\n2. Table 6: ReBRAC w/o LN has a 38.0 average score over all domains, which is noticeably lower than the authors' reproduction of TD3+BC (52.2), leading to my speculation that adding LN alone could achieve great empirical performance. It would be nice if TD3+BC w/ LN alone could be shown. (For reference, Rainbow [1] shows the performance of all individual choices in their Figure 1 and none of them were close to Rainbow.)\n\n3. Table 6: Critic penalty only gives marginal improvement. As the batch size, using LN, number of layers, and tuning $\\gamma$ are mostly hyper-parameters rather than RL algorithm designs, marginal benefits from the critic penalty further limit its algorithmic contribution.\n\n4. According to table 1, SAC-RND seems to have all five components. It would be nice if the authors could elaborate on the difference between SAC-RND and ReBRAC, for example, in section 2 or 3.\n\n\nOverall I find this paper has limited novelty, especially considering some components mentioned are merely hyper-parameters. However, I find it might be a significant contribution to achieve superior performance on 51 datasets, providing potential implementation guidance to the offline RL community. I therefore voted for a boarderline acceptance with a low confidence score.\n\n\n[1] Rainbow: Combining Improvements in Deep Reinforcement Learning. https://arxiv.org/pdf/1710.02298.pdf", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First of all, we would like to thank you for your time reviewing our paper and valuable comments. We believe that the main concern is the limited novelty, which we address first, and then we move to more precise limitations.\n\n------------\n\n### On limited novelty\nWe appreciate the reviewer's comments and accept that while our research represents a judicious amalgamation of existing techniques and hyperparameters, the novelty lies in the distinctive combination and application.\n\nIndeed, insights into performance enhancement – whether through algorithmic or code tweaks or hyperparameter tuning – carry intrinsic merit. Our work served to highlight certain empirical results that may otherwise have remained undisclosed. For instance, the fact that our chosen blend of design choices significantly outperforms other methods is noteworthy. Moreover, it was hitherto unknown that this category of methods performs exceptionally well in challenging environments, such as AntMaze. Previously, the prevalent belief favoured the efficacy of methods like IQL or CQL with ensembles (MSG).\n\nHence, if we scrutinize papers like Rainbow [1] or Dreamer-v3 [2], which are clever syntheses of known elements, should we consider their innovation limited? We hope that this perspective will elucidate our paper's aim and its inherent merits.\n\nWe have also augmented our submission with offline-to-online experiments (refer to the .pdf file). These reveal ReBRAC to be a potent baseline in this context, further expanding our contribution's dimensions. Therefore despite its perceived limited novelty, we propose that our paper provides essential guidance for offline RL implementation, thereby contributing significantly to this field of research.\n\n---------------\n\n### On other outlined weaknesses\n\n> Table 6: ReBRAC w/o LN has a 38.0 average score over all domains, which is noticeably lower than the authors' reproduction of TD3+BC (52.2), leading to my speculation that adding LN alone could achieve great empirical performance. It would be nice if TD3+BC w/ LN alone could be shown. (For reference, Rainbow [1] shows the performance of all individual choices in their Figure 1 and none of them were close to Rainbow.)\n\nThis is a great observation and suggestion. To address it, we include Rainbow-like experiments, i.e., adding each individual choice separately to the TD3+BC (see the attached .pdf file). Notably, LN and deeper networks improve performance of the TD3+BC but still lack far behind the ReBRAC.\n\n\n> According to table 1, SAC-RND seems to have all five components. It would be nice if the authors could elaborate on the difference between SAC-RND and ReBRAC, for example, in section 2 or 3.\n\nSure, we will add elaboration on the difference in the final version. The main difference is that SAC-RND uses penalization based on the RND bonus in both actor and critic, which requires pre-training a predictor network.\n\n--------------\n\n### References\n\n[1] Hessel, Matteo, et al. \"Rainbow: Combining improvements in deep reinforcement learning.\" Proceedings of the AAAI conference on artificial intelligence. Vol. 32. No. 1. 2018.\n\n[2] Hafner, D., Pasukonis, J., Ba, J., & Lillicrap, T. (2023). Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104.\n"}, {"review_id": "2yxd3RPLJT", "reviewer": "Reviewer_DJeX", "summary": "This paper proposes an offline RL algorithm based on TD3+BC and BRAC, integrating popular design elements. The proposed method achieves higher scores on D4RL and V-D4RL datasets.", "questions": "1.\tTable 6: ReBRAC w large batch. What does it mean?\n2.\tThe performance improvements of ReBRAC seem marginal, especially in challenging tasks (door-human, door-cloned….)\n3.\tTable 6: it seems that the critic penalty is not crucial. Is it a universal conclusion, or just due to a lack of hyperparameter tuning?\n4.\tCan authors provide comparisons of computation costs?", "limitations": "Please see the weaknesses.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The idea of exploring the popular design elements in offline RL algorithms is interesting. The authors make a great effort in conducting confirmatory experiments including various ablation studies.", "weaknesses": "Overall, I see the proposed method as an integrated method of existing algorithms. From my perspective, the novelty and contributions are weak. It may be more insightful if the author put more emphasis on the analyses of existing design elements rather than combining them into one new method. I do not see many insightful analyses in the current manuscript.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review. We believe that the main concern is the limited novelty, which we address first, and then we move to more precise limitations.\n\n----------\n\n### On limited novelty\n\nWhile we agree that the paper may seem to feature constrained technical novelty, given it doesn't propose new algorithmic techniques but rather builds upon existing ones, we see merit in studies that clarify determinants of performance, be they algorithmic elements, coding strategies or hyperparameter selections. Such work can, in its own right, offer retrospective novelty.\n\nThis invites us to ponder over some questions: Were we aware that this unpretentious combination of design choices could eclipse the efficiency of all other methods to this extent? Did we know prior to our study that this kind of method could perform exceptionally well in environments requiring strong stitching competencies (AntMaze, as per our understanding, has generally seen dominance from methods like IQL or CQL with ensembles (MSG))? Can we consider such seminal works as Rainbow [1] or Dreamer-v3 [2] with limited novelty, merely because they integrate known elements?\n\nWe hope this dialogue serves to highlight the scope, relevance, and intrinsic merits of our study. As for the analysis of extant design elements, we have taken measures to reference a wealth of existing literature in our description, thereby providing a comprehensive overview. For instance, we pointed out how LayerNorm is instrumental in mitigating catastrophic q-value extrapolations, and that large batches can enhance and balance convergence given adequate data. Even though a more thorough analysis of the precise volume of data needed could constitute a separate study, we admit that it was beyond our current work's scope. The same goes for the increased discount factor that could use more exploration but was not the focus of our paper.\n\nWe hope that our response brings more clarity and understanding to the novelty and significance of our research. We look forward to any further comments that could help us improve and refine our work.\n\n--------------------\n\n### On Questions\n\n> Table 6: ReBRAC w large batch. What does it mean?\n\nReBRAC with the use of large batch (the definition of the large batch can be found in the paper, essentially, the value is fixed to the 1024). We found the use of large batches to be detrimental on AntMaze datasets.\n\n> The performance improvements of ReBRAC seem marginal, especially in challenging tasks (door-human, door-cloned….)\n\nWe highly value your feedback concerning the perceived marginal performance improvements of ReBRAC, especially in challenging tasks. \n\n(1) Although it may seem that the performance gains of our method over SAC-RND on Gym-MuJoCo are minimal, it is essential to note that ReBRAC achieves this level of efficiency sans the requirement for an additional network. Moreover, it surpasses other contenders by a minimum of 10%.\n\n(2) With regard to the AntMaze domain, ReBRAC outperforms the nearest competitor by a margin of 26%, which we believe demonstrates notable superiority.\n\n(3) In the Adroit domain, ReBRAC exhibits an enhanced performance level, outpacing the closest competitor by an average of 9%.\n\n(4) Furthermore, in the V-D4RL test, ReBRAC once again outperforms the nearest competitor, this time by a margin of 16%.\n\nWhile we acknowledge that we were not able to set a new benchmark in the most challenging tasks in the offline setting, we conducted additional experiments in the offline-to-online setting during the rebuttal phase. These tests revealed that our method exhibits superior performance in the Adroit domain and aligns with Cal-QL in the AntMaze domain. We invite you to refer to the attached .pdf for additional details.\n\n> Can authors provide comparisons of computation costs?\n\nExtensive computational costs comparisons can be found in the appendix. Thank you for noticing, we will add a reference to it in the main text.\n\n> Table 6: it seems that the critic penalty is not crucial. Is it a universal conclusion, or just due to a lack of hyperparameter tuning?\n\nWe are not sure what you mean by the lack of hyperparameter tuning in this specific case, could you elaborate on that?"}]

Author rebuttal:
We would like to thank the reviewers for their work, hopefully, we provided a sufficient level of response to all, if not, we're open to continue our discussion. Here, we include additional results requested by the reviewers explicitly or implicitly so in order to provide comprehensive empirical results for our answers:
- Adding design elements separately to the TD3+BC
- Ablating decoupled penalization
- Offline-to-online experiments

Meta-review: The paper explores simple modifications to improve offline RL algorithms. The paper is notable for its extensive and well-designed experiments provide a number of insights and advancements.
Decision: Accept (poster)