CORL: Research-oriented Deep Offline Reinforcement Learning Library
Authors: Denis Tarasov
Alexander Nikulin
Dmitry Akimov
Vladislav Kurenkov
Sergey Kolesnikov
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Offline Reinforcement Learning
TL;DR: Open-source offline and offline-to-online reinforcement learning library

Abstract: CORL is an open-source library that provides thoroughly benchmarked single-file implementations of both deep offline and offline-to-online reinforcement learning algorithms. It emphasizes a simple developing experience with a straightforward codebase and a modern analysis tracking tool. In CORL, we isolate methods implementation into separate single files, making performance-relevant details easier to recognize. Additionally, an experiment tracking feature is available to help log metrics, hyperparameters, dependencies, and more to the cloud. Finally, we have ensured the reliability of the implementations by benchmarking commonly employed D4RL datasets providing a transparent source of results that can be reused for robust evaluation tools such as performance profiles, probability of improvement, or expected online performance.

CORL: Research-oriented Deep Offline Reinforcement Learning Library
Denis Tarasov Tinkoff den. tarasov@tinkoff. ai
Alexander Nikulin Tinkoff a. w p.nikullin@tinnoff....mm
Dmitry Akimov Tinkoff
d. akimov@tinkoff. ai
Vladislav Kurenkov Tinkoff
Sergey Kolesnikov Tinkoff
v. w.kurenkor@tinkoff.oo .ai
s. kol.kllsoooo@ttikkfff
Abstract
CORI P an open-source library that provides thoroughly benchmarked single-file implementations of both deep offline and offline-to-online reinforcement learning algorithms. emphasizes a simple developing experience with straightforward codebase and modern analysis tracking tool. In CORL, we isolate methods implementation into separate single files, making performance-rellvan details easier to recognize. Additionally, help log metrics, hyperparameters, dependencies, and more to the cloud. Finally, we have ensured the reliability of the implementations by benchmarking commonly employed D4RL datasets providing a transparent source of results that can be reused for robust evaluation tools such performance profiles, probability of improvement, or expected online performance.
experiment tracking feature is available
Introduction
Deep Offline Reinforcement Learning (Levine et al. (2020) has been showing significant advancements
numerous domains such as robotics (Smith et al. [022]] Kumar et al., 2021). autonomous driving (Diehl et al. 2021) and recommender systems (Chen et al. 2022). Due to such rapid development, many open-source offline RI solutions emergedd well-known offline RI. techniques in different fields. On the one hand, they introduce offline RL. algorithms standard interfaces and user-friendly APIs, simplifying offline RL methods incorporation into existing projects. On the other hand, introduced abstractions may hinder the learning curve for newcomers and the ease of adoption for researchers interested developing new algorithms. One needs to understand the modularity design (several files on average), which (1) can be comprised of thousands lines of code or (2) can hardly for a novel method
help RI. practitioners understand
improve
In this technical report, we take different perspective on an offline RL library and also incorporate emerging interest the affline-to-online setup. We propose CORL (Clean Offline Reinforcement Learning) minimalistic and isolated single-file implementations of deep offline and offfine-to--nilee RI. algorithms, supported by open-sourced D4RL (Fu al., (020) benchmark results. The uncom- plicated design allows practitioners to read and understand the implementations of the algorithms straightforwarddy Moreover, CORL supports optional integration with experiments tracking tools such Weighs&Biasse (Bewwald 2020), providing practitioners with convenient way to analyze
CORL CORRL httpe///ggttaa.ooo/aaaaaaaaaaaaa http://ghhhacccom/aaaaae/agggges
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

the results and behavior of all algorithms, not merely relying on a final performance commonly reported in papers.
We hope that the CORL library will help offline RI newcomers study implemented algorithms and aid the researchers in quickly modifying existing methods without fighting through different levels of abstraction. Finally, the obtained results may serve as reference point for D4RL benchmarks avoiding the need re-implement and tune existing algorithms' hyperparameters.
Yaml Configuration File
Single-File Implementation
Experiment Tracking Log
environment params algorithm params
AWAC / BC CQL / EDAC /QLL/S/-/////////// TD3+BC
Wandb logs Wandb reports
python http:///wd....tooopeeeeeeeaaraaa -num-epochs=50
The illustration of the CORL library design. Single-file implementation takes yaml configuration file with both environment and algorithm parameters to run the experiment, which logs all required statistics to Weights& Biases (Biewald 2020).
Figure 1:
2
Related Work
Since the Atari breakthrough (Mnih et al.
2015), numerous open-source RL frameworks and libraries
have been developed over the last years:
Dhariwal et al. 2017 Hill al. 2018 Castro al. 2018
Gauci et al. 2018 Keng & Graesser 2017 garage contributors 2019 Duan et al. [2016] Kolesnikov
Hrinchuk 2019; Fujita et al. 2021 Liang al. 2018 Fujita al. 2021 Liu et al. 2021 Huang
et al. 1021 Weng et al.) 2021; Stooke & Abbeel, 2019). locusing on different perspectives of
the RL. For example,
baselines (Hill et al.) 2018) provides many deep RL implementations
that carefully reproduce results to back up RL practitioners with reliable baselines during methods comparison. On the other hand, Ray (Liang al.) 2018) focuses on implementations scalability and production-friente usage. Finally, more nuanced solutions exist, such Dopamine (Castro et al., 2018), which emphasizes different DQN variants, or ReAgent (Gauci al. 2018), which applies RL to the RecSys domain.
At the same time, the offline RL branch and especially offline-to-online, which we are interested in this paper, are not yet covered as much: the only library that precisely focuses on offline RL setting is d3ripy (Takuma Seno 202T). While CORL also focuses on offline RL methods (Nair
et al., 2020; Kumar et al., 2020 Kostrikov et al. 2021 Fujimoto & Gu 2021 An et al.) 1 Chen al. (202)) similar d3ripy. takes different perspective on library design and provides non-modular independent algorithms implementations. More precisely, CORL does not introduce additional abstractions to make offline RL more general but instead gives an "easy-to-hack" starter kit for research needs. Finally, CORL also provides recent offline-to-onliee solutions (Nair et al.
2020, Kumar et al., 2020; Kostrikov et al., 202T: Wu et al. 2022] Nakamoto et al.) 2023
Tarasov
al. 2023) that are gaining interest among rescarchers and practitioners.

---

Although CORL does not represent the first non-modular RL library, which is more likely the CleanRL (Huang al. ) case. has two significant differences from its predecessor. First, CORL is focused on offline and offline-to-online RI., while CleanRI. implements online RI. algorithms. Second, CORL. intends to minimize the complexity of the requirements and external dependencies. To be more concrete, CORL does not have additional requirements with abstractions such as stable-boselines (Hill et al. 2018 or envpool (Weng et al. 2022) but instead implements everything from scratch in the codebase.
3 CORL Design
Single-File Implementations
Implementational subtleties significantly impact agent performance in deep RL (Henderson al. 2018 Engstrom et al., 2020 Fujimoto & Gu 202T). Unfortunately, user-friendly abstractions and general interfaces, the core idea behind modular libraries, encapsulate and often hide these important nuances from the practitioners. For such reason, CORL unwraps these details by adopting single-file implementations. To be more concrete, we put environment details, algorithms hyperparameters. and evaluation parameters into a single file For example, we provide
wmw_percent_bc.pp (404 LO as baseline algorithm for offline RL methods comparison,
Ld3 be._b (511 LOC) as a competitive minimalistic offline RL algorithm FFujimoto & Gu 2021
dt.py (540 LOC) proach (Chen et al. 2021)
example of the recently proposed trajectory optimization ap-
Figure deppccs an overall library design. To avoid cver-comllicated offline implementations, we treat offline and offline-to to-online versions of the same algorithms separately. While such design produces code duplications among realization, it has several essential benefits from the both educational and research perspective:
Smooth learning curve. Having the entire code in one place makes understanding all its aspects more straightforward. In other words, one may find it easier to dive into 540 LOC of single-file Decision Transformer (Chen et al. 202I) implementation rather than 10+ files of the original implementatiorff
Simple prototyping. As we are not interested in the code's general applicability, we could make it implementation-speen sspeccff......siii.... Such design also removes the need for inheritance from general primitives or their refactoring, reducing abstraction overhead to zero. At the same time, this idea gives complete freedom during code modification.
Faster debugging. Without additional abstractions, implementation simplifies to a single for-loop with global Python name scope. Furthermore, such flat architecture makes accessing and inspecting any created variable easier during training, which is crucial in the presence of modifications and dehugging.
Configuration files
Although it a typical pattern use a command line interface (CLI) for single- experiments the research community, CORL slightly improves it with predefined configuration files. Utilizing YAML parsing through CLI, for each experiment, we gather all environment and algorithm hyperparameters into such files so that one can use them as an initial setup. We found that such setup (1) simplifies experiments, eliminating the need to keep all dggrrttm-enviromme---eee and (2) keeps it convenient with the familiar CLJ approach.
parameters in mind,
"We follow the PEP8 style guide with maximum line length of 89, which increases I.OC bit.
5 LLines Code
Original Decision Transformer implementation:

---

Experiment Tracking
Offline RL evaluation another challenging aspect of the current offline RL state (urenkov & Kolesnikov 2022). To face this uncertainty, CORL supports integration with Weights&Biases [0iewald[ a modern experiment tracking tool. With each experiment, CORL automatically saves (1) source code, (2) dependencies (requirements..x)) (3) hardware setup, (4) OS environment variables, (5) hyperparameters, (6) training, and system metrics, (7) logs (stdout, stderr). See Appendix B for an example.
Although, Weights& Biases a proprietary solution, other alternatives, such as Tensorboard (Abadi et al.] 2013) or Aim (Arakelyan et al. (020), could be used within a few lines of code change. It is also important to note that with Weights& Biases tracking, one could easily use CORL with sweeps or public reports.
We found full metrics tracking during the training process necessary for reasons. First, it removes the possible bias of the final or best performance commonly reported in papers. For example, one could evaluate offline RL performance as max archived score, while another uses the average scores over N (last) evaluations (Takuma Seno, 2021). Second, it provides an opportunity for advanced performance analysis such as EOP (Kurenkov & Kolesnikov 2022) RLiable (Agarwal al.) 202T). In short, when provided with all metrics logs, one can utilize all performance statistics, not merely relying on commonly used alternatives.
Benchmarking D4RL
4.1
Offline
In our library, we implemented the following offline algorithms: N% Behavioral Cloning (BC), TD3
+ BC (Fujimoto & Gu 2021). CQL (Kumar et al. 2020), IQL Kostrikov et al. 202T). AWAC (Nair let al. 2020). ReBRAC (Tarasov al. (023), SAC-N, EDAC (An et al., (021), and Decision Transformer (DT) (Chen et al. 2021). We evaluated every algorithm on the D4RL benchmark (Fu cet al. 2020), focusing on Gym- MuloCo, Maze2D, AntMaze, and Adroit tasks. Each algorithm was run for one million gradient steps and evaluated using ten episodes for Gym-MuJoCo and Adroit tasks. For Maze2d, hyperparameters proposed in the original works (see (pppeddix D for details) as much as possible.
100 evaluation episodes. In experiments, tried to rely the
The final performance is reported in Table 1 and the maximal performance in Table 2 The scores are normalized to the range between 0 and 100 (Fu et al., 2020). Following the recent work by Takuma Seno (202I), report the last and best-obtained scores to illustrate each algorithm's 's potential performance and overfitting properties. Figure shows the performance profiles and probability of improvement of ReBRAC over other algorithms (Agarwal et al. 2021). See Appendixx A for complete
training performance graphs.
Based on these results, we make several valuable observations. First, ReBRAC, IQL and AWAC are the most competitive baselines in offline setup on average. Note that AWAC is often omitted in recent works.
Observation 1: ReBRAC, IQL and AWAC are the strongest offline baselines on average.
Second, EDAC outperforms all other algorithms on Gym-MuJoCo by a significant margin, and to our prior knowledge, there are still no algorithms that perform much better these tasks. SAC-N shows the best performance on Maze2d tasks. However, simultancously, SAC-N and EDAC cannot solve AntMaze tasks and perform poorly in the Adroit domain.
Observation 2: SAC-N and EDAC are the strongest baselines for Gym-MuloCo and Maze2d, but they perform poorly on both AntMaze and Adroit domains.
Third, during our experiments, we observed that the hyperparameters proposed for CQL in Kumar et al. (2020) do not perform as well as claimed on most tasks. CQL is extremely sensitive to the
7 NN is percentage of best trajectories with the highest return used for training. We omit the percentage when is equal to 100. "Except SAC-N. EDAC, and DT due their original hyperparameters. See See Appendix D for details.

---

27%
AM6C
T22+
Algorithm Y
Algorithm X
SAC-N
ReBRAC
10%
COL
Flacc
ReBRAC
BC
2.50
ReBRAC
DT
RebRAC
103+8C
BeDRKC
O0L
REBRAC
SAC N
REBRAC
EDNC
ReBRAC
AWAC
ReBRAC
10L
120
0.65
0.70 0.75
0.00
0.85
0.90
0.95
PIX 1)
(b)
Figure 2: (a) Performance profiles after offine training (b) Probability of improvement of ReBRAC to other algorithms after offline training. The curves (Agarwal et al. (02I) are for D4RL benchmark spanning Gym-MuJoCo Maze2d, AntMaze, and Adroit datasets.
Table 1: Normalized performance of the last trained policy on D4RL averaged over 4 random seeds.
batchwech radiam-c2
0.10
46.95
11.37
17.04
18.11 13.3
53 TD
1-26
5599 7.55
9110-1100 2.15
9878 1.04
00.00 0.51
U6.3 1442
W4.5 1653
100.0 3
68.66 1-551
196.6 0.66
41:54 1.05
hoppo madiam redlimi-
2981-2:30 2.00
70.43 1133--5511
62.514221.3
00.19 2.07
95.11 5.27
07.23 6.30
04.00
6.55
10635---0 9-775
hogges eadion experis
42:50-4-400
16.5
101 11:0
101
10.91
107
7:90
10925
101
19
105 19 19.06
199.44 0-33
03.23 16.34
62.34 5:17
43.73
08.53 20.10
19175 5.24
19191 3.87
0.77
$7.47
$3.36
07.67
vn aekketbane mod -edea--------------
21:0 10.15
54.55 0-15226434
4.00 35.82
20.82 3.35
75.99
15.22
82.15
3.05
$4.25
1.25
19,09 2.30
91.10:22213 2.15
99.36-
Mynn-M any
$1.40
09.25
76.45
70.39
75:35
$1.05
$6.7%
$3.55
$1.52
75.84
0.992:3.11 3.15
14.25 2:13
31.43 2 36.35
$4.59
35.54
96.11
0.55
54.35
2-72
185.11
31,7
13,01
18-72
13.71-226.31
3.5.
4.74
1.33.11500
2611
31538
25.55
W.30
01.75
76.11
61.77
$5.00
33.00
15.66
Mannd
1:13
12.99
61.93
33.45
$41.22
25.48
91.3
11200 2-30
0:000:0:0
57.00
aeraao eediiie play 43
000 1:000
0
035 1..11
0100 0000
6535 11.51
66.5
11.1
Cm 1m0m
0:011-300
0.00 0.00
0.20
230
30011-116
800-1030
030 1:55
13.5 76:0
010-1-10 1-10
0.00 0.00
0:002:010
0.3 2:33
0.00 0000
0.00 0000
21.5 15.34
30.25
3,00
6400
1,45
CJ00 010
0:002:0.0
0.002:0000
11:00:1 4-4
1531 16:48
78.09 -.71
6:56
67
jesespon
108 02808-4733 7.78
14 9.96
12333-23333 24.30
199.6 1.47
1141 3.34
13505 9.11
143.66 6335
57.1 15.95
15:55 9-961
116 16 1-33
dbendrasd-1
0.09 0.000
0.29 1.95
433 405
10:33
41:3
1.55
4.35
2.6
1.126
domsapeil
105 0.00
/04.4:2 1.46
.0.3
10c.3
JU32 0022
10665 1:25
10637
= (29
4,35 810
106:29-2:..1
barrer
4::0:::0:0
0:34
0.55
0.90
0.26
0.06 1:900
0:75 1-11--1115
0:15
barzan-cheedrr
0.2 2.0.1
0:1 0::2:::00
0.23 2.22
1220-2 2:11
1130:21000
1.30 = (69
5/00 1.75
0.1-2:011
0.19-2-0.10
1 0.35
0.35
1M66
0.77
35
0.44 0.00
.0.44:00:0
-0.20-2:001
0143 0,33
0.0 0/3
0.12 0/4
6.66 0.30
2:31 2:31-2:9.11
-3.1-=00.11
0.05 0.00
1.0.8
4:00
.036 16:5
0001
4134
bill
1/6
0.17 1.0.4
0.16 1.06
Advil ang Total
48.18
42.69
10.00
35.11
1.59
53.03
68.39
12.43
18.78
49.1
41.6
17.11
76.M
11.16
choice of hyperparameters, and we had to tune them lot to make it work on each domain (see Table 7). For example, AntMaze requires five hidden layers for the critic networks, while other tasks' performance suffers with this number of layers. The issue of sensitivity was already mentioned in prior works as well (An et al. 2021
Ghasemipour et al. 2022).
Observation 3: CQL is extremely sensitive to the choice of hyperparameters and implementation details.
Fourth, we also observe that the hyperparameters do not always work the same way when transferring between Deep Learning frameworks [0] Our implementations of IQL and CQL use Py but the
parameters from reference JAX implementations sometimes strongly underperform (e.g., IQL on Hopper tasks and CQL Adroit).
See also http::////ggtuu.ooo/////aaaaaaaa
https://gitte
5

---

Table 2: Normalized performance of the best trained policy on D4RL averaged over random seeds.
$3,00 22 ML:A
45:0022 5-133
46.0 2.011
5:312:011 9:.332:023
47.0 2:0
48.66 2 -U77
00,02 BA4
12.21 14:31
06.22 22 M192
42.33 0.33
79,99
5.30
94.33
96.39 0:37
9734
2.17
19.882.0.15 0.15
1.30
111.5
14442
100.2
90.40 2:231
toppermaaders in
06:31 2:43
7184 11:77
2004 1:18
1039 1-66
1.10
100-14 1.16
101 11.20
105.6 11114
13:13 1.14
M463
1 11108
113333 0.35
11533 1.0.4
183.6 1.0.0
113384
1.3/66
11518 0..96
113.6
0:11
111.5
01155-01555 16.5
111.601.00
111 1:331
wellerde ewdlanirephggg
68.41
0 7.51
76.69
0.40
4-.37
95.00-000
9-099-9994
09.30.22 6: 0.58
1906 6: 0.95
01.11
069
65.18
=2 1,63
20.99 1.39
03.23-3 2: 1.30
wellerdd modi esvesgori
100-55
0.52
109.0:1:000 0..0
112.31
9.006
111
6.0.8
115/06 8: 0.55
112:49
0.18
11683
186.30 9375
106 0..344
Gyn- MedeCCo
$7.90
$5.35
27.19
16:29-2 0.00
11:0:1111 1.52
99.33 4: 16 16
50
90.05
13
50.0
8.35
162.28 1.70
155.12 6.89
100.58 01 1:97
63.51---7333
20.55 3 6.55
41.83
3.64
197.64 5:366
327.31
1:44
157.51
6 7.55
163.25 : 44.18
197.55
5.80
200.3 51
0.96
182.52 9- 2.68
99.25
19.34
mng
065002 0 2:29
77.50 1.55
92. 0..50-0..5
9-35
94.25 6 0.55
$4.00 1.00
108,09
000
0/00 0.00
42.53 1-:3-.5 36.61
66:55 4.12
03.0 0-3-3:333 2.18
71.25 1-57
53.79 2.316
16:01
16.25
1400
1100 11:03
01.00 0 2.39
2510
6:55 2 1.20
16.30 5.95
1.5-1:10 130
1035 1:531
1130 4.55
$3.90 2: 1.30
V1.75
2.15
1000 1001
1000 1623
11:5 2 0.50
+043
(.44
0.35
4:01:400
15.25
1:334
53381
09.50
1033
11.00
0.50
AntMaaa ang
25.46
31.59
3116
11.45
61.5
77.00
16.71
0100
7:48
11.01
pen-immapp-
MusE 7:45
16.00
9.56 4:19
19400 10400-1445 1.555
106.5
2 10:26
135.35
3.22
56,4 = 5.02
35.84 NL57
77:3 2:30
jeadleaded
11.4 12:27
63.62
14622111:35 11:75
3-38
1476
2.31
11485
4.75
1864
7.15
53,99
55.0
0 7-85
21.1
pescepprosl
13635::3310
14001
6.33
155/2
116.3
119:5:
dnerlhrres-s el dpendoeaber
9:11 4:15
6:77
1238 3.55
117---377
13.0 1:20
0.77
0.00 0055
2:11 7.76
1:34
00.3 2.00
5.13
1,47
6.90
4L21 00.0
20.36 2
0.55
10535 0.35
167.99
105
12:51 4.37
2.3-220.14
1656 9-366
0.50 5/55
115: 0.1
0.25 1000
5.49 2.11
1.6-00.11
127.99 0 0.57
128.5 0.66
12306-5:022
0.9 = 1.12
125.26 8: 0.37
134.44
0.30
91.34 8 47.77
00.44:247..0 47.09
127.35 2:
0:20
1.22
3/0
0.25 0-002
0:11
031-3-00
9:000
0.39 1-111-111
0.70
0:08
iekent copnnvl
10011 5 0.355
19631 9.66
0.33 0:21
1:.331.:::33
516 5.10
11012 8: 6.2
111114 022
4007 =2 1005
06.12 8.75
196.45 0.30
$7.60
Tood ing
$1.37
35.31
$4.00
7630
35.54
76,35
90,12
54.82
04133
$4.37
Observation 4: Hyperparameters are not always transferable between Deep Learning frame- works.
4.2 Offline-to-Online
We also implement the following algorithms in offlnnetooonline setup: CQL (Kumar al. 2020).
IQL (Kostrikov et al., (02T), AWAC (Nair et al., 2020). SPOT (Wu et al. 2022 Cal-QL. Nakaaooto
et al. 2023), ReBRAC (Tarasov et al.) 2023). Inspired by Nakamoto al. (0033) we evaluate
algorithms on AntMaze and Adroit Cloned dataset
[TI Each algorithm is trained offline over I million
steps and tuned using online transitions over another million steps. The AntMaze tasks are evaluated
using 100 episodes, while the Adroit tasks are tested with ten episodes.
Table 3: Normalized performance of algorithms after offline pretraining and online finetuning on D4RL averaged over 4 random seeds.
2013 27
1 8.00
MAB 500 1e
$1.0 9L3
4339 20B
S8 24 2
5500 540
12 2030 10.0 L13
13 1AS2 - Wib She
2825
222
15.00
31.0
12.5
1/6
26.55
92.2
130
38.22 7235
35.50
3.55
28.5
11.00
386
E70 12.4
1612- 2648 1-144
48.35- 55.0 1077221
26.33 76:01222111
50.4 12.4--1....0-03
7A09 Nanto
55 1510 8..0 11.12
1.3 3.5
18200 26
6ik 4333 2008
-013
N.0 a 1182 45 3:31
1:00
200
0.56
2.2 431
125 132
27.77 2549
3.7
1377 - 1.99
633 133
136a8
23.54-m 21.78 1-300
4:12 (422
21.0
263 1033
2001 (1A4
45.4 (610142200)
M612 (005025111
The scores, normalized after the offline stage and online tuning, are reported in Table 3 We also provide finetuning cumulative regret proposed by Nakamoto al. (2023) in Table 4] Cumulative regret is calculated as (1 -1-aa average success rate [2 It is bounded between 0 and 1, indicating the
range of possible values. Lower values of cumulative regret indicate better algorithm efficiency. The performance profiles and probability of improvement of ReBRAC over other algorithms after online finetuning are presented in Figure
Note, Nakamoto et al. (0233 used modified Cloned datasets while we employ original data from D4RL because these datasets are more common to for benchmarking.
As specified by the authors: https: /ttp//gtttttttbt cow http:///ww..aaaaooooaaaaaaa////a

---

Table 4: Cumulative regret of online finetuning calculated 4 random seeds.
average success rate averaged over
Task Name
AWAC
CQL.
IQL.
SPOT
Cal-QI.
ReBRAC
antmaze-uraze--22
0.00
0.07
0.00
0.02
nntmaze-uaazediivra--222
0.88 0.01
0.09 0.01
0.43 0.11
0.22 0.07
0.05 0.01
0.04 0.02
antmaze-medfun-playee
1.00 0.00
0.08 0.01
0.09 + 0.01
0.06 + 0.00
0.04 + 0.01
0.02 + 0.00
antmaze-meiiun-divreeee
1.00 = 0.00
0.08 0.00
0.10 0.01
0.05 = 0.01
0.04 = 0.01
0.03 = 0.00
antmize-large-play-v2
1.00 T 0.00
0.21
0,02
0.34 0.05
0.29 1 0.07
0.13 1. 0.02
0.14 11411 0.05
antmaze-large-diierss---2
1.00 + 0.00
0.21 = 0.03
0.41 = 0.03
0.23 + 0.08
0.13 + 0.02
0.29 + 0.45
AntMaze avg
0.82
0.11
0.24
0.15
0.07
0.10
pen-cloned-vl
0.00
0.37 1
0.58 1 0.02
0,98 1 0.01
door-cloned-vl
0.00
0.83 + 0.03
0.99
1.00
0.18 0.06
hammer-cloned-vl
1.00 0.00
1.00
0.00
0.65
1 0.10
0.98
1 0.01
1.00
0.00
0.12 1 0.03
rclocate-cloned-vl
1.00 0.00
1.00
0.00
1.00 = 0.00
1.00 + 0.00
1.00
0.00
0.9 0.06
Adroit avg
0.99
0.71
Total avg
0.84
0.47
0.43
0.44
0.44
0.19
AMaC
Algorithm
Algorithm X
1A3RAC
1at scre
2.0
RHBRAC
ANAC
100
ReBRAC
REBRAC
SPOT
0.7
0.8
DARL
PIX
(b)
Figure 3: (a) Performance profiles after online tuming (b) Probability of improvement of ReBRAC to other algorithms after online tuning. The curves (Agarwal et (20T)) are for D4RL benchmark spanning AntMaze and Adroit cloned datasets.
AWAC, initially proposed for finetuning purposes, appeared to be the worst of the considered algorithms, where the score is improved only on most straightforward antmaze-umaze-v2 dataset. At the same time, other datasets, performances either stay the same or even drop.
Observation 5: AWAC does not benefit from online tuning the considered tasks.
Cal-QL proposed modification of CQL, which is expected to work better in tonloie setting. However, in our experiments, after finetuning CQL obtained scores which are not very different from Cal-QL.. At the same time, could not make both algorithms solve Adroit tasks
Observation 6: There is big difference between CQL and Cal-QL. On AntMaze, these algorithms perform the best but work poorly on Adroit.
IQL starts with good offline scores on AntMaze, but is less efficient in finetuning than other algorithms except for AWAC. At the same time, IQL and ReBRAC are the only algorithms that notably improve scores after tuning on Adroit tasks, making them the most competitive offline-to- online baselines considering the average score.
The issues Observations and 4. Additional hyperparameters search is needed.
7

---

Observation Considering offline and offline-to to-olie results, IQL. and ReBRAC appear to be the strongest baselines on average.
5
Conclusion
This paper introduced CORL, a single-filt implementation library for offline and offline-to-onling rein- forcement learning algorithms with configuration files and advanced metrics tracking support. In total, we provided implementations of ten offline and six offline-t--nllng algorithms. All implemented approaches were benchmarked on D4RL datasets, closely matching (sometimes overperforming) the reference results, if available. Focusing on implementation clarity and reproducibility, we hope that CORL will help RL practitioners in their research and applications.
This study's benchmarking results and observations are intended to serve as references for future offline reinforcement learning research and its practical applications. By sharing comprehensive logs, researchers can readily access and utilize our results without having to re-run any our experiments, ensuring that the results are replicable.
References
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. URL https: ///ww. tensorflow......o org/ Software available from tensorflow.org.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G Bellemare. Deep reinforcement learning the edge of the statistical precipice. Advances in Neural Information Processing Systems, 2021.
Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty- offline reinforcement learning with diversified q-ensemble. Advances in neural information processing systems, 34:7436-7447 744766-77772222 2021.
Gor Arakelyan, Gevorg Soghomonyan, and The Aim team. Aim, 6 2020. URL httttttps//tthhhi com/aimhubio/ /aim
Lukas Biewald. Experiment tracking with weights and biases, 2020. URL httpps:/www.wwnddb comm Software available from wandb.com.
Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Marc G Belle- mare. Dopamine: A research framework for deep reinforcement learning. arXiv preprint
arXiv:18/2.06// 10, 2018.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 54:15084-15097, 2021.
Minmin Chen, Can Xu, Vince Gatto, Devanshu Jain, Aviral Kumar, and Ed H. Chi. Off-policy actor-critic for recommender systems. Proceedings of the 16th ACM Conference on Recommender Systems, 2022.
Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines. https
//github. wwt. haam//oppna//baaelness 2017.
Christopher P. Diehl, Timo Sievernich, Martin Krüger, Frank Hoffmann, and Torsten Bertram. Umbrella: Uncertainty-awan model-based offline reinforcement learning leveraging planning.
ArXiv, 1bs/2111.11097, 2021.
8

---

Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep rein- forcement learning for continuous control. Proceedings of the 33rd International Conference on International Conference on Machine Learning- Volume 48, ICML 16, pp- 1329-13333 -139-. JMLR.org. 2016.
I Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, L. Rudolph, and Aleksander Madry. Implementation matters in deep A case study on ppo and trpo. In ICLR, 2020.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20145, 2021.
Yasuhiro Fujita, Prabhat Nagarajan, Toshiki Kataoka, and Takahiro Ishikawa. Chainerrl: A deep
reinforcement learning library. Journal of Machine Learning Research, 22(77):1-14, 2021, URL http://jmlr: http:///jml..org/ppaaa wttpppperp/////////// 376. html
The garage contributors. Garage: A toolkit for reproducible reinforcement learning research. https 2019.
//gi com htt.:///wwooggrrrogggarrrree
Jason Gauci, Edoardo Conti, Yitao Liang, Kittipat Virochsiri, Zhengxing Chen, Yuchen He, Zachary Kaden, Vivek Narayanan, and Xiaohui Ye. Horizon: Facebook's open source applied reinforcement learning platform. arXiv preprint 1..iv:/8/1100200000 2018.
Kamyar Ghasemipour, Shixiang Shane Gu, and Ofir Nachum. Why so pessimistic? estimating uncertainties for offiine rl through ensembles, and why their independence matters. Advances in Neural Information Processing Systems, 186881887118,, 2022.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Proceedings of the Thirty-Secons AAAI Confer- ence on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, AAAI'18AAAT 1AAATAAATT///ATTT AAAI Press, 2018. ISBN 978-1-57735-8000-
Ashley Hill, Antonin Raffin, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, Rene Traore, Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford. John Schulman, Szymon Sidor, and Yuhuai Wu. Stable baselines. https://githon com/
hill aal-aa/tta aastaaaseeeeeeeeeeeeeaeeaaaaaaae a/stabbl-bbaeeleeees 2018.
Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, and Jeff Braga. Cleanrl: High- quality single-file implementations of deep reinforcement learning algorithms. arXiv preprint arXiv:2/ 1..08/9 2021.
Wah Loon Keng and Laura Graesser. Slm lab. https://githu/ com/kengz//SMM-LL
|ttges//ggttuu...oom//n//////L 2017.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: /0ti:::4//.9990 6980, 2014.
Sergey Kolesnikov and Oleksii Hrinchuk. Catalyst.rl: A distributed framework for reproducible rl research, 2019. htRp.sttss::... httpppp:/w/xxwv/o/////// attg//bbs/1903 00027
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:21 10.06169,, 2021.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances Neural Information Processing Systems, 33:1 179-19991 2020.
Aviral Kumar, Anikait Singh, Stephen Tian, Chelsea Finn, and Sergey Levine. A workflow for offline model-free robotic reinforcement learning. In 5th Annual Conference on Robot Learning, 2021. URL http:ppwwww.rreee.eeeeeeoo/rrrrr

---

Vladislav Kurenkov and Sergey Kolesnikov. Showing your offline reinforeement learning work: Online evaluation budget matters. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 11729-11752. PMLR, 17-23 Jul 2022. htt.htttps://prcceediie
mlr www.ppress/vvv. http////////kkkvvveee/ www.nkoreeeoooooeeee 22a. html
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems, arXiv preprint rrXiv:2005.01643 2020.
Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg, Joseph E. Gonzalez, Michael I. Jordan, and Ion Stoica. RLlib: Abstractions for distributed reinforcement learning. In International Conference on Machine Learning (ICML), 2018.
Xiao- Xiao-Yang Liu, Zechu Li, Zhaoran Wang. and Jiahao Zheng. ElegantRL: Massively par- allel framework for cloud-native deep reinforcement learning. 2021.
com/
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 2017.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, loannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533 February 2015. ISSN 1476-4687.
doi: 10. 10.1/t8/n 14236.
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets. arXiv preprint (XXx:2000609359) 2020.
Mitsuhiko Nakamoto, Yuexiang Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning. arXiv preprint TrXXi::3033004799 2023.
Laura Smith, Ilya Kostrikov, and Sergey Levine. Walk in the Park: Learning to Walk in 20 Minutes With Model-Free Reinforcement Learning, August 2022.
Adam Stooke and Pieter Abbeel. ripyt: A research code base for deep reinforcement learning in pytorch, 2019. https:::://ar
Michita Imai Takuma Seno. d3ripy: An offline deep reinforcement library. In NeurlPS 2021 Offline Reinforcement Learning Workshop, December 2021.
Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov. Revisiting the minimalist approach to offline reinforcement learning. arXiv preprint arXiv:2305.0098.. 2023.
Jiyy Weng, Huayu Chen, Dong Yan, Kaichao You, Alexis Dubureq, Minghao Zhang, Yi Su, Hang Su, and Jun Zhu. Tianshou: A highly modularized deep reinforcement learning library. arXiv preprint ar arXhv:2107.14177 2021.
Jayy Weng, Min Lin, Shengyi Huang, Bo Liu, Denys Makoviichuk, Viktor Makoviychuk, Zichen Liu, Yufan Song, Ting Luo, Yukun Jiang, Zhongwen Xu, and Shuicheng Yan. EnvPool: A highly parallel reinforcement learning environment execution engine. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave. K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 22401-22421. Curran Associates, neur ips cc/ hpperr files/ wwwees/ppppp//aeee /ppppp//022/fille/
Inc., 2022. URL https::/ppocceeddiiiiee ScetfoeeSdbaa66sfffooo4eeeeeeee Paper -atrrr-atatttsttea aaataasetssand_BBnnnaaaaaae pdf
Jialong Wu, Haixu Wu, Zihan Qiu, Jianmin Wang, and Mingsheng Long. Supported policy optimiza- tion for offline reinforcement learning. arXiv preprint arXiv:2202.062.39 2022.
10

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Seefsection
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments (e.g. for benchmarks)...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as a URL)? [Yes] We release our codebase, configs, and in-depth reports at pettpa//gitonn
ai//ORL
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendixx D
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Sce|Appendix D
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes)
(b) Did you mention the license of the assets? [Yes] See Appendix C
(c) Did you include any assets either in the supplemental material as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]

---

A Additional Benchmark Information
A.1
Offline
120
202
BOAC
helicheetalvve
toggert
watherid
belleer 2ammvo
Debon
(a)
wilmatedd
anomattere rodd
inimate--g
reticated
refocaterd
(c)
(d)
Figure 4: Graphical representation of the normalized performance of the last trained policy on D4RL averaged over 4 random seeds. (a) Gym-MuJoC datasets. (b) Maze2d datasets (c) AntMaze datasets
(d) Adroit datasets
12

---

102
watterid-
(a)
b
100
Jammene
retorate
Datar:
(d)
Figure 5: Graphical representation of the normalized performance of the best trained policy on D4RL averaged over random seeds. (a) Gym-MuJoCo datasets. (b) Maze2d datasets (c) AntMaze datasets (d) Adroit datasets
(a)
(b)
(c)
Figure 6: Training curves for HalfCheetah task.
(a) Medium dataset, (b) Medium-expert dataset, (c) Medium-replay dataset
(a)
(b)
(c)
Figure 7: Training curves for Hopper task.
(a) Medium dataset, (b) Medium-expen expert dataset, (c) Medium-replay dataset
(a)
(b)
(c)
Figure 8: Training curves for Walker2d task.
a)) Medium dataset, (b) Medium-expert dataset, (c) Medium-replay dataset
13

---

Ale
(a)
(b)
(c)
Figure 9: Training curves for Maze2d task.
a)) Medium dataset, (b) Medium-expert dataset, (c) Medium-replay dataset
(a)
(b)
(c)
Al
Al
(d)
(I)
Figure 10: Training curves for AntMaze task.
(a) Umaze dataset, (b) Medium-play dataset, (c) Large-play dataset, (d) Umaze-diverse dataset, (e) Dedium-diverss dataset, (f) Large- dataset
(a)
(b)
(c)
Figure 11: Training curves for Pen task.
(a) Human dataset, (b) Colned dataset, (c) Expert dataset
(a)
(b)
(c)
Figure 12: Training curves for Door task.
(a) Human dataset, (b) Colned dataset, (c) Expert dataset
14

---

(a)
(b)
(c)
Figure 13: Training curves for Hammer task.
(a) Human dataset, (b) Colned dataset, (c) Expert dataset
(a)
(b)
(c)
Figure 14: Training curves for Relocate task.
(a) Human dataset, (b) Colned dataset, (c) Expert dataset
15

---

A.2
Offline-to-online
Now. hor
02
antincrely
Mosec
hamerter
(a)
Figure 15: Graphical representation of the normalized performance of the last trained policy D4RL after online tuning averaged over 4 random seeds. (a) AntMaze datasets (b) Adroit datasets
(b)
(
(d)
(e)
(f)
Figure 16: Training curves for AntMaze task during online tuning.
(a) Umaze dataset, (b) Medium-pla dataset, (c) Large-play dataset, (d) Umaze- diverse dataset, (e)
Sedium-diverss dataset, (f) Large-divens dataset
16

---

400
HAML
0.2
24
(b)
FIME
ROAL
(C)
(d)
Figure 17: Training curves for Adroit Cloned task during online tuning. (a) Pen, (b) Door, (c) Hammer, (d) Relocate
17

---

B Weights&Biases Biases Tracking
Fiter equanturrss by atatrutee
Goop experrunts by antttuue by aEITTUAcs
Sem expernents
Se
espeitenr
Vaniii Velalieedsttl sduttl operitecb
sadinn indiing
Souch by don
Applying detleren settige to chars le ureetting
Chars grapping
H
2012
clidna fouk coparingg
deni www.anaaneddnnnom Score
svavice
Experienn clans
chars
Expeiinor lgp
Bestioee like.
rg carfiz
reguioemenss
205 600
Figure 18: Screenshots of Weights&Baases experiment tracking interface.
C
License
Our codebase released under Apache License 2.0. The D4RL datasets (Fu al. 2020) are released under Apache License 2.0.
18

---

D Experimental Details
We modify reward on AntMaze task by subtracting from reward it done in previous works except CQL and Cal-QL, where (0, 1) are mapped into (-5, 5).
We used original implementation of TD3 BC SAC-N/EDAC SPOT ReBRAC and custom implementations of IQI and CQL/Cal-QU as the basis for ours.
For most of the algorithms and datasets, we use default hyperparameters if available. Configuration files every algorithm and environment are presented in our GitHub repository. Hyperparameters also provided in subsection D.2
All the experiments ran using V100 and A100 GPUs, which took approximately 5000 hours of compute in total.
D.1 Number of update steps and evaluation rate
Following original work, SAC-N and EDAC trained for 3 million steps (except AntMaze, which trained for million steps) in order to obtain state-of-the-art performance and tested every 10000 steps. Decision Transformer (ITT) training is splitted into datasets pass epochs. We train DT for 50 epochs on each dataset and evaluate every 5 epochs. All other algorithms are trained for million steps and evaluated every 5000 steps (50000 for AntMaze). We evaluate every policy for 10 episodes on Gym-MuJoCo and Adroit tasks and for 100 for Maze2d and AntMaze tasks.
D.2 Hyperparameters
Table 5: BC and BC-N% hyperparameters. used for the best trajectories choice.
Hyperparameter
Value
Optimizer
Adam (Kingma & Ba 2014
BC hyperparameters
Learning Rate
3e-4
Mini-batch size
256
Policy hidden dim
256
Architecture
Policy hidden layers
2
Policy activation function
ReLU
Ratio of best trajectories used
0.1
BC-N% hyperparameters
Discount factor!
1.0
Max trajectory length*
1000
httpp:://ggaaabbbco///tttiin/113 BC
http:://gihhbb...ccmmmmmmma////p
httpe///gtthab.coo//tttm/////
httpp:://gituub.com/youut
19

---

Table 6: TD3+BC hyperparameters.
Hyperparameter
Value
Optimizer
Adam
KKingma & Ba 2014
Critic learning rate
3e-4
Actor learning rate
3e-4
Mini-batch size
256
TD3 hyperparameters
Discount factor
0.99
Target update rate
5e-3
Policy noise
0.2
Policy noise clipping
(-0.5, 0.5)
Policy update frequency
Critic hidden dim
256
Critic hidden layers
Critic activation function
ReLU
Architecture
Actor hidden dim
256
Actor hidden layers
Actor activation function
ReLU
TD3+BC hyperparameters
2.5
Table 7: CQL and Cal-QL hyperparameters. Note: used hyperparameters are suboptimal on Adroit for the implementation we provide.
Hyperparameter
Value
Optimizer
Adam
Kingma & Ba 2014
Critic learning rate
3e-4
Actor learning rate
le-4
SAC hyperparameters
Mini-batch size
256
Discount factor
0.99
Target update rate
5e-3
Target entropy Entropy in target
-1 Action Dim
False
Critic hidden dim Critic hidden layers
256
5, AntMaze
3, otherwise
Architecture
Critic activation function Actor hidden dim Actor hidden layers
ReLU
256
3
Actor activation function
ReLU
Lagrange
True, Maze2d and AntMaze
False, otherwise
Offline a
1.0, Adroit
5.0, AntMaze
10.0, otherwise
CQL hyperparameters
Lagrange gap
5, Maze2d
0.8, AntMaze
Pre-training steps
0
Num sampled actions (during eval)
10
Num sampled actions (logsumexp)
10
Mixing ratio
0.5
Cal-QL. hyperparameters
Online
1.0, Adroit
5.0, AntMaze
20

---

Table IQL hyw.epppaaaameerss....
Hyperparameter
Value
Optimizer
Adam Kingma & Ba 2014
Critic learning rate
3e-4
Actor learning rate
3e-4
Value learning rate
3e-4
Mini-batch size
256
Discount factor
0.99
Target update rate
5e-3
Learning rate decay
Cosine
IQL hyperparameters
Deterministic policy
True, Hopper Medium and Medium-replay
False, otherwise
B
6.0, Hopper Medium-expert
10.0, AntMaze
3.0, otherwise
0.9, AntMaze
0.5, Hopper Medium-expert
0.7, otherwise
Critic hidden dim
256
Critic hidden layers
2
Critic activation function
ReLU
Actor hidden dim
256
Architecture
Actor hidden layers
2
Actor activation function
ReLU
Value hidden dim
256
Value hidden layers
2
Value activation function
ReLU
Table 9: AWAC hyperparameters.
Hyperparameter
Value
Optimizer
Adam (Kingma & Ba 2014
Critic learning rate
3e-4
Actor learning rate
3e-4
Mini-batch size
256
AWAC hyperparameters
Discount factor
0.99
Target update rate
5e-3
X
0.1, Maze2d, AntMaze
0.3333, otherwise
Critic hidden dim
256
Critic hidden layers
2
Critic activation function
ReLU
Architecture
Actor hidden dim
256
Actor hidden layers
2
Actor activation function
ReLU
21

---

Table 10: SAC-N and EDAC hyperparameters.
Hyperparameter
Value
Optimizer
Adam (Kingma & Ba 2014)
Critic learning rate
3e-4
Actor learning rate
3e-4
learning rate
3e-4
SAC hyperparameters
Mini-batch size
256
Discount factor
0.99
Target update rate
5e-3
Target entropy
-1 Action Dim
Critic hidden dim
256
Critic hidden layers
3
Critic activation function
ReLUU
Architecture
Actor hidden dim
256
Actor hidden layers
Actor activation function
ReLU
Number of crities
10. HalfCheetah
20, Walker2d
SAC-N hyperparameters
25, AntMaze
200, Hopper Medium-expert, Medium-replay
500, Hopper Medium
Number of critics
10, HalfCheetah
10. Walker2d, AntMaze
EDAC hyperparameters
50, Hopper
/
5.0, HalfCheetah Medium-expert, Walker2d Medium-
1.0, otherwise
22

---

Table 11: DT hyperparameters.
Hyperparameter
Value
Optimizer
AdamW Loshchilov & Hutter 2017
Batch size
256, AntMaze
4096, otherwise
Return-to-go conditioning
(12000, 6000). HalfCheetah
(3600, 1800), Hopper
(5000, 2500), Walker2d
(160, 80), Maze2d umaze
(280, 140), Maze2d medium and large
(1, 0.5), AntMaze
(3100, 1550). Pen
DT hyperparameters
(2900, 1450), Door
(12800, 6400), Hammer
(4300, 2150), Relocate
Reward scale
1.0, AntMaze
0.001, otherwise
Dropout
0.1
Learning rate
0.0008
Adam betas
(0.9, 0.999)
Clip grad norm
0.25
Weight decay
0,0003
Total gradient steps
100000
Linear warmup steps
10000
Number of layers
3
Number of attention heads
Architecture
Embedding dimension
128
Activation function
GELU
23

---

Table 12: SPOT hyperparameters.
Hyperparameter
Value
Optimizer
Adam
Kingma & Ba 2014)
Learning rate
le-3
VAE hyperparameters
Mini-batch size
256
Number of iterations
10a
KL term weight
0.5
Encoder hidden dim
750
Encoder layers
3
VAE architecture
Latent dim
2 x action dim
Decoder hidden dim
750
Decoder layers Optimizer
3
Adam
Kingma Ba 2014
Critic learning rate
3e-4
Actor learning rate
le-4
Mini-bate size
256
Discount factor
0.99
TD3 hyperparameters
Target update rate
5e-3
Policy noise
0.2
Policy noise clipping
(-0.5, 0.5)
Policy update frequency
2
Critic hidden dim
256
Critic hidden layers
2
Critic activation function
ReLU
Architecture
Actor hidden dim
256
Actor hidden layers
2
Actor activation function
ReLU
A
0.05, 0.1, 0.2, 0.5, 1.0, 2.0, AntMaze 1.0, Adroit
SPOT hyperparameters
24


Reviews:
[{"review_id": "OPLGi6lhMY", "reviewer": "Reviewer_aaf5", "summary": "", "questions": "", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The strengths of the paper include:\n\n1) The authors provide clean and modular implementations of common offline RL algorithms. The code is well-organized and easy to understand.\n\n2) The codebase is self-contained and does not require any additional dependencies on third-party libraries. This makes it easy to install and use. \n\n3) The authors provide single configuration files for all D4RL tasks. This makes it easy for users to track all hyperparameters without having to jump to different places.\n\n4) The authors provide comprehensive benchmark results and analysis. This information is valuable for researchers and practitioners who are interested in offline RL.\n\n5) The codebase is integrated with Weights and Biases. This makes it easy to track experiments and share results.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "JqqgOKjKrg", "reviewer": "Reviewer_JMRf", "summary": "", "questions": "", "limitations": "1. I found different training lengths for different seeds in the published results. I downloaded the released `pickle` file of results for the offline algorithms, and found these algorithms and datasets had different number of score evaluations (training lenghts) for different seeds. Does this mean these seeds were not run toghether? It would also affect both the best and final scores. Here are the training lengths of 4 seeds that were not consistent, with corresponding algorithms and datasets:\n```\nAWAC antmaze-umaze-v2 1000 1000 1001 1001\nAWAC maze2d-large-v1 1000 1000 1000 200\nAWAC maze2d-medium-v1 1000 1000 1000 200\nAWAC maze2d-umaze-v1 1000 1000 1000 200\nAWAC walker2d-medium-expert-v2 1000 1000 1000 200 \nAWAC walker2d-medium-replay-v2 1000 1000 1000 200\nAWAC walker2d-medium-v2 1000 1000 1000 200\nAWAC hopper-medium-expert-v2 1000 1000 1000 200\nAWAC hopper-medium-replay-v2 1000 1000 1000 200\nAWAC hopper-medium-v2 1000 1000 1000 200\nAWAC halfcheetah-medium-expert-v2 1000 1000 1000 200\nAWAC halfcheetah-medium-replay-v2 1000 1000 1000 200\nAWAC halfcheetah-medium-v2 1000 1000 1000 200\nSAC-N relocate-human-v1 40 40 40 601\nSAC-N relocate-expert-v1 601 601 601 42\nEDAC relocate-human-v1 44 601 601 601\n```\n2. The codebase is not updated to the latest version of libraries. For example it still uses [`gym`](https://github.com/openai/gym), which is already deprecated and replaced by [`gymnasium`](https://github.com/Farama-Foundation/Gymnasium);  [`mujoco-py`](https://github.com/openai/mujoco-py/) is now replaced by [`mujoco`](https://github.com/deepmind/mujoco/blob/main/python/README.md) (which is also much easier to install); [`torch`](https://pytorch.org/) has updated to 2.0.1. I was trying to install CORL in a Python 3.11 environment and it failed due to some of the libraries requiring a Python version earlier than 3.11. I'm not asking for the libraries to be strictly the latest versions, but at least deprecated versions should be replaced.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "* Every algorithm is isolated into a single file, making it much easier to both understand the code as well as modify it for novel methods.\n* YAML config files are used to adjust and keep track of hyperparameters.\n* Experiment tracking with W&B is integrated.\n* Most results of the implementations are on par with the original implementations or other offline RL codebases.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "FrtAaLayPq", "reviewer": "Reviewer_ueDk", "summary": "", "questions": "", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "(1) The single-file design exposes algorithmic details for easier understanding, debugging, and modification compared to larger modular codebases.\n\n(2) Lack of abstractions and independency between online/offline algorithm variants reduces implementation overhead and grants flexibility.\n\n(3) Built-in benchmarking on D4RL datasets provides reliable baseline results to aid evaluation and avoid reimplementing algorithms.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "su94lLepyh", "reviewer": "Reviewer_BnL5", "summary": "", "questions": "", "limitations": "N/A", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "Having an easy-to-understand codebase for offline RL is highly important to research development. As pointed out by the authors and earlier works such as [1], implementation details are highly relevant to the performance of RL algorithms. CORL makes it easy for researchers to understand the existence of these details and help produce correct reproductions and good science. The authors also shared valuable research engineering insights derived from their reproducible benchmark (which includes tracked experiments), which is helpful for future research.\n\n[1] https://openreview.net/forum?id=r1etN1rtPB", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "jAnV92CPdE", "reviewer": "Reviewer_KZpC", "summary": "", "questions": "", "limitations": "Limitations are not explicitly described in the main text.", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "- CORL enables accurate, quick, and reproducible implementation and empirical comparison of a range of offline and offline-online RL algorithms for the first time\n\n\n- The paper is well-written and the arguments are easy to follow\n\n\n- Figure 1 is useful to quickly understand the merits of CORL", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: **Decision: Recommend Accept**

This paper presents an new open source library of benchmark offline and offline-to-online RL algorithms. Of note is that such implementations all are single-file, often in 400-600 hundred lines of code.  This design decision makes it easy for researchers to understand algorithms, as well as fork them to develop new ones. The library is benchmarked comprehensively on D4RL.

**Reviewer Commendations:**
* Several reviewers praised the code for being clean, modular, and self-contained with minimal dependencies.
* Simliarly the code was also praised for already being 'ready' for practical experimental work, for instance with Weights & Biases integration, and useful yaml files for D4RL.
* Several reviewers praised the variety of algorithms are implemented and their benchmarking on D4RL

**Reviewer Criticisims:**
* Several reviewers noted that there are probably _some_ areas of code reuse & abstractions could be useful - eg the replay buffer.
* Some also pointed out a similarity to prior work in d3rlpy or CleanRL, though these were not criticisms per se.

**AC View**

I agree with the majority of the reviewers that these clean, single-file implementations and the supporting benchmarking around them are likely to be highly useful to the Offline RL community, and recommend this for acceptance.
Decision: Accept (Poster)