SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Tinkoff
den.tarasov@tinkoff.ai
Alexander Nikulin
a.p.nikulin@tinkoff.ai
Dmitry Akimov
d.akimov@tinkoff.ai
Vladislav Kurenkov
v.kurenkov@tinkoff.ai
Sergey Kolesnikov
s.s.kolesnikov@tinkoff.ai
CORL<sup>1</sup> is an open-source library that provides thoroughly benchmarked single-file implementations of both deep offline and offline-to-online reinforcement learning algorithms. It emphasizes a simple developing experience with a straightforward codebase and a modern analysis tracking tool. In CORL, we isolate methods implementation into separate single files, making performance-relevant details easier to recognize. Additionally, an experiment tracking feature is available to help log metrics, hyperparameters, dependencies, and more to the cloud. Finally, we have ensured the reliability of the implementations by benchmarking commonly employed D4RL datasets providing a transparent source of results that can be reused for robust evaluation tools such as performance profiles, probability of improvement, or expected online performance.
Deep Offline Reinforcement Learning (Levine et al., 2020) has been showing significant advancements in numerous domains such as robotics (Smith et al., 2022; Kumar et al., 2021), autonomous driving (Diehl et al., 2021) and recommender systems (Chen et al., 2022). Due to such rapid development, many open-source offline RL solutions emerged to help RL practitioners understand and improve well-known offline RL techniques in different fields. On the one hand, they introduce offline RL algorithms standard interfaces and user-friendly APIs, simplifying offline RL methods incorporation into existing projects. On the other hand, introduced abstractions may hinder the learning curve for newcomers and the ease of adoption for researchers interested in developing new algorithms. One needs to understand the modularity design (several files on average), which (1) can be comprised of thousands of lines of code or (2) can hardly fit for a novel method<sup>3</sup>.
In this technical report, we take a different perspective on an offline RL library and also incorporate emerging interest in the offline-to-online setup. We propose CORL (Clean Offline Reinforcement Learning) – minimalistic and isolated single-file implementations of deep offline and offline-to-online RL algorithms, supported by open-sourced D4RL (Fu et al., 2020) benchmark results. The uncomplicated design allows practitioners to read and understand the implementations of the algorithms straightforwardly. Moreover, CORL supports optional integration with experiments tracking tools such as Weighs&Biases (Biewald, 2020), providing practitioners with a convenient way to analyze
$^{1}$ CORL Repository: https://github.com/corl-team/CORL
$^{2}$ https://github.com/hanjuku-kaso/awesome-offline-rl#oss
<sup>3</sup>https://github.com/takuseno/d3rlpy/issues/141
the results and behavior of all algorithms, not merely relying on a final performance commonly reported in papers.
We hope that the CORL library will help offline RL newcomers study implemented algorithms and aid the researchers in quickly modifying existing methods without fighting through different levels of abstraction. Finally, the obtained results may serve as a reference point for D4RL benchmarks avoiding the need to re-implement and tune existing algorithms' hyperparameters.
python dt.py --config=cfg/dt-hopper.yaml --logdir=logs/dt-hopper --num-epochs=50
Figure 1: The illustration of the CORL library design. Single-file implementation takes ayaml configuration file with both environment and algorithm parameters to run the experiment, which logs all required statistics to Weights&Biases (Biewald, 2020).
Since the Atari breakthrough (Mnih et al., 2015), numerous open-source RL frameworks and libraries have been developed over the last years: (Dhariwal et al., 2017; Hill et al., 2018; Castro et al., 2018; Gauci et al., 2018; Keng & Graesser, 2017; garage contributors, 2019; Duan et al., 2016; Kolesnikov & Hrinchuk, 2019; Fujita et al., 2021; Liang et al., 2018; Fujita et al., 2021; Liu et al., 2021; Huang et al., 2021; Weng et al., 2021; Stooke & Abbeel, 2019), focusing on different perspectives of the RL. For example, stable-baselines (Hill et al., 2018) provides many deep RL implementations that carefully reproduce results to back up RL practitioners with reliable baselines during methods comparison. On the other hand, Ray (Liang et al., 2018) focuses on implementations scalability and production-friendly usage. Finally, more nuanced solutions exist, such as Dopamine (Castro et al., 2018), which emphasizes different DQN variants, or ReAgent (Gauci et al., 2018), which applies RL to the RecSys domain.
At the same time, the offline RL branch and especially offline-to-online, which we are interested in this paper, are not yet covered as much: the only library that precisely focuses on offline RL setting is d3rlpy (Takuma Seno, 2021). While CORL also focuses on offline RL methods (Nair et al., 2020; Kumar et al., 2020; Kostrikov et al., 2021; Fujimoto & Gu, 2021; An et al., 2021; Chen et al., 2021), similar to d3rlpy, it takes a different perspective on library design and provides non-modular independent algorithms implementations. More precisely, CORL does not introduce additional abstractions to make offline RL more general but instead gives an "easy-to-hack" starter kit for research needs. Finally, CORL also provides recent offline-to-online solutions (Nair et al., 2020; Kumar et al., 2020; Kostrikov et al., 2021; Wu et al., 2022; Nakamoto et al., 2023; Tarasov et al., 2023) that are gaining interest among researchers and practitioners.
Although CORL does not represent the first non-modular RL library, which is more likely the CleanRL (Huang et al., 2021) case, it has two significant differences from its predecessor. First, CORL is focused on offline and offline-to-online RL, while CleanRL implements online RL algorithms. Second, CORL intends to minimize the complexity of the requirements and external dependencies. To be more concrete, CORL does not have additional requirements with abstractions such as stable-baselines (Hill et al., 2018) or envpool (Weng et al., 2022) but instead implements everything from scratch in the codebase.
3 CORL Design
Single-File Implementations
Implementational subtleties significantly impact agent performance in deep RL (Henderson et al., 2018; Engstrom et al., 2020; Fujimoto & Gu, 2021). Unfortunately, user-friendly abstractions and general interfaces, the core idea behind modular libraries, encapsulate and often hide these important nuances from the practitioners. For such a reason, CORL unwraps these details by adopting single-file implementations. To be more concrete, we put environment details, algorithms hyperparameters, and evaluation parameters into a single file<sup>4</sup>. For example, we provide
- any_percent(bc.py (404 LOC $^5$ ) as a baseline algorithm for offline RL methods comparison,
-  $td3\_bc.py$  (511 LOC) as a competitive minimalistic offline RL algorithm (Fujimoto & Gu, 2021),
- dt.py (540 LOC) as an example of the recently proposed trajectory optimization approach (Chen et al., 2021)
Figure 1 depicts an overall library design. To avoid over-complicated offline implementations, we treat offline and offline-to-online versions of the same algorithms separately. While such design produces code duplications among realization, it has several essential benefits from the both educational and research perspective:
- Smooth learning curve. Having the entire code in one place makes understanding all its aspects more straightforward. In other words, one may find it easier to dive into 540 LOC of single-file Decision Transformer (Chen et al., 2021) implementation rather than  $10+$  files of the original implementation<sup>6</sup>.
- Simple prototyping. As we are not interested in the code's general applicability, we could make it implementation-specific. Such a design also removes the need for inheritance from general primitives or their refactoring, reducing abstraction overhead to zero. At the same time, this idea gives us complete freedom during code modification.
- Faster debugging. Without additional abstractions, implementation simplifies to a single for-loop with a global Python name scope. Furthermore, such flat architecture makes accessing and inspecting any created variable easier during training, which is crucial in the presence of modifications and debugging.
Although it is a typical pattern to use a command line interface (CLI) for single-file experiments in the research community, CORL slightly improves it with predefined configuration files. Utilizing YAML parsing through CLI, for each experiment, we gather all environment and algorithm hyperparameters into such files so that one can use them as an initial setup. We found that such setup (1) simplifies experiments, eliminating the need to keep all algorithm-environment-specific parameters in mind, and (2) keeps it convenient with the familiar CLI approach.
4We follow the PEP8 style guide with a maximum line length of 89, which increases LOC a bit.
5Lines Of Code
$^{6}$ Original Decision Transformer implementation: https://github.com/kzl/decision-transformer
Experiment Tracking
Offline RL evaluation is another challenging aspect of the current offline RL state (Kurenikov & Kolesnikov, 2022). To face this uncertainty, CORL supports integration with Weights&Biases (Biewald, 2020), a modern experiment tracking tool. With each experiment, CORL automatically saves (1) source code, (2) dependencies (requirements.txt), (3) hardware setup, (4) OS environment variables, (5) hyperparameters, (6) training, and system metrics, (7) logs (stdout, stderr). See Appendix B for an example.
Although, Weights&Biases is a proprietary solution, other alternatives, such as Tensorboard (Abadi et al., 2015) or Aim (Arakelyan et al., 2020), could be used within a few lines of code change. It is also important to note that with Weights&Biases tracking, one could easily use CORL with sweeps or public reports.
We found full metrics tracking during the training process necessary for two reasons. First, it removes the possible bias of the final or best performance commonly reported in papers. For example, one could evaluate offline RL performance as max archived score, while another uses the average scores over  $N$  (last) evaluations (Takuma Seno, 2021). Second, it provides an opportunity for advanced performance analysis such as EOP (Kurenkov & Kolesnikov, 2022) or RLiable (Agarwal et al., 2021). In short, when provided with all metrics logs, one can utilize all performance statistics, not merely relying on commonly used alternatives.
4 Benchmarking D4RL
4.1 Offline
In our library, we implemented the following offline algorithms:  $N\%$  Behavioral Cloning (BC), TD3 + BC (Fujimoto & Gu, 2021), CQL (Kumar et al., 2020), IQL (Kostrikov et al., 2021), AWAC (Nair et al., 2020), ReBRAC (Tarasov et al., 2023), SAC-N, EDAC (An et al., 2021), and Decision Transformer (DT) (Chen et al., 2021). We evaluated every algorithm on the D4RL benchmark (Fu et al., 2020), focusing on Gym-MuJoCo, Maze2D, AntMaze, and Adroit tasks. Each algorithm was run for one million gradient steps and evaluated using ten episodes for Gym-MuJoCo and Adroit tasks. For Maze2d, we use 100 evaluation episodes. In our experiments, we tried to rely on the hyperparameters proposed in the original works (see Appendix D for details) as much as possible.
The final performance is reported in Table 1 and the maximal performance in Table 2. The scores are normalized to the range between 0 and 100 (Fu et al., 2020). Following the recent work by Takuma Seno (2021), we report the last and best-obtained scores to illustrate each algorithm's potential performance and overfitting properties. Figure 2 shows the performance profiles and probability of improvement of ReBRAC over other algorithms (Agarwal et al., 2021). See Appendix A for complete training performance graphs.
Based on these results, we make several valuable observations. First, ReBRAC, IQL and AWAC are the most competitive baselines in offline setup on average. Note that AWAC is often omitted in recent works.
Observation 1: ReBRAC, IQL and AWAC are the strongest offline baselines on average.
Second, EDAC outperforms all other algorithms on Gym-MuJoCo by a significant margin, and to our prior knowledge, there are still no algorithms that perform much better on these tasks. SAC-N shows the best performance on Maze2d tasks. However, simultaneously, SAC-N and EDAC cannot solve AntMaze tasks and perform poorly in the Adroit domain.
Observation 2: SAC-N and EDAC are the strongest baselines for Gym-MuJoCo and Maze2d, but they perform poorly on both AntMaze and Adroit domains.
Third, during our experiments, we observed that the hyperparameters proposed for CQL in Kumar et al. (2020) do not perform as well as claimed on most tasks. CQL is extremely sensitive to the
$^7 N$  is a percentage of best trajectories with the highest return used for training. We omit the percentage when it is equal to 100.
${}^{8}$  Except SAC-N,EDAC,and DT due to their original hyperparameters. See Appendix D for details.
(b)
Figure 2: (a) Performance profiles after offline training (b) Probability of improvement of ReBRAC to other algorithms after offline training. The curves (Agarwal et al., 2021) are for D4RL benchmark spanning Gym-MuJoCo, Maze2d, AntMaze, and Adroit datasets.
choice of hyperparameters, and we had to tune them a lot to make it work on each domain (see Table 7). For example, AntMaze requires five hidden layers for the critic networks, while other tasks' performance suffers with this number of layers. The issue of sensitivity<sup>9</sup> was already mentioned in prior works as well (An et al., 2021; Ghasemipour et al., 2022).
Observation 3: CQL is extremely sensitive to the choice of hyperparameters and implementation details.
Fourth, we also observe that the hyperparameters do not always work the same way when transferring between Deep Learning frameworks  $^{10}$ . Our implementations of IQL and CQL use PyTorch, but the parameters from reference JAX implementations sometimes strongly underperform (e.g., IQL on Hopper tasks and CQL on Adroit).
See also https://github.com/aviralkumar2907/CQL/issues/9, https://github.com/tinkoff-ai/CORL/issues/14 and https://github.com/young-geng/CQL/issues/5
$^{10}$ https://github.com/tinkoff-ai/CORL/issues/33
Observation 4: Hyperparameters are not always transferable between Deep Learning frameworks.
4.2 Offline-to-Online
We also implement the following algorithms in offline-to-online setup: CQL (Kumar et al., 2020), IQL (Kostrikov et al., 2021), AWAC (Nair et al., 2020), SPOT (Wu et al., 2022) Cal-QL (Nakamoto et al., 2023), ReBRAC (Tarasov et al., 2023). Inspired by Nakamoto et al. (2023), we evaluate algorithms on AntMaze and Adroit Cloned datasets<sup>11</sup>. Each algorithm is trained offline over 1 million steps and tuned using online transitions over another 1 million steps. The AntMaze tasks are evaluated using 100 episodes, while the Adroit tasks are tested with ten episodes.
Table 3: Normalized performance of algorithms after offline pretraining and online finetuning on D4RL averaged over 4 random seeds.
Task Name	AWAC	CQL	IQL	SPOT	Cal-QL	ReBRAC
antmaze-umaze-v2	52.75 ± 8.67 → 98.75 ± 1.09	94.00 ± 1.58 → 99.50 ± 0.87	77.00 ± 0.71 → 96.50 ± 1.12	91.00 ± 2.55 → 99.50 ± 0.50	76.75 ± 7.53 → 99.75 ± 0.43	98.00 ± 1.82 → 74.75 ± 49.17
antmaze-umaze-diverse-v2	56.00 ± 2.74 → 0.00 ± 0.00	9.50 ± 9.91 → 99.00 ± 1.22	59.50 ± 9.55 → 63.75 ± 25.02	36.25 ± 2.17 → 95.00 ± 3.67	32.00 ± 27.79 → 98.50 ± 1.12	73.75 ± 15.32 → 98.0 ± 3.36
antmaze-medium-play-v2	0.00 ± 0.00 → 0.00 ± 0.00	59.00 ± 11.18 → 97.75 ± 1.30	71.75 ± 2.95 → 89.75 ± 1.09	67.25 ± 10.47 → 97.25 ± 1.30	71.75 ± 3.27 → 98.75 ± 1.64	87.5 ± 4.35 → 98.0 ± 1.82
antmaze-medium-diverse-v2	0.00 ± 0.00 → 0.00 ± 0.00	63.50 ± 6.84 → 97.25 ± 1.92	64.25 ± 1.92 → 92.25 ± 2.86	73.75 ± 7.29 → 94.50 ± 1.66	62.00 ± 4.30 → 98.25 ± 1.48	85.25 ± 2.5 → 98.75 ± 0.5
antmaze-large-play-v2	0.00 ± 0.00 → 0.00 ± 0.00	28.75 ± 7.76 → 88.25 ± 1.28	38.50 ± 8.73 → 64.50 ± 17.04	31.50 ± 12.58 → 87.00 ± 3.24	31.75 ± 8.87 → 97.25 ± 1.79	68.5 ± 7.1 → 31.5 ± 38.75
antmaze-large-diverse-v2	0.00 ± 0.00 → 0.00 ± 0.00	35.50 ± 3.64 → 91.75 ± 3.96	26.75 ± 3.77 → 64.25 ± 4.15	17.50 ± 7.26 → 81.00 ± 14.14	44.00 ± 8.69 → 91.50 ± 3.91	67.0 ± 12.24 → 72.25 ± 48.18
AntMaze avg	18.12 → 16.46 (-1.66)	48.38 → 95.58 (+47.20)	56.29 → 78.50 (+22.21)	52.88 → 92.38 (+39.50)	53.04 → 97.33 (+24.29)	79.99 → 78.87(-1.11)
pen-cloned-v1	88.66 ± 15.10 → 86.82 ± 11.12	-2.76 ± 0.08 → -1.28 ± 2.16	84.19 ± 3.96 → 102.02 ± 20.75	6.19 ± 5.21 → 42.63 ± 20.09	-2.66 ± 0.04 → -2.68 ± 0.12	74.04 ± 13.82 → 138.15 ± 3.71
door-cloned-v1	0.93 ± 1.66 → 0.01 ± 0.00	-0.33 ± 0.01 → -0.33 ± 0.01	1.19 ± 0.93 → 20.34 ± 9.32	-0.21 ± 0.14 → 0.02 ± 0.31	-0.33 ± 0.01 → -0.33 ± 0.01	0.06 ± 0.04 → 102.38 ± 9.54
hammer-cloned-v1	1.80 ± 3.01 → 0.24 ± 0.04	0.56 ± 0.55 → 2.85 ± 4.81	1.35 ± 0.32 → 57.27 ± 28.49	3.97 ± 6.39 → 3.73 ± 4.99	0.25 ± 0.04 → 0.17 ± 0.17	6.53 ± 3.86 → 124.65 ± 8.51
relocate-cloned-v1	-0.04 ± 0.04 → -0.04 ± 0.01	-0.33 ± 0.01 → -0.33 ± 0.01	0.04 ± 0.04 → 0.32 ± 0.38	-0.24 ± 0.01 → -0.15 ± 0.05	-0.31 ± 0.05 → -0.31 ± 0.04	0.69 ± 0.71 → 6.96 ± 5.3
Adroit Avg	22.84 → 21.76 (-1.08)	-0.72 → 0.22 (+0.94)	21.69 → 44.99 (+23.3)	2.43 → 11.81 (+9.38)	-0.76 → -0.79 (-0.03)	20.33 → 93.03 (+72.7)
Total avg	20.01 → 18.58 (-1.43)	28.74 → 57.44 (+28.7)	42.45 → 65.10 (+22.65)	32.70 → 60.15 (+27.45)	31.52 → 58.08 (+26.56)	56.12 → 84.53 (+28.41)
The scores, normalized after the offline stage and online tuning, are reported in Table 3. We also provide finetuning cumulative regret proposed by Nakamoto et al. (2023) in Table 4. Cumulative regret is calculated as  $(1 - \text{average success rate})^{12}$ . It is bounded between 0 and 1, indicating the range of possible values. Lower values of cumulative regret indicate better algorithm efficiency. The performance profiles and probability of improvement of ReBRAC over other algorithms after online finetuning are presented in Figure 3.
<sup>11</sup>Note, Nakamoto et al. (2023) used modified Cloned datasets while we employ original data from D4RL because these datasets are more common to for benchmarking.
<sup>12</sup>As specified by the authors: https://github.com/nakamotoo/Cal-QL/issues/1
Table 4: Cumulative regret of online finetuning calculated as  $1 - \text{average success rate}$  averaged over 4 random seeds.
Task Name	AWAC	CQL	IQL	SPOT	Cal-QL	ReBRAC
antmaze-umaze-v2	0.04 ± 0.01	0.02 ± 0.00	0.07 ± 0.00	0.02 ± 0.00	0.01 ± 0.00	0.10 ± 0.20
antmaze-umaze-diverse-v2	0.88 ± 0.01	0.09 ± 0.01	0.43 ± 0.11	0.22 ± 0.07	0.05 ± 0.01	0.04 ± 0.02
antmaze-medium-play-v2	1.00 ± 0.00	0.08 ± 0.01	0.09 ± 0.01	0.06 ± 0.00	0.04 ± 0.01	0.02 ± 0.00
antmaze-medium-diverse-v2	1.00 ± 0.00	0.08 ± 0.00	0.10 ± 0.01	0.05 ± 0.01	0.04 ± 0.01	0.03 ± 0.00
antmaze-large-play-v2	1.00 ± 0.00	0.21 ± 0.02	0.34 ± 0.05	0.29 ± 0.07	0.13 ± 0.02	0.14 ± 0.05
antmaze-large-diverse-v2	1.00 ± 0.00	0.21 ± 0.03	0.41 ± 0.03	0.23 ± 0.08	0.13 ± 0.02	0.29 ± 0.45
AntMaze avg	0.82	0.11	0.24	0.15	0.07	0.10
pen-cloned-v1	0.46 ± 0.02	0.97 ± 0.00	0.37 ± 0.01	0.58 ± 0.02	0.98 ± 0.01	0.08 ± 0.01
door-cloned-v1	1.00 ± 0.00	1.00 ± 0.00	0.83 ± 0.03	0.99 ± 0.01	1.00 ± 0.00	0.18 ± 0.06
hammer-cloned-v1	1.00 ± 0.00	1.00 ± 0.00	0.65 ± 0.10	0.98 ± 0.01	1.00 ± 0.00	0.12 ± 0.03
relocate-cloned-v1	1.00 ± 0.00	1.00 ± 0.00	1.00 ± 0.00	1.00 ± 0.00	1.00 ± 0.00	0.9 ± 0.06
Adroit avg	0.86	0.99	0.71	0.89	0.99	0.32
Total avg	0.84	0.47	0.43	0.44	0.44	0.19
(b)
Figure 3: (a) Performance profiles after online tuning (b) Probability of improvement of ReBRAC to other algorithms after online tuning. The curves (Agarwal et al., 2021) are for D4RL benchmark spanning AntMaze and Adroit cloned datasets.
AWAC, initially proposed for finetuning purposes, appeared to be the worst of the considered algorithms, where the score is improved only on the most straightforward antmaze-umaze-v2 dataset. At the same time, on other datasets, performances either stay the same or even drop.
Observation 5: AWAC does not benefit from online tuning on the considered tasks.
Cal-QL was proposed as a modification of CQL, which is expected to work better in offline-to-online setting. However, in our experiments, after finetuning CQL obtained scores which are not very different from Cal-QL. At the same time, we could not make both algorithms solve Adroit tasks<sup>13</sup>.
Observation 6: There is no big difference between CQL and Cal-QL. On AntMaze, these algorithms perform the best but work poorly on Adroit.
IQL starts with good offline scores on AntMaze, but it is less efficient in finetuning than other algorithms except for AWAC. At the same time, IQL and ReBRAC are the only algorithms that notably improve its scores after tuning on Adroit tasks, making them the most competitive offline-to-online baselines considering the average score.
<sup>13</sup>The issues are Observations 3 and 4. Additional hyperparameters search is needed.
Observation 7: Considering offline and offline-to-online results, IQL and ReBRAC appear to be the strongest baselines on average.
This paper introduced CORL, a single-file implementation library for offline and offline-to-online reinforcement learning algorithms with configuration files and advanced metrics tracking support. In total, we provided implementations of ten offline and six offline-to-online algorithms. All implemented approaches were benchmarked on D4RL datasets, closely matching (sometimes overperforming) the reference results, if available. Focusing on implementation clarity and reproducibility, we hope that CORL will help RL practitioners in their research and applications.
This study's benchmarking results and observations are intended to serve as references for future offline reinforcement learning research and its practical applications. By sharing comprehensive logs, researchers can readily access and utilize our results without having to re-run any of our experiments, ensuring that the results are replicable.