SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. URL https://www.tensorflow.org/. Software available from tensorflow.org.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in Neural Information Processing Systems, 2021.
Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-based offline reinforcement learning with diversified q-ensemble. Advances in neural information processing systems, 34:7436-7447, 2021.
Gor Arakelyan, Gevorg Soghomonyan, and The Aim team. Aim, 6 2020. URL https://github.com/aimhubio/aim.
Lukas Biewald. Experiment tracking with weights and biases, 2020. URL https://www.wandb.com/. Software available from wandb.com.
Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Marc G Bellemare. Dopamine: A research framework for deep reinforcement learning. arXiv preprint arXiv:1812.06110, 2018.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084-15097, 2021.
Minmin Chen, Can Xu, Vince Gatto, Devanshu Jain, Aviral Kumar, and Ed H. Chi. Off-policy actor-critic for recommender systems. Proceedings of the 16th ACM Conference on Recommender Systems, 2022.
Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines. https://github.com/openai/baselines, 2017.
Christopher P. Diehl, Timo Sievernich, Martin Krüger, Frank Hoffmann, and Torsten Bertram. Umbrella: Uncertainty-aware model-based offline reinforcement learning leveraging planning. ArXiv, abs/2111.11097, 2021.
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See section 3
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes] We release our codebase, configs, and in-depth reports at https://github.com/tinkoff-ai/CORL
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix D
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix D
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] See Appendix C
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
A Additional Benchmark Information
A.1 Offline
(d)
Figure 4: Graphical representation of the normalized performance of the last trained policy on D4RL averaged over 4 random seeds. (a) Gym-MuJoCo datasets. (b) Maze2d datasets (c) AntMaze datasets (d) Adroit datasets
Figure 5: Graphical representation of the normalized performance of the best trained policy on D4RL averaged over 4 random seeds. (a) Gym-MuJoCo datasets. (b) Maze2d datasets (c) AntMaze datasets (d) Adroit datasets
(a)
Figure 6: Training curves for HalfCheetah task.
(a) Medium dataset, (b) Medium-expert dataset, (c) Medium-replay dataset
(a)
Figure 7: Training curves for Hopper task.
(a) Medium dataset, (b) Medium-expert dataset, (c) Medium-replay dataset
(a)
Figure 8: Training curves for Walker2d task.
(a) Medium dataset, (b) Medium-expert dataset, (c) Medium-replay dataset
(b)
(a) Medium dataset, (b) Medium-expert dataset, (c) Medium-replay dataset
(c)
Figure 9: Training curves for Maze2d task.
(a)
(d)
Figure 10: Training curves for AntMaze task.
(a) Umaze dataset, (b) Medium-play dataset, (c) Large-play dataset, (d) Umaze-diverse dataset, (e) Medium-diverse dataset, (f) Large-diverse dataset
(a)
Figure 11: Training curves for Pen task.
(a) Human dataset, (b) Colned dataset, (c) Expert dataset
(a)
Figure 12: Training curves for Door task.
(a) Human dataset, (b) Colned dataset, (c) Expert dataset
(b)
Figure 13: Training curves for Hammer task.
(a)
Figure 14: Training curves for Relocate task.
A.2 Offline-to-online
(b)
Figure 15: Graphical representation of the normalized performance of the last trained policy on D4RL after online tuning averaged over 4 random seeds.
(a) AntMaze datasets (b) Adroit datasets
(a)
(d)
Figure 16: Training curves for AntMaze task during online tuning.
(a) Umaze dataset, (b) Medium-play dataset, (c) Large-play dataset, (d) Umaze-diverse dataset, (e) Medium-diverse dataset, (f) Large-diverse dataset
(d)
Figure 17: Training curves for Adroit Cloned task during online tuning.
B Weights&Biases Tracking
Figure 18: Screenshots of Weights&Biases experiment tracking interface.
C License
Our codebase is released under Apache License 2.0. The D4RL datasets (Fu et al., 2020) are released under Apache License 2.0.
We modify reward on AntMaze task by subtracting 1 from reward as it is done in previous works except CQL and Cal-QL, where (0, 1) are mapped into  $(-5, 5)$ .
We used original implementation of  $\mathrm{TD3 + BC^{14}}$ , SAC-N/EDAC<sup>15</sup>, SPOT<sup>16</sup>, ReBRAC<sup>17</sup> and custom implementations of IQL<sup>18</sup> and CQL/Cal-QL<sup>19</sup> as the basis for ours.
For most of the algorithms and datasets, we use default hyperparameters if available. Configuration files for every algorithm and environment are presented in our GitHub repository. Hyperparameters are also provided in subsection D.2.
All the experiments ran using V100 and A100 GPUs, which took approximately 5000 hours of compute in total.
D.1 Number of update steps and evaluation rate
Following original work, SAC-  $N$  and EDAC are trained for 3 million steps (except AntMaze, which is trained for 1 million steps) in order to obtain state-of-the-art performance and tested every 10000 steps. Decision Transformer (DT) training is split into datasets pass epochs. We train DT for 50 epochs on each dataset and evaluate every 5 epochs. All other algorithms are trained for 1 million steps and evaluated every 5000 steps (50000 for AntMaze). We evaluate every policy for 10 episodes on Gym-MuJoCo and Adroit tasks and for 100 for Maze2d and AntMaze tasks.
D.2 Hyperparameters
Table 5: BC and BC-  $N\%$  hyperparameters.  $\dagger$  used for the best trajectories choice.
Hyperparameter	Value
BC hyperparameters	Optimizer	Adam (Kingma & Ba, 2014)
Learning Rate	3e-4
Mini-batch size	256
Architecture	Policy hidden dim	256
Policy hidden layers	2
Policy activation function	ReLU
BC-N% hyperparameters	Ratio of best trajectories used	0.1
Discount factor†	1.0
Max trajectory length†	1000
$^{14}$ https://github.com/sfujim/TD3_BC
15https://github.com/snu-mllab/EDAC
16https://github.com/thuml/SPOT
$^{17}$ https://github.com/tinkoff-ai/ReBRAC
18https://github.com/gwthomas/IQL-PyTorch
19https://github.com/young-geng/CQL
Table 6: TD3+BC hyperparameters.
Hyperparameter	Value
TD3 hyperparameters	Optimizer	Adam (Kingma & Ba, 2014)
Critic learning rate	3e-4
Actor learning rate	3e-4
Mini-batch size	256
Discount factor	0.99
Target update rate	5e-3
Policy noise	0.2
Policy noise clipping	(-0.5, 0.5)
Policy update frequency	2
Architecture	Critic hidden dim	256
Critic hidden layers	2
Critic activation function	ReLU
Actor hidden dim	256
Actor hidden layers	2
Actor activation function	ReLU
TD3+BC hyperparameters	α	2.5
Table 7: CQL and Cal-QL hyperparameters. Note: used hyperparameters are suboptimal on Adroit for the implementation we provide.
Hyperparameter	Value
SAC hyperparameters	Optimizer	Adam (Kingma & Ba, 2014)
Critic learning rate	3e-4
Actor learning rate	1e-4
Mini-batch size	256
Discount factor	0.99
Target update rate	5e-3
Target entropy	-1 · Action Dim
Entropy in Q target	False
Architecture	Critic hidden dim	256
Critic hidden layers	5, AntMaze
3, otherwise
Critic activation function	ReLU
Actor hidden dim	256
Actor hidden layers	3
Actor activation function	ReLU
CQL hyperparameters	Lagrange	True, Maze2d and AntMaze
False, otherwise
Offline α	1.0, Adroit
5.0, AntMaze
10.0, otherwise
Lagrange gap	5, Maze2d
0.8, AntMaze
Pre-training steps	0
Num sampled actions (during eval)	10
Num sampled actions (logsumexp)	10
Cal-QL hyperparameters	Mixing ratio	0.5
Online α	1.0, Adroit
5.0, AntMaze
Table 8: IQL hyperparameters.
Hyperparameter	Value
IQL hyperparameters	Optimizer	Adam (Kingma & Ba, 2014)
Critic learning rate	3e-4
Actor learning rate	3e-4
Value learning rate	3e-4
Mini-batch size	256
Discount factor	0.99
Target update rate	5e-3
Learning rate decay	Cosine
Deterministic policy	True, Hopper Medium and Medium-replay False, otherwise
β	6.0, Hopper Medium-expert 10.0, AntMaze 3.0, otherwise
τ	0.9, AntMaze 0.5, Hopper Medium-expert 0.7, otherwise
Architecture	Critic hidden dim	256
Critic hidden layers	2
Critic activation function	ReLU
Actor hidden dim	256
Actor hidden layers	2
Actor activation function	ReLU
Value hidden dim	256
Value hidden layers	2
Value activation function	ReLU
Table 9: AWAC hyperparameters.
Hyperparameter	Value
AWAC hyperparameters	Optimizer	Adam (Kingma & Ba, 2014)
Critic learning rate	3e-4
Actor learning rate	3e-4
Mini-batch size	256
Discount factor	0.99
Target update rate	5e-3
λ	0.1, Maze2d, AntMaze
0.3333, otherwise
Architecture	Critic hidden dim	256
Critic hidden layers	2
Critic activation function	ReLU
Actor hidden dim	256
Actor hidden layers	2
Actor activation function	ReLU
Table 10: SAC-  $N$  and EDAC hyperparameters.
Hyperparameter	Value
SAC hyperparameters	Optimizer	Adam (Kingma & Ba, 2014)
Critic learning rate	3e-4
Actor learning rate	3e-4
α learning rate	3e-4
Mini-batch size	256
Discount factor	0.99
Target update rate	5e-3
Target entropy	-1 · Action Dim
Architecture	Critic hidden dim	256
Critic hidden layers	3
Critic activation function	ReLU
Actor hidden dim	256
Actor hidden layers	3
Actor activation function	ReLU
SAC-N hyperparameters	Number of critics	10, HalfCheetah
20, Walker2d
25, AntMaze
200, Hopper Medium-expert, Medium-replay
500, Hopper Medium
EDAC hyperparameters	Number of critics	10, HalfCheetah
10, Walker2d, AntMaze
50, Hopper
μ	5.0, HalfCheetah Medium-expert, Walker2d Medium-expert
1.0, otherwise
Table 11: DT hyperparameters.
Hyperparameter	Value
DT hyperparameters	Optimizer	AdamW (Loshchilov & Hutter, 2017)
Batch size	256, AntMaze
4096, otherwise
Return-to-go conditioning	(12000, 6000), HalfCheetah
(3600, 1800), Hopper
(5000, 2500), Walker2d
(160, 80), Maze2d umaze
(280, 140), Maze2d medium and large
(1, 0.5), AntMaze
(3100, 1550), Pen
(2900, 1450), Door
(12800, 6400), Hammer
(4300, 2150), Relocate
Reward scale	1.0, AntMaze
0.001, otherwise
Dropout	0.1
Learning rate	0.0008
Adam betas	(0.9, 0.999)
Clip grad norm	0.25
Weight decay	0.0003
Total gradient steps	100000
Linear warmup steps	10000
Architecture	Number of layers	3
Number of attention heads	1
Embedding dimension	128
Activation function	GELU
Table 12: SPOT hyperparameters.
Hyperparameter	Value
VAE hyperparameters	Optimizer	Adam (Kingma & Ba, 2014)
Learning rate	1e-3
Mini-batch size	256
Number of iterations	10^5
KL term weight	0.5
Encoder hidden dim	750
Encoder layers	3
VAE architecture	Latent dim	2 × action dim
Decoder hidden dim	750
Decoder layers	3
Optimizer	Adam (Kingma & Ba, 2014)
Critic learning rate	3e-4
Actor learning rate	1e-4
Mini-batch size	256
TD3 hyperparameters	Discount factor	0.99
Target update rate	5e-3
Policy noise	0.2
Policy noise clipping	(-0.5, 0.5)
Policy update frequency	2
Architecture	Critic hidden dim	256
Critic hidden layers	2
Critic activation function	ReLU
Actor hidden dim	256
Actor hidden layers	2
Actor activation function	ReLU
SPOT hyperparameters	λ	0.05, 0.1, 0.2, 0.5, 1.0, 2.0, AntMaze 1.0, Adroit