SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 6: Normalized performance after offline pretraining and online finetuning on D4RL. Baselines scores except TD3+BC are taken from Tarasov et al. (2022). ReBRAC and TD3+BC scores are averaged over ten random seeds, and all others are averaged over four as in Tarasov et al. (2022).
Task Name	TD3 + BC	IQL	SPOT	Cal-QL	ReBRAC, our
antmaze-u-v2	66.8 → 91.4	77.00 → 96.50	91.00 → 99.50	76.75 → 99.75	97.8 → 99.8
antmaze-u-d-v2	59.1 → 48.4	59.50 → 63.75	36.25 → 95.00	32.00 → 98.50	85.7 → 98.1
antmaze-m-p-v2	59.2 → 94.8	71.75 → 89.75	67.25 → 97.25	71.75 → 98.75	78.4 → 97.7
antmaze-m-d-v2	62.6 → 94.1	64.25 → 92.25	73.75 → 94.50	62.00 → 98.25	78.6 → 98.5
antmaze-l-p-v2	21.5 → 0.1	38.50 → 64.50	31.50 → 87.00	31.75 → 97.25	47.0 → 39.5
antmaze-l-d-v2	9.5 → 0.4	26.75 → 64.25	17.50 → 81.00	44.00 → 91.50	66.7 → 77.6
AntMaze avg	46.4 → 54.8 (+8.4)	56.29 → 78.50 (+22.21)	52.88 → 92.38 (+39.50)	53.04 → 97.33 (+24.29)	75.7 → 85.2 (+9.5)
pen-c-v1	86.1 → 110.3	84.19 → 102.02	6.19 → 43.63	-2.66 → -2.68	91.8 → 152.0
door-c-v1	0.0 → 3.4	1.19 → 20.34	-0.21 → 0.02	-0.33 → -0.33	0.4 → 104.9
hammer-c-v1	2.4 → 11.6	1.35 → 57.27	3.97 → 3.73	0.25 → 0.17	4.1 → 131.2
relocate-c-v1	-0.1 → 0.1	0.04 → 0.32	-0.24 → -0.15	-0.31 → -0.31	0.0 → 12.3
Adroit Avg	22.1 → 31.3 (+9.2)	21.69 → 44.99 (+23.3)	2.43 → 11.81 (+9.38)	-0.76 → -0.79 (-0.03)	24.0 → 100.1 (+76.1)
Total avg	36.7 → 45.4 (+8.7)	42.45 → 65.10 (+22.65)	32.70 → 60.15 (+27.45)	31.52 → 58.08 (+26.56)	55.0 → 91.1 (+36.1)
Table 7: Cumulative regret of online finetuning calculated as  $1 - \text{average success rate}$ . Baselines scores except TD3+BC are taken from Tarasov et al. (2022). ReBRAC and TD3+BC regrets are averaged over ten random seeds, and all others are averaged over four as in Tarasov et al. (2022).
Task Name	TD3 + BC	CQL	IQL	SPOT	Cal-QL	ReBRAC, our
antmaze-umaze-v2	0.09 ± 0.08	0.02 ± 0.00	0.07 ± 0.00	0.02 ± 0.00	0.01 ± 0.00	0.00 ± 0.00
antmaze-umaze-diverse-v2	0.47 ± 0.16	0.09 ± 0.01	0.43 ± 0.11	0.22 ± 0.07	0.05 ± 0.01	0.06 ± 0.13
antmaze-medium-play-v2	0.12 ± 0.05	0.08 ± 0.01	0.09 ± 0.01	0.06 ± 0.00	0.04 ± 0.01	0.03 ± 0.01
antmaze-medium-diverse-v2	0.09 ± 0.02	0.08 ± 0.00	0.10 ± 0.01	0.05 ± 0.01	0.04 ± 0.01	0.02 ± 0.00
antmaze-large-play-v2	0.99 ± 0.00	0.21 ± 0.02	0.34 ± 0.05	0.29 ± 0.07	0.13 ± 0.02	0.36 ± 0.30
antmaze-large-diverse-v2	0.99 ± 0.01	0.21 ± 0.03	0.41 ± 0.03	0.23 ± 0.08	0.13 ± 0.02	0.10 ± 0.07
AntMaze avg	0.45	0.11	0.24	0.15	0.07	0.09
pen-cloned-v1	0.30 ± 0.10	0.97 ± 0.00	0.37 ± 0.01	0.58 ± 0.02	0.98 ± 0.01	0.08 ± 0.00
door-cloned-v1	0.97 ± 0.03	1.00 ± 0.00	0.83 ± 0.03	0.99 ± 0.01	1.00 ± 0.00	0.26 ± 0.10
hammer-cloned-v1	0.92 ± 0.14	1.00 ± 0.00	0.65 ± 0.10	0.98 ± 0.01	1.00 ± 0.00	0.13 ± 0.02
relocate-cloned-v1	0.99 ± 0.01	1.00 ± 0.00	1.00 ± 0.00	1.00 ± 0.00	1.00 ± 0.00	0.85 ± 0.07
Adroit avg	0.79	0.99	0.71	0.89	0.99	0.33
Total avg	0.59	0.47	0.43	0.44	0.44	0.18
ReBRAC exhibits competitive performance, surpassing four out of six AntMaze datasets and achieving state-of-the-art results in terms of final scores on Adroit tasks. On average, ReBRAC outperforms its closest competitor, Cal-CQL, which was specifically designed for the offline-to-online problem in the concurrent work (Nakamoto et al., 2023).
Regarding regret, ReBRAC outperforms all other algorithms, with Cal-QL being the sole exception, showing slightly better results on average only in the AntMaze domain.
4.4 Ablating Design Choices
To better understand the source of improved performance, we conducted an ablation study on the modifications made to the algorithm. Results can be found in Table 8. Additional ablation studies
for all datasets can be found in Appendix G. One modification at a time was disabled, while all other changes were retained, including layer normalization in the critic network, additional linear layers in the actor and critic networks, adding an MSE penalty to the critic and actor loss. In the case of AntMaze, we also attempted to use the default  $\gamma$  value instead of the increased one. To further demonstrate the efficacy of our modifications, we also ran our implementation as equivalent to the original TD3+BC, with all changes disabled and hyperparameters were taken from the original paper. This experiment serves to show that the improved scores are due to the proposed changes in the algorithm and not just different implementations. Furthermore, we searched for the regularization parameter for our implementation of TD3+BC to demonstrate that tuning this parameter is not the sole source of improvement. Moreover, we tested the TD3 + BC by adding each of the ReBRAC's modifications independently to demonstrate that each individual modifications is not sufficient for achieving performance of ReBRAC where the combination of modification appears.
Additionally, we run ablation which to validate the importance of decoupling by disabling it and searching the best penalty parameter value from the set of all previously used values listed in the Appendix B.
Table 8: ReBRAC's design choices ablations: each modification was disabled while keeping all the others. For brevity, we report the mean of average normalized scores over four unseen training seeds across domains. We also include tuned results for TD3+BC to highlight that the improvement does not come from hyperparameter search. For dataset-specific results, please refer to Appendix G.
Ablation	Gym-MuJoCo	AntMaze	Adroit	All
TD3+BC, paper	-	27.3	0.0	-
TD3+BC, our	63.4	18.5	52.3	52.2
TD3+BC, tuned	71.8 (-10.9%)	27.9 (-62.9%)	53.5 (-25.9%)	58.3 (-19.2%)
TD3+BC w/ γ change	-	17.5 (-76.7%)	-	-
TD3+BC w/ LN	71.4 (-11.4%)	35.6 (-52.7%)	55.6 (-4.1%)	60.2 (-16.6%)
TD3+BC w/ large batch	14.4 (-82.1%)	0.0 (-100.0%)	1.6 (-97.2%)	7.9 (-89.0%)
TD3+BC w/ layer	71.2 (-11.6%)	44.1 (-41.4%)	56.4 (-2.7%)	61.9 (-14.2%)
ReBRAC w/o large batch	75.9 (-5.8%)	-	-	-
ReBRAC w large batch	-	41.0 (-45.6%)	55.4 (-4.6%)	-
ReBRAC w/o γ change	-	21.0 (-72.1%)	-	-
ReBRAC w/o LN	59.2 (-26.5%)	0.0 (-100.0%)	25.1 (-56.7%)	38.0 (-47.3%)
ReBRAC w/o layer	78.5 (-2.6%)	18.1 (-75.9%)	59.0 (+1.7%)	61.9 (-14.2%)
ReBRAC w/o actor penalty	22.8 (-71.7%)	0.1 (-99.8%)	0.0 (-100.0%)	11.4 (-84.2%)
ReBRAC w/o critic penalty	81.1 (+0.6%)	72.2 (-4.1%)	56.9 (-1.8%)	71.5 (-0.9%)
ReBRAC w/o decoupling	79.8 (-0.9%)	76.9 (+2.1%)	56.7 (-2.2%)	71.6 (-0.8%)
ReBRAC	80.6	75.3	58.0	72.2
The ablation results show that ReBRAC outperforms TD3+BC not because of different implementations or actor regularization parameter choice. All domains suffer when the LayerNorm is disabled, leading to the halved average performance overall. Removing additional layers leads to a notable drop in AntMaze tasks, while on Gym-MuJoCo decrease is small, and on Adroit tasks, we can see a slight boost. The algorithm fails to learn most tasks when the actor penalty is disabled. Notably, the critic penalty plays a minor role in improving the performance on most of the problems as well as the decoupling penalties. Using standard batch size on Gym-MuJoCo tasks significantly decreases final scores, while using the increased discount factor for AntMaze is crucial for obtaining state-of-the-art performance.
Based on the conducted ablations studies, the proposed configuration of design choices leads to the best performance on average. Note that tuning these choices for each task independently makes it possible to get even higher scores than we report in Section 4.1. We also can change the number of hidden layers in the networks, leading to better performance, see Section 4.5. But in real life, algorithm evaluation might be costly, so we limit ourselves to tuning only regularization parameters.
4.5 Stacking Even More Layers
As ablations show, the depth of the network plays an important role when solving AntMaze and HalfCheetah tasks (see Appendix G). We conduct additional experiments to check how the perfor
mance depends on the network depth in more detail. Ball et al. (2023) used networks of depth four when solving AntMaze tasks. Our goal is to find the point where the performance saturates. For this, we run our algorithm on AntMaze tasks increasing the number of layers to six while keeping other parameters unchanged. We attempt to increase actor and critic separately and at once. We also decreased each network's size by one layer similarly. Results can be found in Figure 2.
Several conclusions can be drawn from the results. First, further increments in the number of layers can lead to better results on the AntMaze domain when layers are scaled up to five. For six layers, performance drops or does not improve. Second, decreasing the critic's size leads to the worst performance on most datasets. Lastly, there is no clear pattern on how the performance changes even within a single domain. The drop on six layers is the only common feature that can be seen. On average, four critic layers and three actor layers were the best. Changing only the actor's network is more stable on average.
Figure 2: Impact of networks' depth on the final performance for the AntMaze tasks. Scores are averaged over four unseen training seeds. Shaded areas represent one standard deviation across seeds. These graphics demonstrate that one can achieve marginally better scores by tuning the number of layers for certain tasks.
4.6 Penalization Sensitivity Analysis
Following Kurenkov & Kolesnikov (2022), we demonstrate the sensitivity of ReBRAC to the choice of  $\beta_{1}$  and  $\beta_{2}$  hyperparameters under uniform policy selection on D4RL tasks using Expected Online Performance (EOP) and comparing it to the TD3+BC and IQL. EOP shows the best performance expected depending on the number of policies that can be deployed for online evaluation. Results are demonstrated in Table 9. As one can see, approximately ten policies are required for ReBRAC to attain ensemble-free state-of-the-art performance. ReBRAC's EOP is higher for any number of online policies when compared to TD3+BC and better than IQL on all domains when the evaluation budget is larger than two policies. See Appendix F for EOP separated by tasks.
Table 9: Expected Online Performance (Kurenkov & Kolesnikov, 2022) under uniform policy selection aggregated over D4RL domains across four training seeds. This demonstrates the sensitivity to the choice of hyperparameters given a certain budget for online evaluation. For dataset-specific results, please see Appendix F.
Domain	Algorithm	1 policy	2 policies	3 policies	5 policies	10 policies	15 policies	20 policies
Gym-MuJoCo	TD3+BC	49.8 ± 21.4	61.0 ± 14.5	65.3 ± 9.3	67.8 ± 3.9	-	-	-
IQL	65.0 ± 9.1	69.9 ± 5.6	71.7 ± 3.5	72.9 ± 1.7	73.6 ± 0.8	73.8 ± 0.7	74.0 ± 0.6
ReBRAC	62.0 ± 17.1	70.6 ± 9.9	73.3 ± 5.5	74.8 ± 2.1	75.6 ± 0.8	75.8 ± 0.6	76.0 ± 0.5
AntMaze	TD3+BC	6.9 ± 7.0	10.7 ± 6.8	13.0 ± 6.0	15.5 ± 4.6	-	-	-
IQL	29.8 ± 15.5	38.0 ± 15.4	43.1 ± 13.8	48.7 ± 10.2	53.2 ± 4.4	54.3 ± 2.1	54.7 ± 1.2
ReBRAC	67.9 ± 10.0	73.6 ± 7.4	76.1 ± 5.5	78.3 ± 3.4	79.9 ± 1.7	80.4 ± 1.1	-
Adroit	TD3+BC	23.6 ± 19.9	34.6 ± 17.7	40.6 ± 14.5	46.4 ± 9.8	-	-	-
IQL	53.1 ± 0.7	53.5 ± 0.6	53.7 ± 0.5	53.9 ± 0.3	54.1 ± 0.2	54.2 ± 0.2	54.2 ± 0.1
ReBRAC	44.1 ± 18.4	53.2 ± 10.9	56.1 ± 6.1	57.8 ± 2.3	58.6 ± 0.9	58.9 ± 0.7	59.1 ± 0.6
Ensemble-free offline RL methods. In recent years, many offline reinforcement learning algorithms were developed. TD3+BC (Fujimoto & Gu, 2021) represents a minimalist approach to offline RL, which incorporates a Behavioral Cloning component into the actor loss, enabling online actor-critic algorithms to operate in an offline setting. CQL (Kumar et al., 2020) drives the critic network to assign lower values to out-of-distribution state-action pairs and higher values to in-distribution pairs. IQL (Kostrikov et al., 2021) proposes a method for learning a policy without sampling out-of-distribution actions.
Despite this, more sophisticated methods may be necessary to achieve state-of-the-art results in an ensemble-free setup. For instance, Chen et al. (2022b); Akimov et al. (2022) pre-train different forms of encoders for actions, then optimize the actor to predict actions in the latent space. SPOT (Wu et al., 2022) pre-trains Variational Autoencoder and uses its uncertainty to penalize actor for sampling OOD actions while SAC-RND (Nikulin et al., 2023) applies Random Network Distillation and penalizes actor and critic.
Ensemble-based offline RL methods. A significant number of works in offline reinforcement learning have also leveraged ensemble methods for uncertainty estimation. The recently introduced SAC-N (An et al., 2021) algorithm outperformed all previous approaches on D4RL Gym-MuJoCo tasks; however, it necessitated large ensembles for some tasks, such as the hopper task, which required an ensemble size of 500 and imposed a significant computational burden. To mitigate this, the EDAC algorithm was introduced in the same work, which utilized ensemble diversification to reduce the ensemble size from 500 to 50. Despite the reduction, the ensemble size remains substantial compared to ensemble-free alternatives. It is worth mentioning that neither SAC-N nor EDAC is capable of solving the complex AntMaze tasks (Tarasov et al., 2022).
Another state-of-the-art algorithm in the Gym-MuJoCo tasks is RORL (Yang et al., 2022), which is a modification of SAC-N that makes the Q function more robust and smooth by perturbing state-action pairs with the use of out-of-distribution actions. RORL also requires an ensemble size of up to 20. On the other hand, MSG (Ghasemipour et al., 2022) utilizes independent targets for each ensemble member and achieves good performance on the Gym-MuJoCo tasks with an ensemble size of four but requires 64 ensemble members to achieve state-of-the-art performance on the AntMaze tasks.
Design choices ablations. Ablations of different design choices on established baselines are very limited, especially in offline RL. It is only shown by Fujimoto & Gu (2021) that CQL performs poorly if proposed non-algorithmic differences are eliminated. A parallel study (Ball et al., 2023) shows that some of the considered modifications (LayerNorm and networks depth) are important when used in online RL with offline data setting, which is different from pure offline.
6 Conclusion, Limitations, and Future work
In this work, we revisit recent advancements in the offline RL field over the last two years and incorporate a modest set of improvements to a previously established minimalistic TD3+BC baseline. Our experiments demonstrate that despite these limited updates, we can achieve more than competitive results on offline and offline-to-online D4RL and offline V-D4RL benchmarks under different hyperparameter budgets.
Despite the noteworthy results, our work is limited to one approach and a subset of possible design changes. It is imperative to explore the potential impact of these modifications on other offline RL methods (e.g., IQL, CQL, MSG) and to investigate other design choices used in offline RL, e.g., learning rate schedules, dropout (like in IQL), wider networks, or selection between stochastic and deterministic policies.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in Neural Information Processing Systems, 2021.
Jialong Wu, Haixu Wu, Zihan Qiu, Jianmin Wang, and Mingsheng Long. Supported policy optimization for offline reinforcement learning. arXiv preprint arXiv:2202.06239, 2022.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361, 2019.
Rui Yang, Chenjia Bai, Xiaoteng Ma, Zhaoran Wang, Chongjie Zhang, and Lei Han. Rorl: Robust offline reinforcement learning via conservative smoothing. arXiv preprint arXiv:2206.02829, 2022.
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint arXiv:1708.03888, 2017.
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training bert in 76 minutes. arXiv preprint arXiv:1904.00962, 2019.
Zifeng Zhuang, Kun LEI, Jinxin Liu, Donglin Wang, and Yilang Guo. Behavior proximal policy optimization. In International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=3c13LptpIph.
In order to generate the results presented in Table 2 Table 3 and Table 4, we conducted a hyperparameter search and selected the best results from the final evaluations for each dataset. Our algorithm was implemented using JAX for the D4RL benchmark. For V-D4RL, we implement our approach using PyTorch adopting the TD3+BC implementation from Clean Offline RL (Tarasov et al., 2022). The experiments were conducted on V100 and A100 GPUs.
Gym-MuJoCo and Adroit tasks. Our study utilized the latest version of the datasets - v2 for Gym-MuJoCo and v1 for Adroit. The agents were trained for one million steps and evaluated over ten episodes.
For ReBRAC, we fine-tuned the  $\beta_{1}$  parameter for the actor, which was selected from 0.001, 0.01, 0.05, 0.1. Similarly, the  $\beta_{2}$  parameter for the critic was selected from a range of 0, 0.001, 0.01, 0.1, 0.5. The selected best parameters for each dataset are reported in Table 11.
For TD3+BC here and in the AntMaze domain, we use the same grid used in ReBRAC for actor regularization parameter  $\alpha$  and add the default value of 0.4.
For IQL here and in the AntMaze domain, we selected  $\beta$  value from a range of 0.5, 1, 3, 6, 10 and IQL  $\tau$  value from a range of 0.5, 0.7, 0.9, 0.95. We used the implementation from Clean Offline RL (Tarasov et al., 2022) and kept other parameters unchanged.
For SAC-RND in Adroit domain we tune  $\beta_{1}$  (actor parameter) in the range of 0.5, 1.0, 2.5, 5.0, 10.0 and  $\beta_{2}$  (critic parameter) in the range of 0.01, 0.1, 1.0, 5.0, 10.0.
AntMaze tasks. In our work, we utilized v2 of the datasets. It's worth noting that previous studies have reported results using v0 datasets, which were found to contain numerous issues<sup>2</sup>. Each agent was trained for 1 million steps and evaluated over 100 episodes. Following Chen et al. (2022a), we modified the reward function by multiplying it by 100.
For ReBRAC, the  $\beta_{1}$  (actor) and  $\beta_{2}$  (critic) hyperparameters were carefully selected from the respective ranges of 0.0005, 0.001, 0.002, 0.003 and 0, 0.0001, 0.0005, 0.001. In addition, the actor and critic learning rates were optimized from 0.0001, 0.0002, 0.0003, 0.0005 and 0.0003, 0.0005, 0.001, respectively. The optimal hyperparameters for each dataset are presented in Table 11.
We also modified the  $\gamma$  value for ReBRAC when addressing these tasks, driven by the following motivation. The length of the episodes in AntMaze can be as long as 1000 steps, while the reward is sparse and can only be obtained at the end of the episode. As a result, the discount for the reward with the default  $\gamma$  can be as low as  $0.99^{1000} = 4 \cdot 10^{-5}$ , which is extremely low for signal propagation, even when multiplying the reward by 100. By increasing  $\gamma$  to 0.999, the minimum discount value becomes  $0.999^{1000} = 0.36$ , which is more favorable for signal propagation.
V-D4RL. We used single-task datasets without distraction with a resolution of  $84 \times 84$  pixels. For ReBRAC  $\beta_{1}$  (actor) parameter was selected from the range of  $\{0.03, 0.1, 0.3, 1.0\}$  and  $\beta_{2}$  (critic) parameter from the range of  $\{0.0, 0.001, 0.005, 0.01, 0.1\}$ .
Offline-to-offline. We used the same parameters for the offline-to-online setup with the only difference of setting  $\beta_{2}$  to zero and linearly decaying  $\beta_{1}$  to half of its initial value during the online stage.
$^{2}$ https://github.com/Farama-Foundation/D4RL/issues/77
B.1 ReBRAC
Table 10: ReBRAC's general hyperparameters.
Parameter	Value
optimizer	Adam Kingma & Ba (2014)
batch size	1024 on Gym-MuJoCo, 256 on other
learning rate (all networks)	1e-3 on Gym-MuJoCo, 3e-4 on Adroit and V-D4RL, 1e-4 on Antmaze
tau (τ)	5e-3
hidden dim (all networks)	256
num hidden layers (all networks)	3
gamma (γ)	0.999 on AntMaze, 0.99 on other
nonlinearity	ReLU
Table 11: ReBRAC's best hyperparameters used in D4RL benchmark.
Task Name	β1 (actor)	β2 (critic)
halfcheetah-random	0.001	0.1
halfcheetah-medium	0.001	0.01
halfcheetah-expert	0.01	0.01
halfcheetah-medium-expert	0.01	0.1
halfcheetah-medium-replay	0.01	0.001
halfcheetah-full-replay	0.001	0.1
hopper-random	0.001	0.01
hopper-medium	0.01	0.001
hopper-expert	0.1	0.001
hopper-medium-expert	0.1	0.01
hopper-medium-replay	0.05	0.5
hopper-full-replay	0.01	0.01
walker2d-random	0.01	0.0
walker2d-medium	0.05	0.1
walker2d-expert	0.01	0.5
walker2d-medium-expert	0.01	0.01
walker2d-medium-replay	0.05	0.01
walker2d-full-replay	0.01	0.01
antmaze-umaze	0.003	0.002
antmaze-umaze-diverse	0.003	0.001
antmaze-medium-play	0.001	0.0005
antmaze-medium-diverse	0.001	0.0
antmaze-large-play	0.002	0.001
antmaze-large-diverse	0.002	0.002
pen-human	0.1	0.5
pen-cloned	0.05	0.5
pen-expert	0.01	0.01
door-human	0.1	0.1
door-cloned	0.01	0.1
door-expert	0.05	0.01
hammer-human	0.01	0.5
hammer-cloned	0.1	0.5
hammer-expert	0.01	0.01
relocate-human	0.1	0.01
relocate-cloned	0.1	0.01
relocate-expert	0.05	0.01
Table 12: ReBRAC's best hyperparameters used in V-D4RL benchmark.
Task Name	β1 (actor)	β2 (critic)
walker-walk-random	0.03	0.1
walker-walk-medium	0.03	0.005
walker-walk-expert	0.1	0.01
walker-walk-medium-expert	0.3	0.005
walker-walk-medium-replay	0.3	0.01
cheetah-run-random	0.1	0.01
cheetah-run-medium	0.1	0.1
cheetah-run-expert	0.01	0.01
cheetah-run-medium-expert	1.0	0.001
cheetah-run-medium-replay	0.03	0.1
humanoid-walk-random	1.0	0.01
humanoid-walk-medium	1.0	0.005
humanoid-walk-expert	1.0	0.1
humanoid-walk-medium-expert	1.0	0.005
humanoid-walk-medium-replay	1.0	0.001
Table 13: IQL's best hyperparameters used in D4RL benchmark.
Task Name	β	IQL τ
halfcheetah-random	3.0	0.95
halfcheetah-medium	3.0	0.95
halfcheetah-expert	6.0	0.9
halfcheetah-medium-expert	3.0	0.7
halfcheetah-medium-replay	3.0	0.95
halfcheetah-full-replay	1.0	0.7
hopper-random	1.0	0.95
hopper-medium	3.0	0.7
hopper-expert	3.0	0.5
hopper-medium-expert	6.0	0.7
hopper-medium-replay	6.0	0.7
hopper-full-replay	10.0	0.9
walker2d-random	0.5	0.9
walker2d-medium	6.0	0.5
walker2d-expert	6.0	0.9
walker2d-medium-expert	1.0	0.5
walker2d-medium-replay	0.5	0.7
walker2d-full-replay	1.0	0.7
antmaze-umaze	10.0	0.7
antmaze-umaze-diverse	10.0	0.95
antmaze-medium-play	6.0	0.9
antmaze-medium-diverse	6.0	0.9
antmaze-large-play	10.0	0.9
antmaze-large-diverse	6.0	0.9
pen-human	1.0	0.95
pen-cloned	10.0	0.9
pen-expert	10.0	0.8
door-human	0.5	0.9
door-cloned	6.0	0.7
door-expert	0.5	0.7
hammer-human	3.0	0.9
hammer-cloned	6.0	0.7
hammer-expert	0.5	0.95
relocate-human	1.0	0.95
relocate-cloned	6.0	0.9
relocate-expert	10.0	0.9
B.2 IQL
B.3 TD3+BC
Table 14: TD3+BC's best hyperparameters used in D4RL benchmark.
Task Name	α
halfcheetah-random	0.001
halfcheetah-medium	0.01
halfcheetah-expert	0.4
halfcheetah-medium-expert	0.1
halfcheetah-medium-replay	0.05
halfcheetah-full-replay	0.01
hopper-random	0.4
hopper-medium	0.05
hopper-expert	0.1
hopper-medium-expert	0.1
hopper-medium-replay	0.4
hopper-full-replay	0.01
walker2d-random	0.001
walker2d-medium	0.4
walker2d-expert	0.05
walker2d-medium-expert	0.1
walker2d-medium-replay	0.1
walker2d-full-replay	0.1
antmaze-umaze	0.4
antmaze-umaze-diverse	0.4
antmaze-medium-play	0.003
antmaze-medium-diverse	0.003
antmaze-large-play	0.003
antmaze-large-diverse	0.003
pen-human	0.1
pen-cloned	0.4
pen-expert	0.4
door-human	0.1
door-cloned	0.4
door-expert	0.1
hammer-human	0.4
hammer-cloned	0.4
hammer-expert	0.4
relocate-human	0.1
relocate-cloned	0.1
relocate-expert	0.4
Table 15: SAC-RND's best hyperparameters used in D4RL Adroit tasks.
Task Name	β1 (actor)	β2 (critic)
pen-human	1.0	10.0
pen-cloned	2.5	0.01
pen-expert	10.0	5.0
door-human	5.0	0.01
door-cloned	5.0	1.0
door-expert	10.0	10.0
hammer-human	10.0	0.01
hammer-cloned	1.0	1.0
hammer-expert	2.5	10.0
relocate-human	5.0	0.01
relocate-cloned	5.0	1.0
relocate-expert	10.0	10.0
C Comparison to Ensemble-based Methods