SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Denis Tarasov Vladislav Kurenkov Alexander Nikulin Sergey Kolesnikov Tinkoff
{den.tarasov, v.kurenkov, a.p.nikulin, s.s.kolesnikov}@tinkoff.ai
Recent years have witnessed significant advancements in offline reinforcement learning (RL), resulting in the development of numerous algorithms with varying degrees of complexity. While these algorithms have led to noteworthy improvements, many incorporate seemingly minor design choices that impact their effectiveness beyond core algorithmic advances. However, the effect of these design choices on established baselines remains understudied. In this work, we aim to bridge this gap by conducting a retrospective analysis of recent works in offline RL and propose ReBRAC, a minimalistic algorithm that integrates such design elements built on top of the TD3+BC method. We evaluate ReBRAC on 51 datasets with both proprioceptive and visual state spaces using D4RL and V-D4RL benchmarks, demonstrating its state-of-the-art performance among ensemble-free methods in both offline and offline-to-online settings. To further illustrate the efficacy of these design choices, we perform a large-scale ablation study and hyperparameter sensitivity analysis on the scale of thousands of experiments. $^{1}$
Interest of the reinforcement learning (RL) community in the offline setting has led to a myriad of new algorithms specifically tailored to learning highly performant policies without the ability to interact with an environment (Levine et al., 2020; Prudencio et al., 2022). Yet, similar to the advances in online RL (Engstrom et al., 2020; Henderson et al., 2018), many of those algorithms come with an added complexity - design and implementation choices beyond core algorithmic innovations, requiring a delicate effort in reproduction, hyperparameter tuning, and causal attribution of performance gains.
Indeed, the issue of complexity was already raised in the offline RL community by Fujimoto & Gu (2021); the authors highlighted veiled design and implementation-level adjustments (e.g., different architectures or actor pre-training) and then demonstrated how a simple behavioral cloning regularization added to the TD3 (Fujimoto et al., 2018) constitutes a strong baseline in the offline setting. This minimalistic and uncluttered algorithm, TD3+BC, has become a de-facto standard baseline to be compared against. Indeed, most new algorithms juxtapose against it and claim significant gains over (Akimov et al., 2022; An et al., 2021; Nikulin et al., 2023; Wu et al., 2022; Chen et al., 2022b; Ghasemipour et al., 2022). However, application of newly emerged design and implementation choices to this baseline is still missing.
In this work, we build upon Fujimoto & Gu (2021) line of research and ask: what is the extent to which newly emerged minor design choices can advance the minimalistic offline RL algorithm? The answer is illustrated in Figure 1: we propose an extension to TD3+BC, ReBRAC (Section 3), that simply adds on recently appeared design decisions upon it. We test our algorithm on both proprioceptive and
Our implementation is available at https://github.com/DT6A/ReBRAC
(b)
Figure 1: (a) The schema of our approach ReBRAC (b) Performance profiles (c) Probability of improvement. The curves (Agarwal et al., 2021) are for D4RL benchmark spanning all Gym-MuJoCo, AntMaze, and Adroit datasets (Fu et al., 2020).
visual state space problems using D4RL (Fu et al., 2020) and V-D4RL (Lu et al., 2022) benchmarks (Section 4) demonstrating its state-of-the-art performance across ensemble-free methods. Moreover, our approach demonstrates state-of-the-art performance in offline-to-offline setup on D4RL datasets (Section 4.3) while not being specifically designed for this setup. To further highlight the efficacy of the proposed modifications, we then conduct a large-scale ablation study (Section 4.4). We hope the described approach can serve as a strong baseline under different hyperparameter search budgets (Section 4.6), further accentuating the importance of seemingly minor design choices introduced along with core algorithmic innovations.
A standard Reinforcement Learning problem is defined as a Markov Decision Process (MDP) with the tuple  $\{S, A, P, R, \gamma\}$ , where  $S \subset \mathbb{R}^n$  is the state space,  $A \subset \mathbb{R}^m$  is the action space,  $P: S \times A \to S$  is the transition function,  $R: S \times A \to \mathbb{R}$  is the reward function, and  $\gamma \in (0,1)$  is the discount factor. The ultimate objective is to find a policy  $\pi(a|s)$  that maximizes the cumulative discounted return  $\mathbb{E}_{\pi} \sum_{t=0}^{\infty} \gamma^t R(s_t, a_t)$ . This policy improves by interacting with the environment, observing states, and taking actions that provide rewards.
In offline RL, policies cannot interact with the environment and can only access a static transaction dataset  $D$  collected by one or more other policies. This setting presents new challenges, such as estimating values for state-action pairs not included in the dataset while exploration is unavailable (Levine et al., 2020).
2.2 Behavior Regularized Actor-Critic
Behavior Regularized Actor-Critic (BRAC) is an offline RL framework introduced in Wu et al. (2019). The core idea behind BRAC is that actor-critic algorithms can be penalized in two ways to solve offline RL tasks: actor penalization and critic penalization. In this framework, the actor objective is represented as in Equation (1), and the critic objective as in Equation (2), where  $F$  is a divergence function between dataset actions and policy actions distributions. The differences from a vanilla actor-critic are highlighted in blue.
$$
\pi = \underset {\pi} {\operatorname {a r g m a x}} \mathbb {E} _ {(s, a) \sim D} [ Q _ {\theta} (s, \pi (s)) - \alpha \cdot F (\pi (s), a) ] \tag {1}
$$
$$
\theta = \underset {\theta} {\operatorname {a r g m i n}} \mathbb {E} _ {\substack {(s, a, r, s ^ {\prime}, \hat {a} ^ {\prime}) \sim D \\ a ^ {\prime} \sim \pi (s ^ {\prime})}} \left[ \left(Q _ {\theta} (s, a) - \left(r + \gamma \left(Q _ {\bar {\theta}} \left(s ^ {\prime}, a ^ {\prime}\right) - \alpha \cdot F \left(a ^ {\prime}, \hat {a} ^ {\prime}\right)\right)\right)\right) ^ {2} \right] \tag{2}
$$
In the original work, various choices of  $F$  were evaluated when used as the regularization term for the actor or critic. The authors tested KL divergence, Kernel MMD, and Wasserstein distance but did not observe any consistent advantage. Finally, it is essential to note that, originally, both regularizations coefficients had the same weight.
Subsequently, TD3+BC (Fujimoto & Gu, 2021) was introduced, utilizing Mean Squared Error (MSE) as the regularization term  $\bar{F}$  for the actor. TD3+BC is considered to be the minimalist approach to offline RL as it modifies existing RL algorithms by simply adding behavior cloning term into actor loss which is easy to implement and does not bring any significant computational overhead.
3 ReBRAC: Distilling Key Design Choices
In this section, we describe the proposed method along with the discussion of the new design choices met in the offline RL literature (Table 1). Our approach is a more general version of BRAC built on top of the TD3+BC (Fujimoto & Gu, 2021) algorithm with different modifications in design while keeping it simple (Figure 1). We refer to our method as Revisited BRAC (ReBRAC).
Table 1: Adoption of implementation and design choices beyond core algorithmic advancements in some recently introduced algorithms.
Modification	TD3+BC	CQL	EDAC	MSG	CNF	LB-SAC	SAC-RND
Deeper networks	✘	✓	✓	✓	✓	✓	✓
Larger batches	✘	✘	✘	✘	✓	✓	✓
Layer Normalization	✘	✘	✘	✘	✘	✓	✓
Decoupled penalization	✘	✘	✘	✘	✘	✘	✓
Adjusted discount factor	✘	✘	✘	✘	✘	✘	✓
Deeper Networks The use of deeper neural networks has been a critical factor in the success of many Deep Learning models, with model quality generally increasing as depth increases, provided there is enough data to support this scaling (Kaplan et al., 2020). Similarly, recent studies in RL (Neumann & Gros, 2022; Sinha et al., 2020) and offline RL (Lee et al., 2022; Kumar et al., 2022) have demonstrated the importance of depth in achieving high performance. Although most offline RL algorithms are based on SAC (Haarnoja et al., 2018) or TD3 (Fujimoto et al., 2018), which by default employ two hidden layers, recent work (Kumar et al., 2020) uses three hidden layers for SAC instead, which appears to be an important change (Fujimoto & Gu, 2021). The change in network size has been adopted by later works (An et al., 2021; Yang et al., 2022; Zhuang et al., 2023; Nikulin et al., 2023) and may be one of the critical modifications that improve final performance.
The original BRAC and TD3+BC algorithms used only two hidden layers for their actor and critic networks, while most state-of-the-art solutions use deeper networks. Specifically, three hidden layers have become a common choice for recent offline RL algorithms. In ReBRAC, we follow this trend and use three hidden layers for the actor and critic networks. Additionally, we provide an ablation study in Section 4.5 to investigate the effect of the number of layers on ReBRAC's performance.
LayerNorm LayerNorm (Ba et al., 2016) is a widely used technique in deep learning that helps improve network convergence. In Hiraoka et al. (2021), authors add dropout and LayerNorm to different RL algorithms, notably boosting their performance. This technique is also applied in Smith et al. (2022), and it appears that boost is achieved primarily because of LayerNorm. Specifically, in offline RL, various studies have tested the effect of normalizations between layers (Bhatt et al., 2019; Kumar et al., 2022; Nikulin et al., 2022, 2023). A parallel study by Ball et al. (2023) empirically shows that LayerNorm helps to prevent catastrophic value extrapolation for the Q function when using offline datasets in online RL. Following these works, in our approach, we also apply LayerNorm between each layer of the critic networks.
Larger Batches Another technique to accelerate neural network convergence is large batch optimization (You et al., 2017, 2019). While studying batch sizes larger than 256 is limited, some prior works have used them. For instance, the convergence of SAC-N was accelerated in Nikulin et al. (2022). More recently proposed algorithms also use larger batches for training, although without providing detailed analyses (Akimov et al., 2022; Nikulin et al., 2023).
The usage of large batches in offline RL is still understudied, and its benefits and limitations are not fully understood. Our experiments show that in some domains, using large batches can lead to significant performance improvements, while in others, it might not have a notable impact or
even drop the performance (see Table 8). We increased the batch size to 1024 samples and scaled the learning rate for D4RL Gym-MuJoCo tasks, following the approach proposed by Nikulin et al. (2022).
Actor and critic penalty decoupling The original BRAC framework proposed penalizing the actor and the critic with the same magnitude. Most of the previous algorithms restrict only actor (Fujimoto & Gu, 2021; Wu et al., 2022) or only critic (Kumar et al., 2020; An et al., 2021; Ghasemipour et al., 2022). TD3-CVAE (Rezaifar et al., 2022) penalizes both using the same coefficients, while another study, SAC-RND (Nikulin et al., 2023), shows that decoupling the penalization in offline RL is beneficial for algorithm performance, although ablations on using only one of the penalties are missing.
Our method allows simultaneous penalization of actor and critic with decoupled parameters. Inspired by TD3+BC (Fujimoto & Gu, 2021), Mean Squared Error (MSE) is used as a divergence function  $F$  which we found simple and effective. The actor objective is shown in Equation (3), and the critic objective is shown in Equation (4). Differences from the original actor-critic are highlighted in red. Following TD3+BC (Fujimoto & Gu, 2021), the Q function is normalized to make the algorithm less sensitive to regularization parameters. Nonetheless, we forego the utilization of state normalization, as initially suggested in TD3 + BC, driven by our intention to execute the algorithm online and the observation that this adjustment typically results in negligible impact. Since our approach principally builds upon TD3+BC, the differences in their performances should be considered the most important ones and do not appear only because of the additional hyperparameters search.
$$
\pi = \underset {\pi} {\operatorname {a r g m a x}} \mathbb {E} _ {(s, a) \sim D} \left[ Q _ {\theta} (s, \pi (s)) - \beta_ {1} \cdot (\pi (s) - a) ^ {2} \right] \tag {3}
$$
$$
\theta = \underset {\theta} {\operatorname {a r g m i n}} \mathbb {E} _ {\substack {(s, a, r, s ^ {\prime}, \hat {a} ^ {\prime}) \sim D \\ a ^ {\prime} \sim \pi \left(s ^ {\prime}\right)}} \left[ \left(Q _ {\theta} (s, a) - \left(r + \gamma \left(Q _ {\bar {\theta}} \left(s ^ {\prime}, a ^ {\prime}\right) - \beta_ {2} \cdot \left(a ^ {\prime} - \hat {a} ^ {\prime}\right) ^ {2}\right)\right)\right) ^ {2} \right] \tag{4}
$$
Discount factor  $\gamma$  value change The choice of discount factor is an important aspect in solving RL problems (Jiang et al., 2015). A recent study (Hu et al., 2022) suggests that decreasing the default value of  $\gamma$  from 0.99 may lead to better results in offline RL settings. In contrast, in SPOT (Wu et al., 2022), the authors increased the value of  $\gamma$  up to 0.995 when fine-tuning AntMaze tasks with sparse rewards, which resulted in state-of-the-art solutions. Similarly, in the offline setting SAC-RND (Nikulin et al., 2023), increasing  $\gamma$  also achieved high performance on the same domain. The choice of increased  $\gamma$  for AntMaze tasks was motivated by the sparse reward, i.e., a low  $\gamma$  value may not propagate the training signal well. However, further ablations are needed to understand if the change in the parameter is directly responsible for the improved performance. In our experiments, we also find that increasing  $\gamma$  from the default value of 0.99 to 0.999 is vital for improved performance on this set of tasks (see Table 8).
4.1 Evaluation on offline D4RL
We evaluate the proposed approach on three sets of D4RL tasks: Gym-MuJoCo, AntMaze, and Adroit. For each domain, we consider all of the available datasets. We compare our results to several ensemble-free baselines, including TD3+BC (Fujimoto & Gu, 2021), IQL (Kostrikov et al., 2021), CQL (Kumar et al., 2020) and SAC-RND (Nikulin et al., 2023).
The majority of the hyperparameters are adopted from TD3+BC, while  $\beta_{1}$  and  $\beta_{2}$  parameters from Equation (3) and Equation (4) are tuned. We examine the sensitivity to these parameters in Section 4.6. For a complete overview of the experimental setup and details, see Appendix A.
Following Wu et al. (2022), we tune hyperparameters over four seeds (referred to as training seeds) and evaluate the best parameters over ten new seeds (referred to as unseen training seeds), reporting the average performance of the last checkpoints for D4RL tasks. On V-D4RL, we use two and five seeds, respectively. This helps to avoid overfitting during hyperparameters search and outputs more just and reproducible results. For a fair comparison, we tune TD3+BC and IQL following the same protocol. We also rerun SAC-RND on Gym-MuJoCo and AntMaze tasks and tune it for the Adroit
Table 2: Average normalized score over the final evaluation and ten unseen training seeds on GymMuJoCo tasks. CQL scores were taken from An et al. (2021). The symbol  $\pm$  represents the standard deviation across the seeds. To make a fair comparison against TD3+BC and IQL, we extensively tuned their hyperparameters.
Task Name	TD3+BC	IQL	CQL	SAC-RND	ReBRAC, our
halfcheetah-random	30.9 ± 0.4	19.5 ± 0.8	31.1 ± 3.5	27.6 ± 2.1	29.5 ± 1.5
halfcheetah-medium	54.7 ± 0.9	50.0 ± 0.2	46.9 ± 0.4	66.4 ± 1.4	65.6 ± 1.0
halfcheetah-expert	93.4 ± 0.4	95.5 ± 2.1	97.3 ± 1.1	102.6 ± 4.2	105.9 ± 1.7
halfcheetah-medium-expert	89.1 ± 5.6	92.7 ± 2.8	95.0 ± 1.4	108.1 ± 1.5	101.1 ± 5.2
halfcheetah-medium-replay	45.0 ± 1.1	42.1 ± 3.6	45.3 ± 0.3	51.2 ± 3.2	51.0 ± 0.8
halfcheetah-full-replay	75.0 ± 2.5	75.0 ± 0.7	76.9 ± 0.9	81.2 ± 1.3	82.1 ± 1.1
hopper-random	8.5 ± 0.7	10.1 ± 5.9	5.3 ± 0.6	19.6 ± 12.4	8.1 ± 2.4
hopper-medium	60.9 ± 7.6	65.2 ± 4.2	61.9 ± 6.4	91.1 ± 10.1	102.0 ± 1.0
hopper-expert	109.6 ± 3.7	108.8 ± 3.1	106.5 ± 9.1	109.8 ± 0.5	100.1 ± 8.3
hopper-medium-expert	87.8 ± 10.5	85.5 ± 29.7	96.9 ± 15.1	109.8 ± 0.6	107.0 ± 6.4
hopper-medium-replay	55.1 ± 31.7	89.6 ± 13.2	86.3 ± 7.3	97.2 ± 9.0	98.1 ± 5.3
hopper-full-replay	97.9 ± 17.5	104.4 ± 10.8	101.9 ± 0.6	107.4 ± 0.8	107.1 ± 0.4
walker2d-random	2.0 ± 3.6	11.3 ± 7.0	5.1 ± 1.7	18.7 ± 6.9	18.4 ± 4.5
walker2d-medium	77.7 ± 2.9	80.7 ± 3.4	79.5 ± 3.2	92.7 ± 1.2	82.5 ± 3.6
walker2d-expert	110.0 ± 0.6	96.9 ± 32.3	109.3 ± 0.1	104.5 ± 22.8	112.3 ± 0.2
walker2d-medium-expert	110.4 ± 0.6	112.1 ± 0.5	109.1 ± 0.2	104.6 ± 11.2	111.6 ± 0.3
walker2d-medium-replay	68.0 ± 19.2	75.4 ± 9.3	76.8 ± 10.0	89.4 ± 3.8	77.3 ± 7.9
walker2d-full-replay	90.3 ± 5.4	97.5 ± 1.4	94.2 ± 1.9	105.3 ± 3.2	102.2 ± 1.7
Average	70.3	72.9	73.6	82.6	81.2
Table 3: Average normalized score over the final evaluation and ten unseen training seeds on AntMaze tasks. CQL scores were taken from Ghasemipour et al. (2022). The symbol  $\pm$  represents the standard deviation across the seeds. To make a fair comparison against TD3+BC and IQL, we extensively tuned their hyperparameters.
Task Name	TD3+BC	IQL	CQL	SAC-RND	ReBRAC, our
antmaze-umaze	66.3 ± 6.2	83.3 ± 4.5	74.0	97.0 ± 1.5	97.8 ± 1.0
antmaze-umaze-diverse	53.8 ± 8.5	70.6 ± 3.7	84.0	66.0 ± 25.0	88.3 ± 13.0
antmaze-medium-play	26.5 ± 18.4	64.6 ± 4.9	61.2	38.5 ± 29.4	84.0 ± 4.2
antmaze-medium-diverse	25.9 ± 15.3	61.7 ± 6.1	53.7	74.7 ± 10.7	76.3 ± 13.5
antmaze-large-play	0.0 ± 0.0	42.5 ± 6.5	15.8	43.9 ± 29.2	60.4 ± 26.1
antmaze-large-diverse	0.0 ± 0.0	27.6 ± 7.8	14.9	45.7 ± 28.5	54.4 ± 25.1
Average	28.7	58.3	50.6	60.9	76.8
domain. In other cases, we report results from previous works, meaning that scores for other methods can be lower if evaluated under our protocol.
The results of our tests on D4RL's Gym-MuJoCo, AntMaze, and Adroit tasks are available in Table 2, Table 3, Table 4, respectively. The mean-wise best results among algorithms are highlighted with bold, and the second best performance is underlined. Our approach, ReBRAC, achieves state-of-the-art results on Gym-MuJoCo, AntMaze, and Adroit tasks outperforming all baselines on average, except SAC-RND on Gym-MuJoCo tasks, which is slightly better. Performance profiles and probability of improvement (Agarwal et al., 2021) in Figure 1b and Figure 1c also demonstrate that ReBRAC is competitive when compared to the algorithms that we contrast against. Our method is also comparable to ensemble-based approaches (see Appendix C for additional comparisons).
4.2 Evaluation on offline V-D4RL
In addition to testing ReBRAC on D4RL, we evaluated its performance on V-D4RL benchmark (Lu et al., 2022). Our motivation for doing so was the fact that scores on D4RL Gym-MuJoCo tasks have saturated in recent years, and even after the introduction of ensemble-based offline RL methods
Table 4: Average normalized score over the final evaluation and ten unseen training seeds on Adroit tasks. BC and CQL scores were taken from Yang et al. (2022). The symbol  $\pm$  represents the standard deviation across the seeds. To make a fair comparison against TD3+BC and IQL, we extensively tuned their hyperparameters.
Task Name	BC	TD3+BC	IQL	CQL	SAC-RND	ReBRAC, our
pen-human	34.4	81.8 ± 14.9	81.5 ± 17.5	37.5	5.6 ± 5.8	103.5 ± 14.1
pen-cloned	56.9	61.4 ± 19.3	77.2 ± 17.7	39.2	2.5 ± 6.1	91.8 ± 21.7
pen-expert	85.1	146.0 ± 7.3	133.6 ± 16.0	107.0	45.4 ± 22.9	154.1 ± 5.4
door-human	0.5	-0.1 ± 0.0	3.1 ± 2.0	9.9	0.0 ± 0.1	0.0 ± 0.0
door-cloned	-0.1	0.1 ± 0.6	0.8 ± 1.0	0.4	0.2 ± 0.8	1.1 ± 2.6
door-expert	34.9	84.6 ± 44.5	105.3 ± 2.8	101.5	73.6 ± 26.7	104.6 ± 2.4
hammer-human	1.5	0.4 ± 0.4	2.5 ± 1.9	4.4	-0.1 ± 0.1	0.2 ± 0.2
hammer-cloned	0.8	0.8 ± 0.7	1.1 ± 0.5	2.1	0.1 ± 0.4	6.7 ± 3.7
hammer-expert	125.6	117.0 ± 30.9	129.6 ± 0.5	86.7	24.8 ± 39.4	133.8 ± 0.7
relocate-human	0.0	-0.2 ± 0.0	0.1 ± 0.1	0.2	0.0 ± 0.0	0.0 ± 0.0
relocate-cloned	-0.1	-0.1 ± 0.1	0.2 ± 0.4	-0.1	0.0 ± 0.0	0.9 ± 1.6
relocate-expert	101.3	107.3 ± 1.6	106.5 ± 2.5	95.0	3.4 ± 4.5	106.6 ± 3.2
Average w/o expert	11.7	18.0	20.8	11.7	1.0	25.5
Average	36.7	49.9	53.4	40.3	12.9	58.6
Table 5: Average normalized score over the final evaluation and five unseen training seeds on V-D4RL tasks. The score is mapped from a range of [0, 1000] to [0, 100]. The symbol  $\pm$  represents the standard deviation across the seeds.
Environment	Offline DV2	DrQ+BC	CQL	BC	LOMPO	ReBRAC, our
walker-walk	random	28.7 ±13.0	5.5 ±0.9	14.4 ±12.4	2.0 ±0.2	21.9 ±8.1	15.9 ± 2.3
mixed	56.5 ±18.1	28.7 ±6.9	11.4 ±12.4	16.5 ±4.3	34.7 ±19.7	41.6 ± 8.0
medium	34.1 ±19.7	46.8 ±2.3	14.8 ±16.1	40.9 ±3.1	43.4 ±11.1	52.5 ± 3.2
medexp	43.9 ±34.4	86.4 ±5.6	56.4 ±38.4	47.7 ±3.9	39.2 ±19.5	92.7 ± 1.3
expert	4.8 ±0.6	68.4 ±7.5	89.6 ±6.0	91.5 ±3.9	5.3 ±7.7	81.4 ± 10.0
cheetah-run	random	31.7 ±2.7	5.8 ±0.6	5.9 ±8.4	0.0 ±0.0	11.4 ±5.1	12.9 ± 2.2
mixed	61.6±1.0	44.8 ±3.6	10.7 ±12.8	25.0 ±3.6	36.3 ±13.6	46.8 ± 0.7
medium	17.2 ±3.5	53.0 ±3.0	40.9 ±5.1	51.6 ±1.4	16.4 ±8.3	59.0 ± 0.7
medexp	10.4 ±3.5	50.6 ±8.2	20.9 ±5.5	57.5 ±6.3	11.9 ±1.9	58.3 ± 11.7
expert	10.9 ±3.2	34.5 ±8.3	61.5 ±4.3	67.4 ±6.8	14.0 ±3.8	35.6 ± 5.3
humanoid-walk	random	0.1 ±0.0	0.1 ±0.0	0.2 ±0.1	0.1 ±0.0	0.1 ±0.0	0.1 ± 0.0
mixed	0.2 ±0.1	15.9 ±3.8	0.1 ±0.0	18.8 ±4.2	0.2 ±0.0	16.0 ± 2.7
medium	0.2 ±0.1	6.2 ±2.4	0.1 ±0.0	13.5 ±4.1	0.1 ±0.0	9.0 ± 2.3
medexp	0.1 ±0.0	7.0 ±2.3	0.1 ±0.0	17.2 ±4.7	0.2 ±0.0	7.8 ± 2.4
expert	0.2 ±0.1	2.7 ±0.9	1.6 ±0.5	6.1 ±3.7	0.1 ±0.0	2.9 ± 0.9
Average	20.0	30.4	21.9	30.3	15.6	35.5
by An et al. (2021), there has been no notable progress on these tasks. On the other hand, V-D4RL provides a similar set of problems, with datasets collected in the same way as in D4RL but with the agent's observations now being images from the environment.
We tested our algorithm on all available single-task datasets without distractors and compared it to the baselines from the original V-D4RL work (Lu et al., 2022). The results are reported in Table 5. Our proposed approach achieves state-of-the-art or close-to-state-of-the-art results on most of the tasks, and it is the only method that, on average, performs notably better than naive Behavioral Cloning.
4.3 Evaluation on offline-to-online D4RL
The evaluation of offline-to-online performance is a pivotal aspect for reinforcement learning (RL) algorithms, particularly in light of recent developments. In this context, we conducted additional tests on ReBRAC, as it stands out as a promising algorithm for several compelling reasons.
First and foremost, ReBRAC demonstrates a remarkable proficiency following offline pre-training. Secondly, our algorithm shares notable similarities with TD3+BC, a method that has exhibited effectiveness in online fine-tuning as observed by Beeson et al. (Beeson & Montana, 2022)
For the sake of simplicity, we opted to disable critic penalization during the online fine-tuning. Furthermore, we linearly decay the actor's penalty to half of its initial value, following the approach described by Beeson & Montana (2022). Notably, no hyperparameter tuning was performed in this process.
To evaluate our approach in the offline-to-online setting, we followed the methodology outlined by Tarasov et al. (2022). In our comparative analysis, we consider the following algorithms: TD3+BC (Fujimoto & Gu, 2021), IQL (Kostrikov et al., 2021), CQL (Kumar et al., 2020), SPOT (Wu et al., 2022), and Cal-CQL (Nakamoto et al., 2023). The scores after the offline stage and online tuning, are reported in Table 6. We also provide finetuning cumulative regret proposed by Nakamoto et al. (2023) in Table 7.