Title: Revisiting the Minimalist Approach to Offline Reinforcement Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Offline Reinforcement Learning
Abstract: Recent years have witnessed significant advancements in offline reinforcement learning (RL), resulting in the development of numerous algorithms with varying degrees of complexity. While these algorithms have led to noteworthy improvements, many incorporate seemingly minor design choices that impact their effectiveness beyond core algorithmic advances. However, the effect of these design choices on established baselines remains understudied. In this work, we aim to bridge this gap by conducting a retrospective analysis of recent works in offline RL and propose ReBRAC, a minimalistic algorithm that integrates such design elements built on top of the TD3+BC method. We evaluate ReBRAC on 51 datasets with both proprioceptive and visual state spaces using D4RL and V-D4RL benchmarks, demonstrating its state-of-the-art performance among ensemble-free methods in both offline and offline-to-online settings. To further illustrate the efficacy of these design choices, we perform a large-scale ablation study and hyperparameter sensitivity analysis on the scale of thousands of experiments.

Revisiting the Minimalist Approach to Offline Reinforcement Learning

Abstract

Recent years have witnessed significant advancements in offline reinforcement learning (RL), resulting in the development of numerous algorithms with varying degrees of complexity. While these algorithms have led to noteworthy improvements, many incorporate seemingly minor design choices that impact their effectiveness beyond core algorithmic advances. However, the effect of these design choices on established baselines remains understudied. In this work, we aim to bridge this gap by conducting a retrospective analysis of recent works in offline RL and propose ReBRAC, a minimalistic algorithm that integrates such design elements built on top of the TD3+BC method. We evaluate ReBRAC on 51 datasets with both proprioceptive and visual state spaces using D4RL and V-D4RL benchmarks, demonstrating its state-of-the-art performance among ensemble-free methods in both offline and offline-to-online settings. To further illustrate the efficacy of these design choices, we perform a large-scale ablation study and hyperparameter sensitivity analysis on the scale of thousands of experiments. 1

1 Introduction

Interest of the reinforcement learning (RL) community in the offline setting has led to a myriad of new algorithms specifically tailored to learning highly performant policies without the ability to interact with an environment (Levine et al., 2020; Prudencio et al., 2022). Yet, similar to the advances in online RL (Engstrom et al., 2020; Henderson et al., 2018), many of those algorithms come with an added complexity - design and implementation choices beyond core algorithmic innovations, requiring a delicate effort in reproduction, hyperparameter tuning, and causal attribution of performance gains.

Indeed, the issue of complexity was already raised in the offline RL community by Fujimoto & Gu (2021); the authors highlighted veiled design and implementation-level adjustments (e.g., different architectures or actor pre-training) and then demonstrated how a simple behavioral cloning regularization added to the TD3 (Fujimoto et al., 2018) constitutes a strong baseline in the offline setting. This minimalistic and uncluttered algorithm, TD3+BC, has become a de-facto standard baseline to be compared against. Indeed, most new algorithms juxtapose against it and claim significant gains over (Akimov et al., 2022; An et al., 2021; Nikulin et al., 2023; Wu et al., 2022; Chen et al., 2022b; Ghasemipour et al., 2022). However, application of newly emerged design and implementation choices to this baseline is still missing.

In this work, we build upon Fujimoto & Gu (2021) line of research and ask: what is the extent to which newly emerged minor design choices can advance the minimalistic offline RL algorithm? The answer is illustrated in Figure 1: we propose an extension to TD3+BC, ReBRAC (Section 3), that simply adds on recently appeared design decisions upon it. We test our algorithm on both proprioceptive and

ReBRAC

(CQL)

Larger batches

and LayerNorm

(LB-

SAC)

Adjusted

discount factor

(SPOT)

Actor and critic

penalty

(TD3-

CVAE)

MSE and

Q normalization

(TD3+BC)

(a)

0

25

50

75

100

125

150

D4RL Normalized Score (

)

0.00

0.25

0.50

0.75

1.00

Fraction of runs with score  >

ReBRAC

TD3 + BC

IQL

SAC-RND

(b)

0.5

0.6

0.7

0.8

P(X > Y)

ReBRAC

ReBRAC

ReBRAC

IQL

SAC-RND

TD3+BC

(c)

visual state space problems using D4RL (Fu et al., 2020) and V-D4RL (Lu et al., 2022) benchmarks (Section 4) demonstrating its state-of-the-art performance across ensemble-free methods. Moreover, our approach demonstrates state-of-the-art performance in offline-to-offline setup on D4RL datasets (Section 4.3) while not being specifically designed for this setup. To further highlight the efficacy of the proposed modifications, we then conduct a large-scale ablation study (Section 4.4). We hope the described approach can serve as a strong baseline under different hyperparameter search budgets (Section 4.6), further accentuating the importance of seemingly minor design choices introduced along with core algorithmic innovations.

2 Preliminaries

2.1 Offline Reinforcement Learning

Astandard Reinforcement Learning problem is defined as a Markov Decision Process (MDP) with the tuple { S, A, P, R, γ } , where S ⊂ R n is the state space, A ⊂ R m is the action space, P : S × A → S is the transition function, R : S × A → R is the reward function, and γ ∈ (0 , 1) is the discount factor. The ultimate objective is to find a policy π ( a | s ) that maximizes the cumulative discounted return E π ∑ ∞ t =0 γ t R ( s t , a t ) . This policy improves by interacting with the environment, observing states, and taking actions that provide rewards.

In offline RL, policies cannot interact with the environment and can only access a static transaction dataset D collected by one or more other policies. This setting presents new challenges, such as estimating values for state-action pairs not included in the dataset while exploration is unavailable (Levine et al., 2020).

2.2 Behavior Regularized Actor-Critic

Behavior Regularized Actor-Critic (BRAC) is an offline RL framework introduced in Wu et al. (2019). The core idea behind BRAC is that actor-critic algorithms can be penalized in two ways to solve offline RL tasks: actor penalization and critic penalization. In this framework, the actor objective is represented as in Equation (1), and the critic objective as in Equation (2), where F is a divergence function between dataset actions and policy actions distributions. The differences from a vanilla actor-critic are highlighted in blue.

In the original work, various choices of F were evaluated when used as the regularization term for the actor or critic. The authors tested KL divergence, Kernel MMD, and Wasserstein distance but did not observe any consistent advantage. Finally, it is essential to note that, originally, both regularizations coefficients had the same weight.

Subsequently, TD3+BC (Fujimoto & Gu, 2021) was introduced, utilizing Mean Squared Error (MSE) as the regularization term F for the actor. TD3+BC is considered to be the minimalist approach to offline RL as it modifies existing RL algorithms by simply adding behavior cloning term into actor loss which is easy to implement and does not bring any significant computational overhead.

3 ReBRAC: Distilling Key Design Choices

In this section, we describe the proposed method along with the discussion of the new design choices met in the offline RL literature (Table 1). Our approach is a more general version of BRAC built on top of the TD3+BC (Fujimoto & Gu, 2021) algorithm with different modifications in design while keeping it simple (Figure 1). We refer to our method as Re visited BRAC (ReBRAC).

Deeper Networks The use of deeper neural networks has been a critical factor in the success of many Deep Learning models, with model quality generally increasing as depth increases, provided there is enough data to support this scaling (Kaplan et al., 2020). Similarly, recent studies in RL (Neumann & Gros, 2022; Sinha et al., 2020) and offline RL (Lee et al., 2022; Kumar et al., 2022) have demonstrated the importance of depth in achieving high performance. Although most offline RL algorithms are based on SAC (Haarnoja et al., 2018) or TD3 (Fujimoto et al., 2018), which by default employ two hidden layers, recent work (Kumar et al., 2020) uses three hidden layers for SAC instead, which appears to be an important change (Fujimoto & Gu, 2021). The change in network size has been adopted by later works (An et al., 2021; Yang et al., 2022; Zhuang et al., 2023; Nikulin et al., 2023) and may be one of the critical modifications that improve final performance.

The original BRAC and TD3+BC algorithms used only two hidden layers for their actor and critic networks, while most state-of-the-art solutions use deeper networks. Specifically, three hidden layers have become a common choice for recent offline RL algorithms. In ReBRAC, we follow this trend and use three hidden layers for the actor and critic networks. Additionally, we provide an ablation study in Section 4.5 to investigate the effect of the number of layers on ReBRAC's performance.

LayerNorm LayerNorm (Ba et al., 2016) is a widely used technique in deep learning that helps improve network convergence. In Hiraoka et al. (2021), authors add dropout and LayerNorm to different RL algorithms, notably boosting their performance. This technique is also applied in Smith et al. (2022), and it appears that boost is achieved primarily because of LayerNorm. Specifically, in offline RL, various studies have tested the effect of normalizations between layers (Bhatt et al., 2019; Kumar et al., 2022; Nikulin et al., 2022, 2023). A parallel study by Ball et al. (2023) empirically shows that LayerNorm helps to prevent catastrophic value extrapolation for the Q function when using offline datasets in online RL. Following these works, in our approach, we also apply LayerNorm between each layer of the critic networks.

Larger Batches Another technique to accelerate neural network convergence is large batch optimization (You et al., 2017, 2019). While studying batch sizes larger than 256 is limited, some prior works have used them. For instance, the convergence of SAC-N was accelerated in Nikulin et al. (2022). More recently proposed algorithms also use larger batches for training, although without providing detailed analyses (Akimov et al., 2022; Nikulin et al., 2023).

The usage of large batches in offline RL is still understudied, and its benefits and limitations are not fully understood. Our experiments show that in some domains, using large batches can lead to significant performance improvements, while in others, it might not have a notable impact or

even drop the performance (see Table 8). We increased the batch size to 1024 samples and scaled the learning rate for D4RL Gym-MuJoCo tasks, following the approach proposed by Nikulin et al. (2022).

Actor and critic penalty decoupling The original BRAC framework proposed penalizing the actor and the critic with the same magnitude. Most of the previous algorithms restrict only actor (Fujimoto & Gu, 2021; Wu et al., 2022) or only critic (Kumar et al., 2020; An et al., 2021; Ghasemipour et al., 2022). TD3-CVAE (Rezaeifar et al., 2022) penalizes both using the same coefficients, while another study, SAC-RND (Nikulin et al., 2023), shows that decoupling the penalization in offline RL is beneficial for algorithm performance, although ablations on using only one of the penalties are missing.

Our method allows simultaneous penalization of actor and critic with decoupled parameters. Inspired by TD3+BC (Fujimoto & Gu, 2021), Mean Squared Error (MSE) is used as a divergence function F , which we found simple and effective. The actor objective is shown in Equation (3), and the critic objective is shown in Equation (4). Differences from the original actor-critic are highlighted in red. Following TD3+BC (Fujimoto & Gu, 2021), the Q function is normalized to make the algorithm less sensitive to regularization parameters. Nonetheless, we forego the utilization of state normalization, as initially suggested in TD3 + BC, driven by our intention to execute the algorithm online and the observation that this adjustment typically results in negligible impact. Since our approach principally builds upon TD3+BC, the differences in their performances should be considered the most important ones and do not appear only because of the additional hyperparameters search.

Discount factor γ value change The choice of discount factor is an important aspect in solving RL problems (Jiang et al., 2015). A recent study (Hu et al., 2022) suggests that decreasing the default value of γ from 0 . 99 may lead to better results in offline RL settings. In contrast, in SPOT (Wu et al., 2022), the authors increased the value of γ up to 0.995 when fine-tuning AntMaze tasks with sparse rewards, which resulted in state-of-the-art solutions. Similarly, in the offline setting SAC-RND (Nikulin et al., 2023), increasing γ also achieved high performance on the same domain. The choice of increased γ for AntMaze tasks was motivated by the sparse reward, i.e., a low γ value may not propagate the training signal well. However, further ablations are needed to understand if the change in the parameter is directly responsible for the improved performance. In our experiments, we also find that increasing γ from the default value of 0 . 99 to 0 . 999 is vital for improved performance on this set of tasks (see Table 8).

4 Experiments

4.1 Evaluation on offline D4RL

We evaluate the proposed approach on three sets of D4RL tasks: Gym-MuJoCo, AntMaze, and Adroit. For each domain, we consider all of the available datasets. We compare our results to several ensemble-free baselines, including TD3+BC (Fujimoto & Gu, 2021), IQL (Kostrikov et al., 2021), CQL (Kumar et al., 2020) and SAC-RND (Nikulin et al., 2023).

The majority of the hyperparameters are adopted from TD3+BC, while β 1 and β 2 parameters from Equation (3) and Equation (4) are tuned. We examine the sensitivity to these parameters in Section 4.6. For a complete overview of the experimental setup and details, see Appendix A.

Following Wu et al. (2022), we tune hyperparameters over four seeds (referred to as training seeds) and evaluate the best parameters over ten new seeds (referred to as unseen training seeds), reporting the average performance of the last checkpoints for D4RL tasks. On V-D4RL, we use two and five seeds, respectively. This helps to avoid overfitting during hyperparameters search and outputs more just and reproducible results. For a fair comparison, we tune TD3+BC and IQL following the same protocol. We also rerun SAC-RND on Gym-MuJoCo and AntMaze tasks and tune it for the Adroit

domain. In other cases, we report results from previous works, meaning that scores for other methods can be lower if evaluated under our protocol.

The results of our tests on D4RL's Gym-MuJoCo, AntMaze, and Adroit tasks are available in Table 2, Table 3, Table 4, respectively. The mean-wise best results among algorithms are highlighted with bold , and the second best performance is underlined. Our approach, ReBRAC, achieves state-of-the-art results on Gym-MuJoCo, AntMaze, and Adroit tasks outperforming all baselines on average, except SAC-RND on Gym-MuJoCo tasks, which is slightly better. Performance profiles and probability of improvement (Agarwal et al., 2021) in Figure 1b and Figure 1c also demonstrate that ReBRAC is competitive when compared to the algorithms that we contrast against. Our method is also comparable to ensemble-based approaches (see Appendix C for additional comparisons).

4.2 Evaluation on offline V-D4RL

In addition to testing ReBRAC on D4RL, we evaluated its performance on V-D4RL benchmark (Lu et al., 2022). Our motivation for doing so was the fact that scores on D4RL Gym-MuJoCo tasks have saturated in recent years, and even after the introduction of ensemble-based offline RL methods

by An et al. (2021), there has been no notable progress on these tasks. On the other hand, V-D4RL provides a similar set of problems, with datasets collected in the same way as in D4RL but with the agent's observations now being images from the environment.

We tested our algorithm on all available single-task datasets without distractors and compared it to the baselines from the original V-D4RL work (Lu et al., 2022). The results are reported in Table 5. Our proposed approach achieves state-of-the-art or close-to-state-of-the-art results on most of the tasks, and it is the only method that, on average, performs notably better than naive Behavioral Cloning.

4.3 Evaluation on offline-to-online D4RL

The evaluation of offline-to-online performance is a pivotal aspect for reinforcement learning (RL) algorithms, particularly in light of recent developments. In this context, we conducted additional tests on ReBRAC, as it stands out as a promising algorithm for several compelling reasons.

First and foremost, ReBRAC demonstrates a remarkable proficiency following offline pre-training. Secondly, our algorithm shares notable similarities with TD3+BC, a method that has exhibited effectiveness in online fine-tuning as observed by Beeson et al. (Beeson & Montana, 2022)

For the sake of simplicity, we opted to disable critic penalization during the online fine-tuning. Furthermore, we linearly decay the actor's penalty to half of its initial value, following the approach described by Beeson & Montana (2022). Notably, no hyperparameter tuning was performed in this process.

To evaluate our approach in the offline-to-online setting, we followed the methodology outlined by Tarasov et al. (2022). In our comparative analysis, we consider the following algorithms: TD3+BC (Fujimoto & Gu, 2021), IQL (Kostrikov et al., 2021), CQL (Kumar et al., 2020), SPOT (Wu et al., 2022), and Cal-CQL (Nakamoto et al., 2023). The scores after the offline stage and online tuning, are reported in Table 6. We also provide finetuning cumulative regret proposed by Nakamoto et al. (2023) in Table 7.

ReBRAC exhibits competitive performance, surpassing four out of six AntMaze datasets and achieving state-of-the-art results in terms of final scores on Adroit tasks. On average, ReBRAC outperforms its closest competitor, Cal-CQL, which was specifically designed for the offline-to-online problem in the concurrent work (Nakamoto et al., 2023).

Regarding regret, ReBRAC outperforms all other algorithms, with Cal-QL being the sole exception, showing slightly better results on average only in the AntMaze domain.

4.4 Ablating Design Choices

To better understand the source of improved performance, we conducted an ablation study on the modifications made to the algorithm. Results can be found in Table 8. Additional ablation studies

for all datasets can be found in Appendix G. One modification at a time was disabled, while all other changes were retained, including layer normalization in the critic network, additional linear layers in the actor and critic networks, adding an MSE penalty to the critic and actor loss. In the case of AntMaze, we also attempted to use the default γ value instead of the increased one. To further demonstrate the efficacy of our modifications, we also ran our implementation as equivalent to the original TD3+BC, with all changes disabled and hyperparameters were taken from the original paper. This experiment serves to show that the improved scores are due to the proposed changes in the algorithm and not just different implementations. Furthermore, we searched for the regularization parameter for our implementation of TD3+BC to demonstrate that tuning this parameter is not the sole source of improvement. Moreover, we tested the TD3 + BC by adding each of the ReBRAC's modifications independently to demonstrate that each individual modifications is not sufficient for achieving performance of ReBRAC where the combination of modification appear.

Additionally, we run ablation which to validate the importance of decoupling by disabling it and searching the best penalty parameter value from the set of all previously used values listed in the Appendix B.

Table 8: ReBRAC's design choices ablations: each modification was disabled while keeping all the others. For brevity, we report the mean of average normalized scores over four unseen training seeds across domains. We also include tuned results for TD3+BC to highlight that the improvement does not come from hyperparameter search. For dataset-specific results, please refer to Appendix G.

The ablation results show that ReBRAC outperforms TD3+BC not because of different implementations or actor regularization parameter choice. All domains suffer when the LayerNorm is disabled, leading to the halved average performance overall. Removing additional layers leads to a notable drop in AntMaze tasks, while on Gym-MuJoCo decrease is small, and on Adroit tasks, we can see a slight boost. The algorithm fails to learn most tasks when the actor penalty is disabled. Notably, the critic penalty plays a minor role in improving the performance on most of the problems as well as the decoupling penalties. Using standard batch size on Gym-MuJoCo tasks significantly decreases final scores, while using the increased discount factor for AntMaze is crucial for obtaining state-of-the-art performance.

Based on the conducted ablations studies, the proposed configuration of design choices leads to the best performance on average. Note that tuning these choices for each task independently makes it possible to get even higher scores than we report in Section 4.1. We also can change the number of hidden layers in the networks, leading to better performance, see Section 4.5. But in real life, algorithm evaluation might be costly, so we limit ourselves to tuning only regularization parameters.

4.5 Stacking Even More Layers

As ablations show, the depth of the network plays an important role when solving AntMaze and HalfCheetah tasks (see Appendix G). We conduct additional experiments to check how the perfor-

mance depends on the network depth in more detail. Ball et al. (2023) used networks of depth four when solving AntMaze tasks. Our goal is to find the point where the performance saturates. For this, we run our algorithm on AntMaze tasks increasing the number of layers to six while keeping other parameters unchanged. We attempt to increase actor and critic separately and at once. We also decreased each network's size by one layer similarly. Results can be found in Figure 2.

Several conclusions can be drawn from the results. First, further increments in the number of layers can lead to better results on the AntMaze domain when layers are scaled up to five. For six layers, performance drops or does not improve. Second, decreasing the critic's size leads to the worst performance on most datasets. Lastly, there is no clear pattern on how the performance changes even within a single domain. The drop on six layers is the only common feature that can be seen. On average, four critic layers and three actor layers were the best. Changing only the actor's network is more stable on average.

2

3

(ReBRAC)

4

5

6

0

100

umaze

2

3

(ReBRAC)

4

5

6

0

50

medium-play

2

3

(ReBRAC)

4

5

6

0

50

large-play

2

3

(ReBRAC)

4

5

6

50

100

umaze-diverse

2

3

(ReBRAC)

4

5

6

0

50

medium-diverse

2

3

(ReBRAC)

4

5

6

0

50

100

large-diverse

2

3

(ReBRAC)

4

5

6

25

50

75

Average

Number of hidden layers

Normalized score

Critic scale

Actor and Critic scale

Actor scale

layers for certain tasks.

4.6 Penalization Sensitivity Analysis

Following Kurenkov & Kolesnikov (2022), we demonstrate the sensitivity of ReBRAC to the choice of β 1 and β 2 hyperparameters under uniform policy selection on D4RL tasks using Expected Online Performance (EOP) and comparing it to the TD3+BC and IQL. EOP shows the best performance expected depending on the number of policies that can be deployed for online evaluation. Results are demonstrated in Table 9. As one can see, approximately ten policies are required for ReBRAC to attain ensemble-free state-of-the-art performance. ReBRAC's EOP is higher for any number of online policies when compared to TD3+BC and better than IQL on all domains when the evaluation budget is larger than two policies. See Appendix F for EOP separated by tasks.

5 Related Work

Ensemble-free offline RL methods. In recent years, many offline reinforcement learning algorithms were developed. TD3+BC (Fujimoto & Gu, 2021) represents a minimalist approach to offline RL, which incorporates a Behavioral Cloning component into the actor loss, enabling online actor-critic algorithms to operate in an offline setting. CQL (Kumar et al., 2020) drives the critic network to assign lower values to out-of-distribution state-action pairs and higher values to in-distribution pairs. IQL (Kostrikov et al., 2021) proposes a method for learning a policy without sampling out-of-distribution actions.

Despite this, more sophisticated methods may be necessary to achieve state-of-the-art results in an ensemble-free setup. For instance, Chen et al. (2022b); Akimov et al. (2022) pre-train different forms of encoders for actions, then optimize the actor to predict actions in the latent space. SPOT (Wu et al., 2022) pre-trains Variational Autoencoder and uses its uncertainty to penalize actor for sampling OOD actions while SAC-RND (Nikulin et al., 2023) applies Random Network Distillation and penalizes actor and critic.

Ensemble-based offline RL methods. A significant number of works in offline reinforcement learning have also leveraged ensemble methods for uncertainty estimation. The recently introduced SAC-N (An et al., 2021) algorithm outperformed all previous approaches on D4RL Gym-MuJoCo tasks; however, it necessitated large ensembles for some tasks, such as the hopper task, which required an ensemble size of 500 and imposed a significant computational burden. To mitigate this, the EDAC algorithm was introduced in the same work, which utilized ensemble diversification to reduce the ensemble size from 500 to 50. Despite the reduction, the ensemble size remains substantial compared to ensemble-free alternatives. It is worth mentioning that neither SAC-N nor EDAC is capable of solving the complex AntMaze tasks (Tarasov et al., 2022).

Another state-of-the-art algorithm in the Gym-MuJoCo tasks is RORL (Yang et al., 2022), which is a modification of SAC-N that makes the Q function more robust and smooth by perturbing state-action pairs with the use of out-of-distribution actions. RORL also requires an ensemble size of up to 20. On the other hand, MSG (Ghasemipour et al., 2022) utilizes independent targets for each ensemble member and achieves good performance on the Gym-MuJoCo tasks with an ensemble size of four but requires 64 ensemble members to achieve state-of-the-art performance on the AntMaze tasks.

Design choices ablations. Ablations of different design choices on established baselines are very limited, especially in offline RL. It is only shown by Fujimoto & Gu (2021) that CQL performs poorly if proposed non-algorithmic differences are eliminated. A parallel study (Ball et al., 2023) shows that some of the considered modifications (LayerNorm and networks depth) are important when used in online RL with offline data setting, which is different from pure offline.

6 Conlusion, Limitations, and Future work

In this work, we revisit recent advancements in the offline RL field over the last two years and incorporate a modest set of improvements to a previously established minimalistic TD3+BC baseline. Our experiments demonstrate that despite these limited updates, we can achieve more than competitive results on offline and offline-to-online D4RL and offline V-D4RL benchmarks under different hyperparameter budgets.

Despite the noteworthy results, our work is limited to one approach and a subset of possible design changes. It is imperative to explore the potential impact of these modifications on other offline RL methods (e.g., IQL, CQL, MSG) and to investigate other design choices used in offline RL, e.g., learning rate schedules, dropout (like in IQL), wider networks, or selection between stochastic and deterministic policies.

References

Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in Neural Information Processing Systems , 2021.

Dmitriy Akimov, Vladislav Kurenkov, Alexander Nikulin, Denis Tarasov, and Sergey Kolesnikov. Let offline rl flow: Training conservative agents in the latent space of normalizing flows. arXiv preprint arXiv:2211.11096 , 2022.

Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-based offline reinforcement learning with diversified q-ensemble. Advances in neural information processing systems , 34:7436-7447, 2021.

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 , 2016.

Philip J Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learning with offline data. arXiv preprint arXiv:2302.02948 , 2023.

Alex Beeson and Giovanni Montana. Improving td3-bc: Relaxed policy constraint for offline learning and stable online fine-tuning. arXiv preprint arXiv:2211.11802 , 2022.

Aditya Bhatt, Max Argus, Artemij Amiranashvili, and Thomas Brox. Crossnorm: Normalization for off-policy td reinforcement learning. arXiv preprint arXiv:1902.05605 , 2019.

Xi Chen, Ali Ghadirzadeh, Tianhe Yu, Yuan Gao, Jianhao Wang, Wenzhe Li, Bin Liang, Chelsea Finn, and Chongjie Zhang. Latent-variable advantage-weighted policy optimization for offline rl. arXiv preprint arXiv:2203.08949 , 2022a.

Xi Chen, Ali Ghadirzadeh, Tianhe Yu, Jianhao Wang, Alex Yuan Gao, Wenzhe Li, Liang Bin, Chelsea Finn, and Chongjie Zhang. Lapo: Latent-variable advantage-weighted policy optimization for offline reinforcement learning. Advances in Neural Information Processing Systems , 35: 36902-36913, 2022b.

Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep policy gradients: A case study on ppo and trpo. arXiv preprint arXiv:2005.12729 , 2020.

Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 , 2020.

Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems , 34:20132-20145, 2021.

Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actorcritic methods. In International conference on machine learning , pp. 1587-1596. PMLR, 2018.

Seyed Kamyar Seyed Ghasemipour, Shixiang Shane Gu, and Ofir Nachum. Why so pessimistic? estimating uncertainties for offline rl through ensembles, and why their independence matters. arXiv preprint arXiv:2205.13703 , 2022.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning , pp. 1861-1870. PMLR, 2018.

Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Proceedings of the AAAI conference on artificial intelligence , volume 32, 2018.

Takuya Hiraoka, Takahisa Imagawa, Taisei Hashimoto, Takashi Onishi, and Yoshimasa Tsuruoka. Dropout q-functions for doubly efficient reinforcement learning. arXiv preprint arXiv:2110.02034 , 2021.

Hao Hu, Yiqin Yang, Qianchuan Zhao, and Chongjie Zhang. On the role of discount factor in offline reinforcement learning. In International Conference on Machine Learning , pp. 9072-9098. PMLR, 2022.

Nan Jiang, Alex Kulesza, Satinder Singh, and Richard Lewis. The dependence of effective planning horizon on model accuracy. In Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems , pp. 1181-1189, 2015.

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 , 2020.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169 , 2021.

Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems , 33:1179-1191, 2020.

Aviral Kumar, Rishabh Agarwal, Xinyang Geng, George Tucker, and Sergey Levine. Offline qlearning on diverse multi-task data both scales and generalizes. arXiv preprint arXiv:2211.15144 , 2022.

Vladislav Kurenkov and Sergey Kolesnikov. Showing your offline reinforcement learning work: Online evaluation budget matters. In International Conference on Machine Learning , pp. 1172911752. PMLR, 2022.

Kuang-Huei Lee, Ofir Nachum, Mengjiao Yang, Lisa Lee, Daniel Freeman, Winnie Xu, Sergio Guadarrama, Ian Fischer, Eric Jang, Henryk Michalewski, et al. Multi-game decision transformers. arXiv preprint arXiv:2205.15241 , 2022.

Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 , 2020.

Cong Lu, Philip J Ball, Tim GJ Rudner, Jack Parker-Holder, Michael A Osborne, and Yee Whye Teh. Challenges and opportunities in offline reinforcement learning from visual observations. arXiv preprint arXiv:2206.04779 , 2022.

Mitsuhiko Nakamoto, Yuexiang Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning. arXiv preprint arXiv:2303.05479 , 2023.

Oren Neumann and Claudius Gros. Scaling laws for a multi-agent reinforcement learning model. arXiv preprint arXiv:2210.00849 , 2022.

Alexander Nikulin, Vladislav Kurenkov, Denis Tarasov, Dmitry Akimov, and Sergey Kolesnikov. Q-ensemble for offline rl: Don't scale the ensemble, scale the batch size. arXiv preprint arXiv:2211.11092 , 2022.

Alexander Nikulin, Vladislav Kurenkov, Denis Tarasov, and Sergey Kolesnikov. Anti-exploration by random network distillation. arXiv preprint arXiv:2301.13616 , 2023.

Rafael Figueiredo Prudencio, Marcos ROA Maximo, and Esther Luna Colombini. A survey on offline reinforcement learning: Taxonomy, review, and open problems. arXiv preprint arXiv:2203.01387 , 2022.

Shideh Rezaeifar, Robert Dadashi, Nino Vieillard, Léonard Hussenot, Olivier Bachem, Olivier Pietquin, and Matthieu Geist. Offline reinforcement learning as anti-exploration. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pp. 8106-8114, 2022.

Samarth Sinha, Homanga Bharadhwaj, Aravind Srinivas, and Animesh Garg. D2rl: Deep dense architectures in reinforcement learning. arXiv preprint arXiv:2010.09163 , 2020.

Laura Smith, Ilya Kostrikov, and Sergey Levine. A walk in the park: Learning to walk in 20 minutes with model-free reinforcement learning. arXiv preprint arXiv:2208.07860 , 2022.

Denis Tarasov, Alexander Nikulin, Dmitry Akimov, Vladislav Kurenkov, and Sergey Kolesnikov. Corl: Research-oriented deep offline reinforcement learning library. arXiv preprint arXiv:2210.07105 , 2022.

Jialong Wu, Haixu Wu, Zihan Qiu, Jianmin Wang, and Mingsheng Long. Supported policy optimization for offline reinforcement learning. arXiv preprint arXiv:2202.06239 , 2022.

Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361 , 2019.

Rui Yang, Chenjia Bai, Xiaoteng Ma, Zhaoran Wang, Chongjie Zhang, and Lei Han. Rorl: Robust offline reinforcement learning via conservative smoothing. arXiv preprint arXiv:2206.02829 , 2022.

Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint arXiv:1708.03888 , 2017.

Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training bert in 76 minutes. arXiv preprint arXiv:1904.00962 , 2019.

Zifeng Zhuang, Kun LEI, Jinxin Liu, Donglin Wang, and Yilang Guo. Behavior proximal policy optimization. In International Conference on Learning Representations , 2023. URL https: //openreview.net/forum?id=3c13LptpIph .

A Experimental Details

In order to generate the results presented in Table 2 Table 3 and Table 4, we conducted a hyperparameter search and selected the best results from the final evaluations for each dataset. Our algorithm was implemented using JAX for the D4RL benchmark. For V-D4RL, we implement our approach using PyTorch adopting the TD3+BC implementation from Clean Offline RL (Tarasov et al., 2022). The experiments were conducted on V100 and A100 GPUs.

Gym-MuJoCo and Adroit tasks. Our study utilized the latest version of the datasets - v2 for Gym-MuJoCo and v1 for Adroit. The agents were trained for one million steps and evaluated over ten episodes.

For ReBRAC, we fine-tuned the β 1 parameter for the actor, which was selected from 0 . 001 , 0 . 01 , 0 . 05 , 0 . 1 . Similarly, the β 2 parameter for the critic was selected from a range of 0 , 0 . 001 , 0 . 01 , 0 . 1 , 0 . 5 . The selected best parameters for each dataset are reported in Table 11.

For TD3+BC here and in the AntMaze domain, we use the same grid used in ReBRAC for actor regularization parameter α and add the default value of 0 . 4 .

For IQL here and in the AntMaze domain, we selected β value from a range of 0 . 5 , 1 , 3 , 6 , 10 and IQL τ value from a range of 0 . 5 , 0 . 7 , 0 . 9 , 0 . 95 . We used the implementation from Clean Offline RL (Tarasov et al., 2022) and kept other parameters unchanged.

For SAC-RND in Adroit domain we tune β 1 (actor parameter) in the range of 0 . 5 , 1 . 0 , 2 . 5 , 5 . 0 , 10 . 0 and β 2 (critic parameter) in the range of 0 . 01 , 0 . 1 , 1 . 0 , 5 . 0 , 10 . 0 .

AntMaze tasks. In our work, we utilized v2 of the datasets. It's worth noting that previous studies have reported results using v0 datasets, which were found to contain numerous issues 2 . Each agent was trained for 1 million steps and evaluated over 100 episodes. Following Chen et al. (2022a), we modified the reward function by multiplying it by 100.

For ReBRAC, the β 1 (actor) and β 2 (critic) hyperparameters were carefully selected from the respective ranges of 0 . 0005 , 0 . 001 , 0 . 002 , 0 . 003 and 0 , 0 . 0001 , 0 . 0005 , 0 . 001 . In addition, the actor and critic learning rates were optimized from 0 . 0001 , 0 . 0002 , 0 . 0003 , 0 . 0005 and 0 . 0003 , 0 . 0005 , 0 . 001 , respectively. The optimal hyperparameters for each dataset are presented in Table 11.

We also modified the γ value for ReBRAC when addressing these tasks, driven by the following motivation. The length of the episodes in AntMaze can be as long as 1000 steps, while the reward is sparse and can only be obtained at the end of the episode. As a result, the discount for the reward with the default γ can be as low as 0 . 99 1000 = 4 · 10 -5 , which is extremely low for signal propagation, even when multiplying the reward by 100. By increasing γ to 0.999, the minimum discount value becomes 0 . 999 1000 = 0 . 36 , which is more favorable for signal propagation.

V-D4RL. We used single-task datasets without distraction with a resolution of 84 × 84 pixels. For ReBRAC β 1 (actor) parameter was selected from the range of { 0 . 03 , 0 . 1 , 0 . 3 , 1 . 0 } and β 2 (critic) parameter from the range of { 0 . 0 , 0 . 001 , 0 . 005 , 0 . 01 , 0 . 1 } .

Offline-to-offline. We used the same parameters for the offline-to-online setup with the only difference of setting β 2 to zero and lineary decaying β 1 to half of it's initial value during the online stage.

B Hyperparameters

B.1 ReBRAC

B.2 IQL

B.3 TD3+BC

B.4 SAC-RND

C Comparison to Ensemble-based Methods

Comparison of ReBRAC with the ensemble-based methods is presented in Table 16, Table 17, and Table 18. We add the following ensemble-based methods: RORL for each domain (Yang et al., 2022), SAC-N/EDAC (An et al., 2021) for the Gym-MuJoCo and Adroit tasks 3 and MSG (Ghasemipour et al., 2022) for AntMaze tasks 4 . The mean-wise best results among algorithms are highlighted with bold , and the second-best performance is underlined. Our approach, ReBRAC, shows competitive results on the Gym-MuJoCo datasets. On AntMaze tasks, ReBRAC achieves state-of-the-art results among ensemble-free algorithms and a good score compared to ensemble-based algorithms. And on Adroit tasks, our approach outperforms both families of algorithms.

D Feature Normalization

E Computational costs

F Expected Online Performance

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16 17 18 19 20

Policies evaluated online

5

10

15

20

25

30

Expected Online Performance

halfcheetah-random-v2

ReBRAC

TD3 + BC

IQL

(a) halfcheetah-random EOP.

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16 17 18 19 20

Policies evaluated online

45

50

55

60

65

Expected Online Performance

ReBRAC

TD3 + BC

IQL

(b) halfcheetah-medium EOP.

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16 17 18 19 20

Policies evaluated online

20

40

60

80

100

Expected Online Performance

halfcheetah-expert-v2

ReBRAC

TD3 + BC

IQL

(c) halfcheetah-expert EOP.

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16 17 18 19 20

Policies evaluated online

40

50

60

70

80

90

100

Expected Online Performance

ReBRAC

TD3 + BC

IQL

(d) halfcheetah-medium-expert

EOP.

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16 17 18 19 20

Policies evaluated online

20

25

30

35

40

45

50

Expected Online Performance

ReBRAC

TD3 + BC

IQL

(e) halfcheetah-medium-replay

EOP.

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16 17 18 19 20

Policies evaluated online

55

60

65

70

75

80

Expected Online Performance

halfcheetah-full-replay-v2

ReBRAC

TD3 + BC

IQL

(f) halfcheetah-full-replay EOP.

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16 17 18 19 20

Policies evaluated online

4

6

8

10

12

14

16

Expected Online Performance

hopper-random-v2

ReBRAC

TD3 + BC

IQL

(a) hopper-random EOP.

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16 17 18 19 20

Policies evaluated online

20

40

60

80

100

Expected Online Performance

hopper-medium-v2

ReBRAC

TD3 + BC

IQL

(b) hopper-medium EOP.

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16 17 18 19 20

Policies evaluated online

20

40

60

80

100

120

Expected Online Performance

hopper-expert-v2

ReBRAC

TD3 + BC

IQL

(c) hopper-expert EOP.

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16 17 18 19 20

Policies evaluated online

20

40

60

80

100

120

Expected Online Performance

hopper-medium-expert-v2

ReBRAC

TD3 + BC

IQL

(d) hopper-medium-expert EOP.

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16 17 18 19 20

Policies evaluated online

40

50

60

70

80

90

100

110

Expected Online Performance

hopper-medium-replay-v2

ReBRAC

TD3 + BC

IQL

(e) hopper-medium-replay EOP.

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16 17 18 19 20

Policies evaluated online

40

50

60

70

80

90

100

110

Expected Online Performance

hopper-full-replay-v2

ReBRAC

TD3 + BC

IQL

(f) hopper-full-replay EOP.

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16 17 18 19 20

Policies evaluated online

5

10

15

20

Expected Online Performance

walker2d-random-v2

ReBRAC

TD3 + BC

IQL

(a) walker2d-random EOP.

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16 17 18 19 20

Policies evaluated online

20

40

60

80

100

Expected Online Performance

walker2d-medium-v2

ReBRAC

TD3 + BC

IQL

(b) walker2d-medium EOP.

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16 17 18 19 20

Policies evaluated online

20

40

60

80

100

120

Expected Online Performance

walker2d-expert-v2

ReBRAC

TD3 + BC

IQL

(c) walker2d-expert EOP.

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16 17 18 19 20

Policies evaluated online

20

40

60

80

100

120

Expected Online Performance

walker2d-medium-expert-v2

ReBRAC

TD3 + BC

IQL

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16 17 18 19 20

Policies evaluated online

10

20

30

40

50

60

70

80

90

Expected Online Performance

walker2d-medium-replay-v2

ReBRAC

TD3 + BC

IQL

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16 17 18 19 20

Policies evaluated online

40

50

60

70

80

90

100

110

Expected Online Performance

walker2d-full-replay-v2

ReBRAC

TD3 + BC

IQL

(d) walker2d-medium-expert EOP.

(e) walker2d-medium-replay EOP.

(f) walker2d-full-replay EOP.

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16 17 18 19 20

Policies evaluated online

0

20

40

60

80

100

Expected Online Performance

antmaze-umaze-v2

ReBRAC

TD3 + BC

IQL

(a) antmaze-umaze EOP.

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16 17 18 19 20

Policies evaluated online

0

20

40

60

80

Expected Online Performance

ReBRAC

TD3 + BC

IQL

(b) antmaze-medium-play EOP.

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16 17 18 19 20

Policies evaluated online

0

20

40

60

80

Expected Online Performance

antmaze-large-play-v2

ReBRAC

TD3 + BC

IQL

(c) antmaze-large-play EOP.

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16 17 18 19 20

Policies evaluated online

0

20

40

60

80

Expected Online Performance

antmaze-umaze-diverse-v2

ReBRAC

TD3 + BC

IQL

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16 17 18 19 20

Policies evaluated online

0

20

40

60

80

100

Expected Online Performance

ReBRAC

TD3 + BC

IQL

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16 17 18 19 20

Policies evaluated online

0

20

40

60

80

Expected Online Performance

antmaze-large-diverse-v2

ReBRAC

TD3 + BC

IQL

(d) antmaze-umaze-diverse EOP.

(e) antmaze-medium-diverse EOP.

(f) antmaze-large-diverse EOP.

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16 17 18 19 20

Policies evaluated online

20

40

60

80

100

Expected Online Performance

pen-human-v1

ReBRAC

TD3 + BC

IQL

(a) pen-human EOP.

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16 17 18 19 20

Policies evaluated online

20

40

60

80

100

Expected Online Performance

pen-cloned-v1

ReBRAC

TD3 + BC

IQL

(b) pen-cloned EOP.

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16 17 18 19 20

Policies evaluated online

20

40

60

80

100

120

140

160

Expected Online Performance

pen-expert-v1

ReBRAC

TD3 + BC

IQL

(c) pen-expert EOP.

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16 17 18 19 20

Policies evaluated online

0

1

2

3

4

5

6

7

Expected Online Performance

door-human-v1

ReBRAC

TD3 + BC

IQL

(a) door-human EOP.

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16 17 18 19 20

Policies evaluated online

0

1

2

3

4

5

Expected Online Performance

door-cloned-v1

ReBRAC

TD3 + BC

IQL

(b) door-cloned EOP.

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16 17 18 19 20

Policies evaluated online

0

20

40

60

80

100

120

Expected Online Performance

door-expert-v1

ReBRAC

TD3 + BC

IQL

(c) door-expert EOP.

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16 17 18 19 20

Policies evaluated online

0.0

0.5

1.0

1.5

2.0

2.5

Expected Online Performance

hammer-human-v1

ReBRAC

TD3 + BC

IQL

(a) hammer-human EOP.

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16 17 18 19 20

Policies evaluated online

0

5

10

15

20

Expected Online Performance

hammer-cloned-v1

ReBRAC

TD3 + BC

IQL

(b) hammer-cloned EOP.

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16 17 18 19 20

Policies evaluated online

0

20

40

60

80

100

120

140

Expected Online Performance

hammer-expert-v1

ReBRAC

TD3 + BC

IQL

(c) hammer-expert EOP.

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16 17 18 19 20

Policies evaluated online

0.4

0.2

0.0

0.2

0.4

0.6

Expected Online Performance

relocate-human-v1

ReBRAC

TD3 + BC

IQL

(a) relocate-human EOP.

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16 17 18 19 20

Policies evaluated online

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Expected Online Performance

relocate-cloned-v1

ReBRAC

TD3 + BC

IQL

(b) relocate-cloned EOP.

1

2

3

4

5

6

7

8

9

10 11 12 13 14 15 16 17 18 19 20

Policies evaluated online

20

0

20

40

60

80

100

120

Expected Online Performance

relocate-expert-v1

ReBRAC

TD3 + BC

IQL

(c) relocate-expert EOP.

G D4RL tasks ablation

Peer reviews:
comments: SUMMARY:
This paper proposes an offline RL algorithm based on TD3+BC and BRAC, integrating popular design elements. The proposed method achieves higher scores on D4RL and V-D4RL datasets.

STRENGTHS:
The idea of exploring the popular design elements in offline RL algorithms is interesting. The authors make a great effort in conducting confirmatory experiments including various ablation studies.

WEAKNESSES:
Overall, I see the proposed method as an integrated method of existing algorithms. From my perspective, the novelty and contributions are weak. It may be more insightful if the author put more emphasis on the analyses of existing design elements rather than combining them into one new method. I do not see many insightful analyses in the current manuscript.

QUESTIONS:
1.	Table 6: ReBRAC w large batch. What does it mean?
2.	The performance improvements of ReBRAC seem marginal, especially in challenging tasks (door-human, door-cloned….)
3.	Table 6: it seems that the critic penalty is not crucial. Is it a universal conclusion, or just due to a lack of hyperparameter tuning?
4.	Can authors provide comparisons of computation costs?

LIMITATIONS:
Please see the weaknesses.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-25

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper revisited recent methods in the offline RL area and discussed how different design choice impact offline RL methods' performance. In particular, the authors focus on four hyper-parameter choices (i) number of network layers, (ii) using LayerNorm, (iii) batch size and (iv) discounting factor $\\gamma$, as well as one algorithmic choice (v) using actor/critic regularization, and conducted comprehensive experiments to show the importance of different factors. Their method, ReBRAC, achieved strong empirical performance on 51 datasets.\n\nSTRENGTHS:\n1. The paper presents comprehensive experiments.\n\n2. The proposed method demonstrates strong empirical performance in the conducted experiments.\n\n3. The paper is clearly written.\n\n4. This work shows great engineering values and has the potential to offer valuable guidance in design choices for the offline RL community.\n\nWEAKNESSES:\n1. A combination of existing techniques/tricks. In particular, most of the design choices (i) number of network layers, (ii) using LayerNorm, (iii) batch size and (iv) tuning $\\gamma$ are hyper-parameters or standard deep NN techniques. \n\n2. Table 6: ReBRAC w/o LN has a 38.0 average score over all domains, which is noticeably lower than the authors' reproduction of TD3+BC (52.2), leading to my speculation that adding LN alone could achieve great empirical performance. It would be nice if TD3+BC w/ LN alone could be shown. (For reference, Rainbow [1] shows the performance of all individual choices in their Figure 1 and none of them were close to Rainbow.)\n\n3. Table 6: Critic penalty only gives marginal improvement. As the batch size, using LN, number of layers, and tuning $\\gamma$ are mostly hyper-parameters rather than RL algorithm designs, marginal benefits from the critic penalty further limit its algorithmic contribution.\n\n4. According to table 1, SAC-RND seems to have all five components. It would be nice if the authors could elaborate on the difference between SAC-RND and ReBRAC, for example, in section 2 or 3.\n\n\nOverall I find this paper has limited novelty, especially considering some components mentioned are merely hyper-parameters. However, I find it might be a significant contribution to achieve superior performance on 51 datasets, providing potential implementation guidance to the offline RL community. I therefore voted for a boarderline acceptance with a low confidence score.\n\n\n[1] Rainbow: Combining Improvements in Deep Reinforcement Learning. https://arxiv.org/pdf/1710.02298.pdf\n\nQUESTIONS:\nI have no further question as it was clearly written.\n\nLIMITATIONS:\nSee above.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper revisits several minor design choices in recent offline RL literature and equips a standard method TD3+BC (or more generally BRAC) with these designs to attain a strong baseline for offline RL with state-of-the-art performance on both D4RL and V-D4RL benchmarks. These critical designs include deeper networks, LayerNorm, larger batches, decoupled actor-critic penalty, and adjusted discount factor. Extensive benchmarking experiments and ablation studies are conducted across a range of domains.\n\nSTRENGTHS:\n1. Careful look into detailed design choices from a large amount of literature, which naturally motivates the proposed solution\n2. Extensive experiments across D4RL’s Gym-MuJoCo, AntMaze, and Adroit tasks, and even vision-based V-D4RL tasks\n3. Great efforts for a fair comparison with baseline methods, including hyperparameter tuning for baselines and measuring Expected Online Performance\n4. Well written\n\nOverall, I really appreciate the contribution to the offline RL community by revisiting a strong yet overlooked baseline method, BRAC.\n\nWEAKNESSES:\n1. A probably incorrect assertion to a technical detail of BRAC (see Question 1 below)\n2. Limited technical novelty (but I think it doesn't matters for this paper focusing on a retrospective analysis)\n\nQUESTIONS:\n1. If I understand correctly, BRAC has adopted actor penalization and critic penalization at the same time. Indeed, BRAC-v in the original paper adds a penalty to both actor and critic learning objectives (Eq. 6 and 7 in the BRAC paper). Thus, some assertions in this paper (Lines 65 and 111) are incorrect and should be revised. Nevertheless, the idea of decoupling actor and critic penalty has not been explored to my knowledge.\n2. Following the above question, an ablation study comparing decoupled actor-critic penalties and coupled ones (using only one hyperparameter) is missing in Table 6. Can authors present these results?\n3. What do you mean by \"Since our approach principally builds upon TD3+BC, the differences in their performances should be considered the most important ones.\" in line 124?\n4. In line 185, the authors claim, \"ReBRAC outperforms TD3+BC not because of different implementations or actor regularization parameter choice\". However, in my opinion, the differences between  ReBRAC and TD3+BC are indeed implementation choices rather than algorithmic innovations. I understand that the efforts of building ReBRAC upon TD3+BC are non-trivial, but the authors should consider revising this sentence to make it much clearer.\n5. Implementation details for vision-based V-D4RL tasks, including CNN architecture to encode visual observations, are missing in both the main paper and the appendix. Although I have found the implementations in supplementary source code, I recommend clarifying these details in a future revision of the paper.\n6. TD3+BC uses state normalization on Gym-MuJoCo tasks. Did the authors include this implementation detail in their experiments for both TD3+BC and ReBRAC? Since in the TD3+BC paper, state normalization does not impact significantly, in my opinion, it is okay not to include it, but I recommend the authors clarify this in the paper.\n\nLIMITATIONS:\nThis work has discussed its limitations and future work in the conclusion. There does not seem to be any negative social impact of this paper that should be discussed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, a number of recent advancements are added to the minimalist TD3+BC baseline, and the authors found that the resulting algorithm leads to a new SOTA performance on D4RL benchmark with raw state and visual input. Extensive empirical results and ablations are provided and the authors show a dedication to fairness and reliability of their comparisons.\n\nSTRENGTHS:\n**originality**\n- the paper studies existing methods, and the novelty of the work is main on its empirical findings and ablations. However, due to the extensive experiments and attention to fairness and details, these results can be considered novel findings. \n\n**quality**\n- quality is great, paper is well structured, important things are highlighted and performance changes are quantified. \n- technical details are given, things like computational speed discussed, code provided, overall good. \n- authors even tuned and tested on different seeds. \n\n**clarity**\n- paper is very clear and easy to read. \n\n**significance**\n- Finding a new sota baseline for offline RL setting, with a full set of ablations understanding the impact of each of its components can be a good and significant contribution. \n- Although no new algorithm is introduced, these results can be very helpful to researcher who want to really understand what is helping the performance (whether it's an algorithmic component or just a code hack or hyperpararmeter choice).\n\nWEAKNESSES:\n**originality**\n- the paper focuses on an empirical study of existing methods, which will reduce the novelty a bit. \n\nOther than this, I think within its own scope, the paper is very good. No major concern.\n\nQUESTIONS:\n- The authors mentioned that to be fair, the competing methods have been extensively tuned, I wonder how much performance gain you are able to achieved for these competing methods compared to their reported results in their original papers? \n- when you tune for d4rl, do you fine-tune a different set of hyperparamter for each task? Is this done for all algorithms in the comparisons?\n\nLIMITATIONS:\nAuthors discussed the limitations of the work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}