A Clean Slate for Offline Reinforcement Learning
Authors: Matthew Thomas Jackson
Uljad Berdica
Jarek Luca Liesen
Shimon Whiteson
Jakob Nicolaus Foerster
Venue: NeurIPS 2025 oral ()
Keywords: Offline Reinforcement Learning
Evaluation
Open-Source
Primary area: reinforcement_learning
TL;DR: We propose a principled taxonomy, evaluation procedure, and unified algorithm space for offline RL.

Abstract: Progress in offline reinforcement learning (RL) has been impeded by ambiguous problem definitions and entangled algorithmic designs, resulting in inconsistent implementations, insufficient ablations, and unfair evaluations. Although offline RL explicitly avoids environment interaction, prior methods frequently employ extensive, undocumented online evaluation for hyperparameter tuning, complicating method comparisons. Moreover, existing reference implementations differ significantly in boilerplate code, obscuring their core algorithmic contributions. We address these challenges by first introducing a rigorous taxonomy and a transparent evaluation protocol that explicitly quantifies online tuning budgets. To resolve opaque algorithmic design, we provide clean, minimalistic, single-file implementations of various model-free and model-based offline RL methods, significantly enhancing clarity and achieving substantial speed-ups. Leveraging these streamlined implementations, we propose Unifloral, a unified algorithm that encapsulates diverse prior approaches and enables development within a single, comprehensive hyperparameter space. Using Unifloral with our rigorous evaluation protocol, we develop two novel algorithms - TD3-AWR (model-free) and MoBRAC (model-based) - which substantially outperform established baselines. Our implementation is publicly available at https://github.com/EmptyJackson/unifloral.

A Clean Slate for Offline RL
Matthew T. Jackson
Uljad Berdica"
Jarek Liesen'
Shimon Whiteson
Jakob N. Foerster
University of Oxford
jaacksnn, jadb, jarek) ox ac.
Abstract
Progress in offline reinforcement learning (RL) has been impeded by ambiguous problem definitions and entangled algorithmic designs, resulting in inconsistent implementations, insufficient ablations, and unfair evaluations. Although offline RL explicitly avoids environment interaction, prior methods frequently employ extensive, undocumented online evaluation for hyperparameter tuning, compli- cating method comparisons. Moreover, existing reference implementations differ significantly boilerplate code, obscuring their core algorithmic contributions. We address these challenges by first introducing rigorous taxonomy and transparent evaluation procedure that explicitly quantifies online tuning budgets. To resolve opaque algorithmic design, we provide clean, minimalistic, single-file implemen- tations of various model-free and model-based offline RL methods, significantly enhancing clarity and achieving substantial speed-ups. Leveraging these stream- lined implementations, we propose Unifloral, a unified algorithm that encapsulates diverse prior approaches within single, comprehensive hyperparameter space, enabling algorithm development in shared hyperparameter space. Using Unifloral with our rigorous evaluation procedure, we develop two novel algorithms TD3- AWR (model-free and MoBRAC (model- (model-based) which substantially outperform established baselines. All code for this project can found our public codebase.
Introduction
Offline reinforcement learning (RL) the task of learning effective policies from pre-collected, static datasets-is critical for applying RL in real-world settings where online experimentation is expensive or risky. Despite significant interest [1-5], the field has struggled to converge on clear, actionable insights. Algorithms and methods proliferate rapidly but no broadly agreed-upon conclusions or standardized benchmarks have emerged [6]. This undermines both practical application and theoretical progress. In this work, identify and address primary problems that contribute to stagnation and confusion in offline RL research: an ambiguous problem setting and opaque algorithmic design.
Problem 1: Ambiguous Problem Setting Recent work in offline RL has lacked rigorously artic- ulated definition or standardized evaluation procedure. The broad mission statement, learning from a static dataset without direct environment interaction, is prone misinterpretation that skews pro- posed methods towards impractical evaluation practices. Existing literature implicitly relaxes various definitions concerning critical details such hyperparameter tuning allowances [4, 7], the extent of post-deployment policy adaptation [8], and the specifics of evaluation procedures [9]. Consequently. comparisons between methods are confounded each study might assume fundamentally different
'Equal contrihution.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

experimental conditions. While some approaches restrict tuning based on related dataset perfor- mance [10], most upproaches extensively tune hyperparameters on the target environment [11, 5, 12]. Using the target environment to tune hyperparameters needs a large number of online evaluations, which is in conflict with the basic premise of offline RI..
Solution A Novel Taxonomy and Evaluation Procedure We first introduce a rigorous and explicit taxonomy of offline RL evaluation variants (Section 3.1) and specify the find to be implicitly adopted by most prior research. To facilitate consistent and transparent evaluation, we propose rigorous procedure for this setting (Section 3.2) that evaluates algorithmic performance using a fixed hyperparameter range across multiple datasets. This procedure explicitly quantifies performance at various permissible levels of online hyperparameter tuning, i.e., interactions with the targer environment, thus providing clarity about the practical deployment requirements of each method. To ensure interface for performing this evaluation procedure, thereby empowering future work to evaluate offline RL algorithms robustly and transparently.
of adoption and reproducibility, release straightforward software
Problem 2: Opaque Algorithmic Design Offline RL methods are often presented as intricate bundles with intertwining algorithmic components, implemeetatioo--ppeiiie details, and unclear tuning procedures. Researchers compare proposed methods to baseline performance quoted directly from prior publications [6], inadvertently propagating these methodological issues. As result, it is difficult isolate the impact of individual methodological choices. Thus, the state-of-the-an remains ambiguous, with no method demonstrating uniformly strong performance across all datasets [13-15].
Solution 2: Consistent Reimplementations and Unified Algorithm We first dissect the novel components of prior algorithms by defining phylogenetic tree based on their compositional structure (Section 4.1). We use this representation to provide single-file file reimplementations of wide range of offline RL. methods. These minimal implementations eliminate extraneous code differences and highlight fundamental components, as well as achieving average training speedups of 131 131.5x and 74.8x against OfflinRLL-Kit [16] and CORL [17], two leading offline RL libraries. Furthermore, we propose a unified offline RL algorithm (Unifloral, Section 4.2) that integrates core components from various prior methods into one coherent framework. Crucially, Unifloral provides single, unified hyperparameter space containing all of these algorithms.
Leveraging Unifloral with our evaluation procedure, we introduce two novel offline RL methods: model-free approach (TD3-AWR, Section 5.1) and model-based one (MoBRAC, Section 5.2). These methods demonstrate substantial performance improvements over established baselines, vali- dating both our unified methodology and rigorous evaluation framework.
Offline Training
Pre-deployment
Post-deployment
Setting
1. Zero-shot
π
Example: Autonomous sourch and resoue
2a. Pre-deployment policy selection
π
XN
Exampie: Autonomous vehicier with sefety driver testing
D
2b. Post-deployment policy selection
Enample: Autonomuus seurch and
3. Pofline-to-onlimg
π
XN
Example: Mulii step language reasoning models
Figure 1: Formalizing the variants of offline RL- we define range of offline RL variants (Sec- tion 3.1), with policy performance being measured post-deployment. Pre-deployment policy selection (2a) and post-deployment policy selection (2b) use a policy-selection bandit after offline training, whilst (3) uses unrestricted policy updates.

---

2 Preliminaries
2.1 Reinforcement Learning
We apply RL to finite-horizon Markov Decision Process (MDP) defined by the tuple S.A. R, T). Here S is the state space, A is the action space, and is the horizon. D A + (S) is the transition dynamies, defining how the state changes given state and the action taken on that state. A (S) is the set of all possible distributions over s. The scalar reward function is R: S A RR. The environments in this paper all fully observable the Markov state is directly observed at each timestep.
A policy maps state in S to action distribution over A. The policy is trained to maximize the expected return JM for a given MDP M with trajectory length T':
JM
E
Σ 1=0
(I)
2.2 Offline Reinforcement Learning
Offline RL methods use pre-collected dataset to optimize target policy to maximize / by, without online interactions in the environment. This dataset consists of transitions (si, 0i, Ti, 8; 0++1) for i i= N, where 8i+1 E S, 0; E A,,;; r; E IR are the current and next states, action, and reward, respectively. Here, initial states are drawn from the distribution So Sy and trajectories are gathered through a behaviour policy interacting with the environment. Since may be suboptimal, the resulting dataset might not contain sufficient coverage of the environment's 's state space learn an effective policy.
An effective offline RL method must learn policies that generalize from this limited dataset to perform reliably when deployed their environment. Typically, these methods require significant regularization to avoid overestimation bias. For model-free methods, this is commonly done with critic ensembles, where the minimum state value estimated by the ensemble is used for policy optimization. Model-based methods generalize by training dynamics model D(s, a) to predict future states and rewards. This can be used to generate synthetic rollouts from the target policy, allowing for direct optimization of its performance.
3 Refining Evaluation in Offline RL
This section describes taxonomy of offline RL, illustrated in Figure 1, which motivates evaluation procedure in Figure 2. We also outline the procedure detail and use to analyze the performance of set of model-free and model-based algorithms multiple environments.
3.1 Variants of Offline RL
The goal of offline RL is to train an agent using solely offline data, with the objective of maximizing performance from deployment. i.e., the point where the agent is evaluated online. In this setting, deployment marks strict separation between the offline training phase and the online evaluation phase. However, some methods may relax this strict separation in two ways. Firstly, pre-deployment interaction allows the agent to take limited interactions with the environment before deployment improve post-deployment performance. For instance, to tune hyperparameters before selecting policy for deployment. Secondly, dest-deployment adaptation allows the agent to continue learning after deployment, and the performance metric includes all returns collected after deployment. Exam- ples include dataset aggregation from muitiple online episodes
[18], selection from a set of policies trained offline [7, 9], and fine- mee-tunnng single policy [8], all of which can be performed both before and after deployment. While any combination of these is possible, identify four key settings.

---

A Taxonomy of Offfine RL
ZERO-SHOT OFFLINE RL
Train one policy offline, then deploy online with no further adaptation.
No pre-deployment interaction, no post-deployment adaptation.
OFFLINE RL WITH PRE- DEN-OLLOLYETT ONLINE POLICY SELECTION
Train a set of policies offline, select the best policy based on N online evaluations hefore deployment.
Limited pre-deloymment interaction, popt-depl adaptation.
OFFLINE RL WITH POST- DEPLOMMET ONLINE POLICY SELECTION
Train set of policies offline, then deploy online, adaptively selecting a policy every episode
based on online performance.
No pre-deployment interaction, post-deployment adaptation via policy selection.
3. OFFLINE-TO--NLIN RL
Train one policy offline, then deploy online and fine-tune the policy on online data.
Limited pre-deployment interaction and post-deployment adaptation via finetuning.
Many offline RL papers implicitly perform pre-deployment policy selection (Setting 2a), as they report final performance after extensive hyperparameter tuning involving online evaluation [11, 5]. However, due to differences in the number of hyperparameters or computational resources, this tniing process varies in scope across studies. As result, reported performances directly comparable since they reflect not only algorithmic quality but also differences in tuning budgets. Furthermore, these procedures typically assume low-variance estimates of each policy's performance, determined by an indefinite number of online evaluations. This is rarely made explicit as hyperparameter tuning is often considered a technical detail and not part of the method, even though it can dramatically affect performance (Section 3.3).
often
Finally, much prior work has blurred the line between algorithms and hyperparameters in offline RI., proposing different hyperparame- ter values or ranges for each task. This am- biguity enables the same "method" to have dramatically different behaviour across tasks, undermining the assumption of limited interactions by essentially proposing different method for each task. To resolve this, we define an offline RL method to include a fixed hyperparameter range, which remains constant across datasets (see Definition of Offline RL Methods).
Definition of Offline RL Methods
A method in offline RL consists of algorithm and fixed sampling range for each hyperparameter.
3.2 Proposed Evaluation Procedure
We now propose rigorous and practical evaluation procedure for offline RL with pre-deployment policy selection (Setting 2a), as it implicitly is the standard setting for evaluating offline RL methods (see Section 3.1). Our goal is to evaluate offline RL algorithms under fixed budget of N pre- deployment environment interactions used for tuning. We measure this budget in terms of the number of evaluation episodes, reflecting practical deployment constraints where each online interaction can be costly. Whilst the tuning algorithm may be defined as part of the method, most research focuses on offline policy optimization prior tuning. Therefore, provide upper confidence bound (UCB) bandit [19] in our implementation as the default tuning algorithm.
Furthermore, to reflect worldworld limitations, not directly observable, with each pull from the bandit sampling a single episodic return from that policy's s return distribution. This models the high- figh-nariance, sample-limite setting typical in real deployments, where evaluating a policy's 's performance requires interacting with the environment and yields only noisy, episodic feedback. The importance of this is demonstrated by the emergence of distractor policies, as discussed in Section 3.3.
assume that the expected return of each policy is
In essence, our evaluation procedure repeatedly simulates hyperparameter tuning with fixed online budget, using bandit to select single policy for final deployment. This procedure (Figure 2) has two steps: score collection and bandit evaluation.

---

Training
Evaluation
Dataset
Sumple
Gel unline scores
Sample policies
Bandit on unline scures
Calculate Perforunce
Train and evaluate P policies for R episodes each
Bootstrap B times
Figure Overview of our evaluation procedure. Left: We sample hyperparameters, train the corresponding policies, and collect their final evaluation scores. Right: We simulate hyperparameter tuning using the collected scores by subsampling K policy scores and recording the best-arm performance of UCB tuning bandit operating over them.
Firstly, we collect dataset of episodic evaluation scores from policies trained by the target algorithm. To do this, we sample hyperparameter settings (with replacement and random seeds) from the range defined by the method, and then train P corresponding policies. These policies are evaluated online for large number of episodes R and their episodic scores recorded. Following this, the policies may be discarded as only their episodic scores are required for bandit evaluation.
Step 1: Train Policies and Collect Scores
Step 2: Run Bootstrapped Tuning Bandit Using collected episodic evaluation dataset, then repeatedly simulate hyperparameter tuning to measure algorithm performance at different tuning budgets. This is performed by subsampling K policies' (i.e., their corresponding episodic scores) and running a multi-armed bandit over them. In this bandit, each arm corresponds to policy, with each pull sampling one episode's return from the corresponding policy. At each number of pulls N, we evaluate the performance of the algorithm by selecting the policy estimated to have the highest return by the bandit, and taking its true average return. We repeat this process times to obtain bootstrapped estimate of algorithm performance.
Recommended Datasets It is essential evaluate methods diverse distribution of tasks to ensure generality. Alarmingly, the majority of offline RL methods considered in this work were evaluated only on MuloCo and Adroit tasks from the D4RL. suite [20]. While computational budgets may be limited, we argue that they would be better spent considering a wider range of tasks and behaviour policies. In order make environment selection consistent, recommend starting with the following environments, where algorithms currently obtain non-trivial performance: hopper- and
as a representative subset of MuJoCo locomotion; pen-human, pen-cloned, and pen-expert, as algorithms often achieve zero perfect performance on other Adroit environments; kitchen-mixed, maze2d-large, and antmaze-large-divers to provide diversity in the evaluated environments.
medium. serecheetaa-meiimm--e
3.3 Results
In Figure 3, we evaluate range of prior algorithms (list in Appendix A). For this, uniformly sample from the hyperparameter tuning ranges specified each algorithm's 's original paper or the union of ranges when multiple are provided. Generally, algorithm performs better if its curve is closer to the top left corner of plot, representing strong performance after few online interactions. Prior work has typically reported performance after unlimited online tuning, which is the limit of the score with an increasing number of policy evaluations, i.e., the top right corner.
Inconsistent Algorithm Performance No algorithm consistently performs well across all datasets. However, ReBRAC and IQL are competitive for the overall best performing algorithm, with ReBRAC out of 9 datasets and IQL on 4 out of
achieving top performance at some number of evaluations datasets. Even though both of these algorithms are worse than competing baselines on other datasets, believe them be the clearest baselines for future method development, done in Section 5.
We fix K 8 in experiments but encourage future evaluation under other values.
5

---

hopper-andium-22
bellcheelaahaaeddddueeeeeetee expertt
walkerr2-mmeebuueeeeeee teplay--
pea-buman-vl
80
60
80
40
40
20
Mean score
20-
10
pen-claced-vi
pen-expert-vi
mazeld largev1
kotchen mixed-v0
60
200
50
100
40
40
30
100
20
50
50
0
100 10
107
10
10a
10"
101
10a
10P
10
102
Number of policy evaluations
Algorithm
BC CQL
IQL
MoReL
SAC-N
TD3-BG
Figure 3: Evaluation of prior algorithms- mean and 95% CI over 500 bandit rollouts, with K = 8 policy arms subsampled from 20 trained policies each rollout. The x-axis denotes the number of bandit pulls, whilst the y-axis denotes the true expected score of the estimated best arm after x pulls. The full evaluation results in Appendix B.
Overfit Model-Based Methods The model-based algorithms we evaluate-MMOP,, MOReL, and COMBO (Appendix A.2) achieve notably poor performance all non-locomotion datasets, ranking no higher than 6th out of the evaluated algorithms (and failing to beat BC) at any number of policy evaluations. While these results surprising, emphasize that implementation successfully reproduces reference results with the specialized hyperparameters for each dataset (Appendix G). Instead, these results suggest that these methods are deeply overfit to the locomotion datasets they were originally evaluated on (Appendix C), providing a sobering reflection of the field.
Distractor Policy Phenomenon While perfor- mance typically improves as more bandit arms pulled, certain performance curves exhibit distinctive dems-rtepy decreases in measured performance despite additional policy evaluations. To better un- derstand this, we examine the ranked performance distribution of numerous ReBRAC policies trained on hopper-medee medium (Figure 4). This analysis reveals a notable cluster of policies that exhibit suboptimal av- erage performance formance compared to consistently better-performing policies. We refer to these anomalous policies as distractor policies.
Distractor policies can achieve higher scores
80
Scone
60
Man
A2 M mm
possess a higher maximum per-
20
0
20
60
100
120
To demonstrate their impact on evaluation, we sim- ulate the initial phase of a bandit rollout over these policies, i.c., when the bandit enumerates all arms (Figure 9a). Over this phase, we observe clear in- crease in the probability of preferring distractor policy, explaining the initial decrease in evaluation performance. This phenomenon runs counter to the expectation that increasing policy evaluations would monotonically reduce estimator variance and underscores the need to directly consider en- vironment interactions in evaluation, a crucial distinction from prior evaluation methodologies [9]. Further analysis of distractor policies is provided in Appendix D.
Policy index
Figure 4: Ranked ReBRAC performance- blue shaded area, solid and dashed lines rep- resenting the standard deviation, mean, min, and max episodic return, respectively.

---

Elucidating Algorithm Design in Offline RL
In this section, we seek to simplify algorithm design in offline RL. Firstly, we present genealogy of prior algorithms. using it to propose and implement set of compositional reimplementations. Following this, we propose a unified algorithm, Unifloral, capable expressing these methods as well as any combination of their components in single hyperparameter space.
4.1 Disentangling Prior Methods
New offline RL methods are typically derived from preceding ones by adding editing individual com- ponents of the agent's 's objective or architecture. De- spite this, methods typically suffer from a range of unnecessary implementation differences, making it difficult for researchers identify their contribution or fairly compare methods. Even in popular single- file implementations, we observe significant code differences between "parent" and "child" algorithms, which should require only the individual components to be edited. This encourages researchers to compare entire algorithms rather than ablating components. We discuss this and how it informs our code philos- ophy in Appendix E.
Unifloral Speedup Over Other Implementa Lions
OffizetLKit
CORL
16
Unitter all spondd fantan
toz.
CaL
SAC-t
Reattc
BDAC
Merd
cosmd
As solution, we provide single-file reimplementa- tions of range of existing model-free (BC, TD3-BC, ReBRAC, IQL. SAC-N, LB-SAC, EDAC, CQL, DT) and model-based (MOPO, MOReL, COMBO) meth- ods. Our implementation has a number advantages. Firstly, focus on code clarity and minimal code edits between algorithms, leading to a dramatic reduc- tion in code differences between algorithms. Secondly, we implement algorithms in end-to-end compiled JAX, leading to major speed-ups against competing implementations (Figure 5). We believe these implementations will lead to better algorithm understanding and fairer evaluation, as well as enabling powerful experiments on low compute budgets. We verify the correctness of our reimplementations in Appendix G.
Figure 5: Speed up from our JAX reimplementations trained for 1M update steps on HalfCheetah medium single L40S GPU. Our library, Unifloral, is the fastest across the board. Full details can be found in Appendix F.
algorithms
medeiuu eexpert using
4.2 A Unified Hyperparameter Space for Offfine RL
Implementation inconsistency and missing ablations are common flaws of offline RL research. The plethora of design decisions in each algorithm obfuscates evaluating how each feature contributes to the performance. To address this, we combine all components from a range of model-free and model-based algorithms (Appendix A) into unified algorithm and single-file implementation, which we name Unifloral. We start by compiling a minimal subspace components covering the model-free and model-based offline RL algorithms examined in this work (Appendix H). This has range of perrpaarameeess in each of four broad design categories, which we identify from prior algorithms: model design, critic objective, actor objective, and dynamics modelling. A more detailed description of design category is in Appendix I.
Model Design The choice of neural network architecture and optimizer consistent across most offline RL research, with proposed algorithms commonly using multi-layer perceptrons and the Adam optimizer. However, the hyperparameters of these components commonly vary between algorithms. Regarding the model architecture, this includes the number of layers, layer width, and usage of observation and layer normalization. For optimization, this includes the learning rate (shared and actor-specific), learning rate schedule, discount factor, batch size, and Polyak averaging step size. The actor and critic networks can also have different structures, such the size of the critic ensemble and whether the policy stochasticity.
7

---

Critic Objective The core contribution of offline RL research is often novel critic objective [12. 13]. However, many of the components in the proposed objectives are shared with prior work. We define the critic objective as the weighted sum of those components, or selection between them if mutually exclusive, in order to include all referenced methods (except CQL, which we omit due its substandard performance and high complexity). More detail in Appendix 1.1.
Actor Objective
We define the unified actor loss as the weighted sum of three terms:
Lactor = Bg L. + BBC LBC - 3म H(T((-):))
This consists of y loss L. behaviour cloning loss LBC. and policy entropy H(-), with coefficients Sy. SBC, By R controlling the weight of these terms. More detail in Appendix I.2.
Dynamics Modelling We include optional dynamics model training and sampling, broadening Unifloral's coverage include model-based methods. As is standard, models Dg ( _,,,
ensemble of dynamics L.M), D where each D is trained predict state transitions and rewards. Following MOPO, we penalize the agent for going to states where the ensemble disagreement is high as measured by the standard deviation of the model's predictions. More Appendix I.3,
Novel Methods Research with Unifloral
Our unified algorithm and hyperparameter space enable researchers combine different components and search through algorithm designs by only modifying the configuration of the unified implementa- tion. To demonstrate the avenues our work opens up and encourage further research, we provide two "mini-papers" completed entirely by specifying configurations of the unified implementation, without any code changes. We examine model-free and a mooel--based improvement.
5.1 TD3 with Advantage Weighted Regression
Hypothesis In Section 3.3, we show that one of two methods consistently outperformed existing baselines: ReBRAC [5] and IQL [21]. ReBRAC is derived from TD3-BC, meaning it optimizes its actor using TD3 value loss in combination with BC loss term for regularization. In contrast, IQL uses only a BC loss but performs advantage weighted regression (AWR) by weighting the BC loss of each action by its estimated advantage. We hypothesise that substituting the BC term in ReBRAC with AWR, method we name TD3-AWR, would combine the strengths of these methods and lead to improved performance overall.
Evaluation We define TD3-AWR in Unifloral by using the AWR hyperparameters from IQL and the ReBRAC hyperparameters elsewhere. In Figure we show that TD3-AWR's performance curve strictly dominates ReBRAC on 6 out of datasets and is dominated by ReBRAC in only 1. Interestingly, TD3-AWR achieves superior performance to ReBRAC under few policy evaluations- such as in sachheetah-meiium--eee hyperparameters. Similarly, TD3-AWR strictly dominates IQL on 7 datasets, thereby outperforming both of its source algorithms.
and pen-expert- pen-expert-despite searching over a wider range of
medium expert-v2
pen:
lazge2d-arrge-11
105
70
200
50
1.00
80
40
Meen
95
50
30-
90
20-
85
30
50
10
100
10a
10a
10a
104
10a
10"
10a
10a
10a
10'
10a
Number of policy evaluations
Algorithm ReBRAC
IQL
TD3-AWR
Figure 6: TD3-AWR evaluation against ReBRAC and IQL (full results in Appendix J).

---

5.2 Improving Policy Optimization for Model-Based Offline RL
Hypothesis In Section 3.3, we demonstrate the poor performance of model-based methods on non-locomotion environments. Whilst this is partially due overfit hyperparameters, the design space of policy optimizers in model-based methods is underexplored, with all considered methods using SAC-N or CQL (Figure 5). Given the performance improvements from recent methods, we posit that these methods would be more competitive with an alternative policy optimizer. We therefore propose using ReBRAC with synthetic rollouts generated from a MOPO world model, which we name Model- basee Behaviour Regularized Actor- Critic, or MoBRAC.
Evaluation We implement MoBRAC in Unifloral, using the MOPO hyperparameters for dynam- ics model training and sampling, then using the ReBRAC hyperparameters elsewhere. Figure 7 shows how MoBRAC outperforms other model-based methods for all datasets, except for MOPO in maze2d-large-v Under a transparent evaluation budget, we find that MoBRAC outperforms the other model-basee methods in 6 out of 9 datasets and is tied with MOPO for 3 others (Appendix K).
balfcheotah mwddumdddmmmmm- medium-expertt//
pen-human-vv v1
mazeld large.
lttchen-mmzed-ve ni
100
15
Mean score
75
20
20-
5
25
0
0-
10a
10
10
10
10a
Number of policy evaluations
Algorithm
COMRO
MOPO
MoRal
MoRRAC
Figure 7: MoBRAC evaluation against prior model-based algorithms (full results in Appendix K).
Related Work
Our work builds upon several foundational aspects of offline RL,, including evaluation strategies, open-source implementations, and algorithmic unification. Existing evaluation regimes primarily address hyperparameter tuning either through limited online interactions [8, 9] or by estimating policy performance offline [4, 10, 22]. In contrast, our approach introduces an evaluation procedure that re- quires neither reference policies nor additional hyperparameters. offering broader applicability across the entire D4RL benchmark suite. Furthermore, our single-fil implementations draw inspiration from projects such as CORL [17, 23] and CleanRL [24], whilst our unified algorithm, Unifloral, is informed by prior unification attempts [25, 26, 15. 27J. For a comprehensive review, see Appendix
Conclusion
In this work, we addressed critical challenges in problem formulation, evaluation, and algorithm unification in offline RI. We introduced taxonomy that clearly distinguishes between offline RL Briiants-Sppaning zero-shot deployment to approaches with limited pre-deployment tuning or post-deployment adaptation. This categorization exposes the hidden online interactions, such as hyperparameter tuning, that have long confounded fair evaluation and reproducibility. To overcome these issues, we proposed a rigorous evaluation procedure that transparently quantifies the cost of online interactions via noisy, single-episode feedback. Additionally, by dissecting components of existing offline RL algorithms, we developed Unifloral, novel unified offline RL algorithm that combines improvements of many previous methods, enabling seamless ablation of algorithmic components. We demonstrate this with two novel algorithms inside Unifloral, TD3-AWR and MoBRAC, which integrate the strengths of existing methods to achieve superior performance over wide range of tasks. Collectively, our contributions set a new standard for addressing ambiguity in offline RL, promoting rigorous evaluation, and driving reproducible, impactful research in the field.

---

Acknowledgements
The authors thank Michael Beukman, Cong Lu, Jack Parker-Holder. Hugh Bishop, and Nathan Monette for their valuable feedback on the paper. MJ, UB, and JL are funded by the EPSRC Centre for Doctoral Training Autonomous Intelligent Machines and Systems. MJ also funded by Amazon Web Services, UB is also funded by the Rhodes Scholarship and JL is funded by Sony Interactive Entertainment Europe Ltd.
References
[1] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offlie Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems, November 2020. URL http://arxivv org/ htg//bs/2000 01643. arXiv:2005.01643 [cs, stat].
[2] Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: Model- Offline Policy Optimization, November 2020. URL http: arxiv. www.iiiw./..........oo.oo.oooeee aorg/abb/2005 13239. arXiv:2005. 13239 [cs, stat].
[3] Scott Fujimoto and Shixiang Shane Gu. A Minimalist Approach to Offline Reinforcement Learning, December 2021. URL attt:://arxin wwg. 06800 arXiv:2106.06660 [cs, stat].
[4] Tom Le Paine, Cosmin Paduraru, Andrea Michi, Caglar Gulcehre, Konrad Zolna, Alexander Novikov, Ziyu Wang, and Nando de Freitas. Hyperparameter Selection for Offline Reinforce- ment Learning, July 2020. URL http: //tp:///a/vvv////a org/ 2007 09055, arXiv:2007 000055 [cs].
[5] Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov. Revisiting the Minimalist Approach to Offline Reinforcement Learning, October 2023. URL http: / //arxiv... orgg/bs/////00 09836. arXiv:2305.098. [cs].
[6] Rafael Figueiredo Prudencio, Marcos ROA Maximo, and Esther Luna Colombini. A survey on offline reinforcement learning: Taxonomy, review, and open problems. IEEE Transactions on Neural Networks and Learning Systems, 2023.
[7] Ksenia Konyushova, Yutian Chen, Thomas Paine, Caglar Gulcehre, Cosmin Paduraru, Daniel J Mankowitz, Misha Denil, and Nando de Freitas. Active offline policy selection. Advances in Neural Information Prucessing Systems, 34:24631-24644 2021.
[8] Tatsuya Matsushima, Hiroki Furuta, Yutaka Matsuo, Ofir Nachum, and Shixiang Gu. Deployment-efficceet reinforcement learning via model-based offline optimization. arXiv preprint arXiv:2006.0336 2020.
[9] Vladislav Kurenkov and Sergey Kolesnikov. Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget Matters, June 2022. URL 04156. arXiv:2110.04155: [es].
[10] Matthew Smith, Lucas Maystre, Zhenwen Dai, and Kamil Ciosek. A strong baseline for batch imitation learning. arXiv preprint urXiv:2302.02788 2023.
[11] Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. MOReL: Model-Based Offline Reinforcement Learning, March 2021. URL http:://trrt http::/arxxi..rrr abs/ 2005. 2005.05951. 05951. arXiv:2005.05951
[cs, stat].
[12] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-Learning for Offline Reinforcement Learning, August 2020. URL. http://////ax /20/ 04779. arXiv:2006.04779 [cs, stat].
[13] Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble, October 2021. URL http:// http://arxiv. org/abs/2110. 01548, 11.01:2110001588 [cs].
10

---

[14] Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran. Sergey Levine, and Chelsea Finn. COMBO: Conservative Offline Model-Based Policy Optimization, January 2022. URL http://arxiv.. wwwpw//wwwwwaa.........o
arXiv:2102.08363 [es].
[15] Cong Lu, Philip J. Ball, Jack Holder, Michael A. Osbome, and Stephen J. Roberts. Revisiting Design Choices in Offline Model-Based Reinforcement Learning, March 2022. URL
http://arxiv. http:/axxivooor//bba/ orggabs/21000 04135. arXiv:21 10.04135 [es].
[16] Yihao Sun. Offiinerl-kit: An elegant pytorch offline reinforcement learning library. https: /tt///ffllieel--e 2023.
[17] Denis Tarasov, Alexander Nikulin, Dmitry Akimov, Vladislav Kurenkov, and Sergey Kolesnikov. CORL: Research-oriented deep offline reinforcement learning library. In 3rd Offline RL Workshop: Offline RL "Launchpad", 2022. URL attps://openrevieer wwwpurvviieeeee//frrnn.. net/forum? id=SyAS49BBcv.
[18] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth interna- lional conference on artificial intelligence and statistics, pages 627-635. JMLR Workshop and Conference Proceedings, 2011.
[19] Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47:235-256, 2002.
[20] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
[21] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline Reinforcement Learning with Implicit Q-Learning, October 2021. URL http: http://xxvvv....e wttp.//rrggabba/1.../ooo 06169. arXiv:21 10.066169 [cs].
[22] Aviral Kumar, Anikait Singh. Stephen Tian, Chelsea Finn, and Sergey Levine. A Workflow for Offline Model-Free Robotic Reinforcement Learning, September 2021. URL http: //axivv. org/abs/2109 10813. arXiv:2109. [cs].
[23] Soichiro Nishimori. Jax-corl: Clean sigle- file implementations of offline rl algorithms in jax. 2024. URL https://githut. www.cooo/ii.... http//isssmorrrrir/ -OR-C
[24] Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mchta, and JoAGo GM AraAšjo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23(274):1-18, 2022.
[25] Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan. Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improve- ments deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[26] Matteo Hessel, Ivo Danihelka, Fabio Viola, Arthur Guez, Simon Schmitt, Laurent Sifre, Theo- phane Weber, David Silver, and Hado Van Hasselt. Muesli: Combining improvements in policy optimization. In International conference on machine learning, pages 4226. PMLR, 2021.
[27] Harshit Sikehi, Qinqing Zheng, Amy Zhang, and Scott Niekum. Dual rl: Unification and new methods for reinforcement and imitation learning. arXiv preprint arXiv:2302.00856/ 2023.
[28] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870. Pmir, 2018.
[29] Scott Fujimoto, Herke Hoof, and David Meger. Addressing Function Approximation Error in Actor-Crithe Methods. In Proceedingy of the 35th International Conference Machine Learning, pages 1587-1596. PMLR, July 2018. URL https: http:ts::/proceediigggeeooo/ mlr, .pres//
180/fujimotoi html. ISSN: 2640-3498.

---

[30] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. ddvantage-weightee regression: Simple and scalable off-po reinforcement learning. arXiv preprint arXiv:1910.00177, 2019.
[31] Dean Pomerleau. Alvinn: An autonomous land vehicle in a neural network. Advances in neural information processing systems, I, 1988.
[32] Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:/911.11388 2019.
[33] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling, Advances in neural information processing systems, 34:15084-15097,
2021.
[34] Ishita Mediratta, Qingfei You, Minqi Jiang, and Roberta Raileanu. The Generalization Gap in Offline Reinforcement Learning, March 2024. http::tttp:/ttt. http://axxvv...gg/bbba.s// htte/abs/2311..555// arXiv:2312.05742 [es].
[35] Chris Lu, Jakub Kuba, Alistair Letcher, Luke Metz, Christian Schroeder de Witt, and Jakob Foerster. Discovered policy optimisation. Advances in Neural Information Processing Systems, 35:16455-16468, 2022.
[36] Han Wang, Archit Sakhadeo, Adam White, James Bell, Vincent Liu, Xutong Zhao, Puer Liu, Tadashi Kozuno, Alona Fyshe, and Martha White. No more pesky hyperparameters: Offline hyperparameter tuning for rl. arXiv preprint arXiv:2205.087/6, 2022.
[37] Takuma Seno. d3ripy: An offline deep reinforcement library. https://github. com/ 2akuseno/d3r11y, 2020.
[38] Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines3: Reliable reinforcement learning implementations, Journal of machine learming research, 22(268):1-8, 2021.
[39] Joshua Achiam.
Spinning Up in Deep Reinforcement Learning, 2018. URL https: //spinningup.oppnn com/ http/////aattttneee.....emaa html.
[40] Jarek Liesen, Chris Lu, and Robert Lange. rejax, 2024. URL https://github..cm
keraJli/rejax.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We point out problems in offline RL, describe an informative taxonomy Fig- ure I and propose an evaluation procedure, the results of which support our claims Sec- tion 3.3. We then provide extensive explanation on our implementations design and performance (Section 4. I). The unified implementations (Appendix I) and the evaluation procedure (Section 3.2 are used to discover two algorithm that describe and test in Section 5.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We address lack of algorithmic clarity which is inherent in the field we are aiming to unify Section and we acknowledge the limited number of evaluation environ- ments Appendix C. In our full results for the two novel algorithms, Figure 15 and Figure 14, we include challenging environments where our algorithms underperform. However, these results themselves are further evidence of the robustness and reliability of our evaluation procedure.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they seale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess..
13

---

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA].
Justification: This paper does not have theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We plan to open-source our library. Regardless, we have provided with exhaustive detail on algorithm similarity Section and our code philosophy Appendix To encourage transparent and reliable scientific practices, we also provide snippets from the actual code demonstrate our clean and consistent implementations Figure 12 and Figure 13.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., s., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
14

---

(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e-g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Yes, the paper features anonymous repository and is built to optimize for transparent ablations, reproducibility and evaluations. We use standard D4RL [20] datasets available through their API.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/ mublic/guide httpi//ggiiieeeoeeeubbbmiaaaa sow.aamoiiiiioooooccccooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurlPS code and data submission guidelines (https: wwtpccccp///aaececcceeeceeeeeeee wewlcc/ggieeeeeeee sondeeSumisssionPee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, we have a unified hyperparameter space for all the algorithms Table 5 and describe every design decision in Section 4.2 and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
15

---

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: As described in Section 3.2, we provide bootstrapped estimate of each algorithm's 's performance as more online evaluations are allowed to be used for hyperparam- eter selection. We plot mean and 95% confidence intervals everywhere and add mean and standard deviation when relevant, like Figure 4).
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, in Table and Figure we specify the number steps used for the runtime measurements and the hardware, namely single Nvidia-L40S GPU.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental rums well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics /ttps://neurii: httpp//rwwppsppppppeeeeeeeeeeeee
Answer: [Yes]
Justification: There experiments in this work requiring human participants. All the datasets are the public domain and have been used in accordance with their respective licenses.
Guidelines:
16

---

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there is a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA].
Justification: There are societal impacts beyond encouraging the researchers and prac- titioners to report results more unified and transparent way. We provide tools, baselines and demonstrate the effectiveness of our work by discovering two new algorithms.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out, For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA].
Justification:
This is not relevant for the paper. The justification is similar to the above.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
17

---

Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We give credit and cite every piece of code that we use or were inspired by. We emphasize that our paper does not rely on extensive amounts boilerplate code. which is also one of our key contributions (Appendix E). Every dataset we used in accordance with its license and standard community practices.
Guidelines:
existing assets. The answer NA means that the paper does
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, haperswithcom com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if it has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: or [NA]
Justification: We do not provide any assets beyond the implementation, configuration files and package requirements.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA].
Justification:
This paper does not contain such experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
18

---

According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [NA].
Justification: This paper does not involve crowd-sourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The contributions and reproduction of this work does require any LLMs. neither did its ideation and development.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.MM policy (https: /ttp:///eeuuppeeeeeee f20ee/25 /LLM) for what should or should not be described.
19

---

A Algorithm Implementations in Unifloral
A.1 Model-Free Offline RL
Soft Actor-Critic (SAC) by Haarnoja al. [28] is Q-learning method with stochastic actor. The authors use two independently optimized Q-functions and take their minimum for the value function gradient to reduce positive bias in the policy improvements. SAC uses function approximators for both the policy and value functions.
SAC
EDAC Function approximators do not operate well oitoof-distribution (OOD), which poses a significant challenge for offline RL methods that rely fixed dataset of logged trajectories. An et al. [13] propose increasing the size of the Q-function ensemble. They find that SAC requires large ensemble avoid optimistic value estimations for OOD actions as the cosine similarity of the gradients increases. To minimize this similarity within the ensemble, the authors propose Diversified Actor-Critic (EDAC), which adds an ensemble similarity penalty to the Q-function loss in SAC. We refer to SAC with more than two members in the ensemble as SAC-N.
the
CQL Optimistic value estimations when bootstrapping from OOD actions is persisting issue in offiine RL. Kumar et al. [12] propose learning conservative Q-function that lower bounds the true value. They perform SAC updates to the Q-function with an additional minimization term that uses the value of randomly sampled actions. Their Conservative Q-Learning (CQL) algorithm is also implemented on top of a SAC-N policy update similar to EDAC.
TD3-BC Fujimoto et al. [29] formulate the Twin-Delayed Policy Deep Deterministic policy gradient algorithm (TD3) to address the value estimation pathology in online RI where the ensemble of Q-networks is updated at a higher frequency than the actor. TD3 also takes the minimum over the critics ensemble as CQL. SAC-N, and EDAC. Follow-up work by Fujimoto and Gu [3] adapts the method for the offline paradigm by adding a behaviour cloning (BC) regularization term to the actor's s updates. This augmented algorithm is commonly referred to TD3-BC. Not having to update two networks in every training step brings significant speed-ups while still matching the highest scores across all D4RL [20] locomotion tasks at an increased stability.
IQL Implicit Q-learning by Kostrikov et al. [21] computationally efficient algorithm that avoids querying out-of-sample actions altogether by using expectile regression. The Q-function is updated using a mean squared error loss on state-action pairs from the dataset. This approximation of the optimal Q-function is used to extract the policy through earantage-weightee regression [30], where each action is weighted according to the exponentiated advantage with an inverse temperature hyperparameter that directs the policy towards higher Q Q-values when increased and approximates behavior cloning [31] when decreased.
ReBRAC Tarasov et al. [5] use the Behavior Regularized Actor-Critic (BRAC) framework [32] and the behavior cloning term from TD3-BC [3] to propose the Revisited BRAC algorithm (ReBRAC). Specifically, they decouple the BC penalty coefficient in the critic and the actor objectives, thus requiring additional hyperameteres to the benefit of higher scores and faster convergence on D4RL. In addition, ReBRAC [5] proposes several improvements, like using deeper networks, training with larger batches, adding layer norms to the critic network, and changing the hyperparameter for tasks with different reward sparsity. However, these design decisions add new hyperparameters with tuning overheads since they are reportedly different for each D4RL dataset.
A.2 Model-Based Offfine RL
MOPO In Model- Offline Policy Optimization (MOPO). Yu et al. [2] argue that offline RL algorithms should be able to go beyond the behaviors in the data manifold to avert sub-optimalities in the dataset and generalize to new tasks to deliver on the promises of real-world deployment. MOPO provides several bounds and theoretical guarantees on behavior policy improvement. The model implemented through an ensemble of multiple dynamics models trained via maximum likelihood. For every policy step during training, the maximum standard deviation of the learned models' prediction at that step is subtracted from the reward. The highest results obtained on short truncated rollouts
20

---

that are 0.5% to 1% of the real environment's 's episode length. The model predictions are used to form the batch for the SAC [28] policy update step.
The model-based offline RL algorithm (MOReL) by Kidambi et al. [11] claims to not require severely truncated rollouts due to learning a pessimistic MDP (P-MDP) that is implemented in similar way to the MOPO dynamics model with an additional early termination condition in the event high ensemble disagreement. This scalar halting threshold is calculated by taking the maximum distance between the predictions of any two models of the ensemble for every state and action pair the dataset. Even for academic demonstration datasets like D4RL, this poses major overhead in addition to model and policy training. The reported rollout length approximating 50% of the original episode length is only achievable through extensive tuning the pessimism coefficient that scales the discrepancy threshold.
MOReL
COMBO Conservative Offline Model-Based Policy Optimization (COMBO) by Yu et al. [14] is implemented top of MOPO [2] with more policy improvement guarantees. They CQL [12] policy update step with an added loss term using transitions from the dataset to penalize Q-values on likely out-of-support state-actions while increasing Q-values on trustworthy pairs. There are many similarities across model-based methods, and many of their algorithmic contributions like the P-MDP from MOReL, uncertainty penalties from MOPÓ, and the policy update from COMBO can be combined through our framework.
A.3 Imitation Learning
This section examines methods that operate outside traditional RL paradigms. These methods identical offline RL datasets and have achieved scores comparable to other offline RL methods when evaluated under the same conditions.
Behavioral cloning (BC), originally formalized by Pomerleau [31], directly optimizes the actor by learning the transitions from the dataset in supervised manner, thus making the final online performance fully reliant on the quality the dataset. Recent work by Kurenkov and Kolesnikov [9] further points out the effectiveness of BC under restricted budgets.
BC
DT Introduced by Chen et al. [33], Decision Transformers (DT) have shown remarkable general- ization [34] in ORL. DT bypasses the need for traditional RL algorithms to discounted rewards and bootstrapping for long-term credit assignment by using the logged environment interactions as a sequence modelling objective. Instead of sampling from policy conditioned the current states, the trained transformer autoregressively generates the next action based on a fixed intra-episode context of previous interaction and target cumulative return. This target return be hyperparameter that significantly increases the tuning overhead if its value is unknown or a way to obtain optimal performance results when the target return is known.
The reward at each step is decremented from the target return, which is referred to as return-to-g at time L. Formally, Rg = Ei= Tr where Tv are the observed rewards. Rather than directly modelling the reward function the model is conditioned on the return-to-go values enable generation based on desired future returns.
The trajectory representation is structured ordered sequence of return-to-go values, states, and actions:
(Ri 81, 01, R2, 82, a2,..., Ri, ST, ar).
where (st, (.) S x A for all timesteps t.
During online evaluation, the model is initialized with desired target return and an initial state so Sy. After executing action a,, the received reward is subtracted from the target: R R₁ Tt-
21

---

B Full Results from the Proposed Evaluation Procedure
Iopparsomddimm222
100
100
60
60
20
pea-clocediv1
expert-v]
Mean
20-
50
102
101
100
sitnaaz-largo--iveree-z
moan26-large-v1
kitchen -i
40
150
30
10
100
16a
101
10P
Number of policy evaluations
Algortthm
COL.
IOL.
MoRal
SAC-N
COMBO
EDAC
MOPO
REBRAC
TD3-BC
Figure 8: Evaluation prior algorithms mean and 95% CI over 500 bandit rollouts, with K policy arms subsampled from 20 trained policies each rollout. The r-axis denotes the number of bandit pulls, whilst the y-axis denotes the true expected score of the estimated best arm after pulls.
22

---

C Evaluation Benchmarks in Prior Work
Table 1: Evaluations performed in the papers introducing the offline RL. algorithms we consider. A "v" indicates complete evaluation, "~" indicates a partial evaluation, and "-" indicates that the domain was not evaluated. MuJoCo locomotion is the most widely studied domain, although random and expert datasets often omitted. Atari experiments are limited to only 5 datasets (Breakout, Qbert, Pong, Seaquest, and Asterix). Notably, the model-based offline RL works referenced here only evaluate on locomotion tasks, which may explain their dramatic performance collapse on non-locomotion tasks.
D4RL Fu et al. [20]
Algorithm
Locomotion Adroit Kitchen Mazeld AutMaze Minigrid
Carla
Flow
Atari
CQL
EDAC [13]
[Q1]21] [21]
ReBRACIST
SAC-N [13]
TD3-BC
COMBO [14]
MOPO []]
MOReL [11]
D Distractor Policy Phenomenon
Here, we show additional observations from the analysis distractor policies Section 3.3.
Bandit overestimates values of distractor policies
105
0.65
Probubiitt of preferiing distmccor policy
100
0.60
0.55
score
90
0.50
0.45
80
0.40
Number subtsmpped policies
32
0.35
70
16
0.30
20
40
60
80
100
120
10"
10
10"
Number of pulls
Number policy evaluations
(a) Probability of preferring distractor policy (inside dashed orange lines Figure against the number of pulls (mean over 100K random policy orderings). The probability of preferring an unstable policy increases over time.
(b) The number of subsampled policies influences evaluation behaviour the number policies in- creases. we observe greater dip selected-policy performance from our UCB bandit. This is due to the presence of distractor policies (Figure 4), which achieve higher peak performance with lower mean.
23

---

E
Code Philosophy
E.1
Single-file
We follow the community's preference for single-file algorithm implementations with integrated loggers and evaluations [17, 23, 35, 24]. All of our model-free algorithm implementations are self- contained, with every object necessary to set the hyperparameters, run the training loop, and evaluate the policy included in a single file. As model--aseed methods typically run sequential dynamics and policy training phases, we implement a single-file dynamics training script that saves trained model checkpoints. These can then be imported by any of the policy training scripts for the model-basee algorithms.
E.2
Consistent
Even within the same library, algorithm implementations often differ in boilerplate code. We change the minimum number of lines between implementations to control for implementation differences and help developers. Guided by the design genealogy illustrated in Figure 10a, we first ensure the single file implementation of the base algorithms like BC and SAC-N is clear and concise (Figure 10b) and then make minimal differences from their algorithmic ancestors (Figure 10c).
Figure 11 shows the minimal differences between clean implementations of each algorithm, and Figure 11b shows the line differences from CQL. We acknowledge that prior implementations do not directly seek to minimize the differences between single-file implementations, but we believe to be beneficial feature for research. See Figure 12 and Figure 13 for a more complete illustration of the full code.
Tile Lengtte Bont Mppetthmm
Code Defhernooss Bettwer Phment Chilc Alguctthme
Hesiforn CORE
Uniiionl DOTL
BC
SAC N
TDXI-CC
400
325
ReBRAC
MORd.
COMBO
(a) Genealogy of algorithms.
(b) Length of root algorithms.
(c) Length of diff from parent.
Figure 10: We provide clean and consistent single-file implementations, as demonstrated by compact implementations and minimal differences between algorithms.
Algoritho: "sac_n"
Difference from the CQL Implementation within each repository
algoriths: "edac"
updato
1200
Blax vatue, goad
guardaat value, gead, aur-Tue)
Ditierrt Layes
(n(oursen):
Guored suopply batch, ebs, batch actionn
return Jno, squarel(q jnp, expane.dtas(tary -131), sum(-13 aean(]
Inp. squaret(e,
jnp. targetarget, -1])
634
less value, sun(-1), m=n))
544
diveruity. loss a divepppdveeesty..l.. lassty___________aaaa (na_sfhhhhbhhh.oaoaa abs, batch action)
liversity_loss C1 Cargs nunteritics 13) diversity rean()
crstic_loss valur_loss args.ets dtversity..oss
230
return critic.loc. loss, Lostue-loss, diversity. less)
critic, less, critic.grad Lo, fatase.fttggntttaaa vec.g parans)
Ceritic, Ccalun, agent_state vee parans)
diversity less)),
Losloss_fft
BDAG
updated.q agent_state vec.g. apply. www.denntsggnnerereetnoo apent_staaee
_rrppace ptpppppeeee.
(a) Using command line tool diff on our implementations of SAC-N and EDAC.
(b) Implementation length difference of each al- gorithm from CQL in their respective repository.
Figure 11: Analysis of algorithmic differences between offline RL implementations.
24

---

withe
Tay
unto wuw.aaa Lit she poees oil
cntienn
unteriig sart ae1
Jor schree
leton De Sep gou
4eb Saaneey slantee))
actiin Jaaen' appn Dreamm Oo altec mncinite
CnC..10 Jem es dannry
Inn. teaill milee
dae Lon
Figure 12: All code edits across implementations, from left right: SAC-N, CQL, and EDAC.
25

---

Figure 13: Full code difference for SAC-N, EDAC, and CQL from left right. The code for the final evaluation loop is omitted to illustrate the consistency of the algorithm implementations.
26

---

Reimplementation Training Time
Table 2: Speed up from our JAX implementations, training time in minutes. Algorithms trained for 1M update steps on HalfCheetah Scheetah-medium expert using single L40S GPU. Our library, Unifloral, the fastest across the board.
Algorithm OffiineL--Ki Kit CORL JAX-CORL Unifloral
BC
19.8
15.0
1.7
TD3-BC
56.1
42.5
6.9
3.1
IQL
79.7
65.0
5.2
4.0
ReBRAC
8.7
6.8
SAC-N
107.5
98.8
7.7
CQL
203.9
180.3
20.7
9.8
EDAC
127.1
113.0
20.8
MОPО
168.1
14.0
MOReL
14.0
COMBO
289.6
22.0
G Results Reproduction
Table Performance our algorithm reimplementations over 5 training seeds, Mean+Std.
COMBO
CQL
EDAC
MOPO
ReBRAC
SAC-N
TD3-BC
HallC TallChe
Expert
0.4
9.3
33
2.3
62.7 9.3
43.0
0.9
988
9.
Medium
4.2
1.2
6.99
52.2
4.
728
72.1 1.6
68.613
65.2
4..6
Medium-Expen
5944 109
936+4.7
611
1028+1.1
92.3+3.1
80.
1922
632268
104.5
1034
92.9335
Medium-Roplay
3..
54.44
13.6
55.2111
558
43
65.0
654:35
49.1
5.4
44
Random
2.2
1.6
30.7
168 13.3
12.5=30
30.5
31.8
16.9
17.8
1.6
12
Expet
109,5:3333
12.5 15.3
1,3
4.2
10555
222008
1066
1.
93.8
12.2
1.9.4.3.1
Hopper
Modium
5..7
3.1
7.
1008
5.67
46.6
27.0
10.4
1018
7
6..3
Medium-Exppe Expent
4..6
2.
12.2330
1.9.9
4.
47.2
77.0
44.4
108.0
9
105.2
Medium-Replay
7.55
Random
4.9.1.88
2..0
22.0 128
22.6 15.2
7.10.11
31.1000
13.
7.
26.6 10.5
8.1007
Expert
108.5
22.6
2.4
112.7 79.0
55.5
10.7
194
1124
322
110.3
Walker2d
Medium
63.8
84.5+04
87.9
75.1 40.9
84.0=20
81.3+26
164+
36.9
84.3+2.
8.9
84.5
Medium-Eapert
108.1
00
889
363
0129
1118
1100
28.7
111.6
114.
110.1
Medium-Replay
7..
7.44
Random
0.4
3.4+22
9.1
2.+000
4.
43+63
03+
173*89
20.7
0.3 0.3
Table 3 presents the results achieved by method reimplementations locomotion datasets, matching the performance of prior implementations [17]. For our library's completeness, we also implement the Decision Transformer (DT) [33] using the hyperparameters from CORL [17].
Table 4: Performance of our Decision Transformer implementation over 5 training seeds, Stdn
Ens:
Dataset
Decision Transformer
Expert
92.9 + 0.1
HalfCheetah
Medium
42.8
Medium- Expert
92.5 0.2
Medium Replay
37.8 1.3
Expert
110.2 + 1.5
Hopper
Medium
Medium--xpper Expert
111.3 3: 0.5
Mediumn Replay
25.5
Expect
108.4 4
Walker2d
Medium
6.03
Medium- Expert
108.1 +: 0.3
Medium- Medium-Roplay
53.244 13.7
27

---

H
Unifloral Hyperparameters
Table 5: Hyperparameters of prior algorithms in Uniftoral light gray values indicate inactive settings.
Hyperparameter
IQL
SAC-N
EDAC
TD3-BC
ReBRAC
Batch size
256
256
256
256
1024
Actor learning rate
3e-4
3e-4
3e-4
3e-4
le-3
Critic learning
3e-4
3e-4
3e-4
3e-4
1e-3
Learning rate schedule
cosine
constant
constant
constant
constant
Discount factor 7 Polyak step size
0.99
0.99
0.99
0.99
0.99
0.005
0.005
0,005
0.005
0.005
Normalize observations
True
False
False
True
False
Actor layers
2
Actor hidden size
256
256
256
256
256
Actor layer normalization
False
False
False
False
True
Deterministic policy
False
False
False
True
True
Deterministic eval
True
False
False
False
False
Apply tanh to mean
True
False
False
True
True
Learn action std Log std min Log std max
True
False
False
False
False
-20.0
-5.0
-5.0
-5.0
-5.0
2.0
2.0
2.0
2.0
2.0
# of critics
2
[5-200]
[10-50]
2
2
Critic layers
2
3
Critic hidden size
256
256
256
256
256
Critic layer normalization
False
False
False
False
True
Actor BC coefficient
1.0
0.0
0.0
[5e-4-1.0]
Actor Q coefficient
0.0
1.0
1.0
[1.0-4.0]
1.0
Use Q target in actor
False
False
False
False
False
Normalize Q loss
False
False
False
True
Truc
Q aggregation method
min
min
min
first
min
Use AWR
True
False
False
False
False
AWR temperature
[0.5-10.0]
1.0
1.0
1.0
1.0
AWR advantage clip
100.0
100.0
100.0
100.0
100.0
Critic BC coefficient
0.0
0.0
0.0
0.0
[0-0.1]
of critic updates per step
Diversity coefficient
0.0
0.0
[0.0-1e3]
0.0
0.0
Policy noise
0.0
0.0
0.0
0.2
0.2
Noise clip
0.0
0.0
0.0
0.5
0.5
Use target actor
False
False
False
True
True
Use entropy loss
False
True
True
False
False
Actor entropy coefficient
0.0
1.0
1.0
0.0
0.0
Critic entropy coefficient
0.0
1.0
1.0
0.0
0.0
Use value target
False
False
False
False 0.8
False
Value expectile
[0.5-0.9]
0.8
0.8
0.8
28

---

I Unified Algorithm Details
In this section we write out the different design decisions in a unified notation.
I.1 Critic Objective
First, compute the value target using one of two methods, selectable via the method configuration:
U/+1 {
u($+11) min on (st+1, clip(år+1 (e, Emin, Emax). Amin amax)
where N is the numher of ensemble members, is value function trained with expectile regression (as in IQL. [21]) 04-11 ~ x x(a++1|8t4] is an action sampled from æ (or Polyak averaged target
policy), (0, o²) is random action noise with standard deviation a, and Cmins Cuaxx Omin- and amax are clipping ranges. The value target is then augmented with behaviour cloning and entropy terms (coefficients aBc and a4)), defined
0 = 4++1 + OBc (ă++1 - a441) + or (((T(1((-)))
(5)
which is then used to compute the value loss,
L > (gn(se; 0e) - (r d). d -x41)))
(6)
Finally, we add the critic diversity loss term from EDAC [13] with coefficient dav giving the final critic loss
Lemte Lu + N-1
Ordiv
Σ
(V),)))))))) Va, (5(,
L.
Actor Objective
We write the actor loss as:
Lactor = Bg Lg + BBC LBc - Bu H(((--s::)..
(8)
This consists of loss Lg. behaviour cloning loss LBc, and policy entropy H(-), with coefficients Bg. SBC, Bu E R controlling the weight these terms.
The first term, L, is defined simply by selectable aggregation function over the nentwwrkk ensemble, with the minimum being the most common choice,
min N (gn (sc, at)) Lgg
N Ew In (8,, az)
qo (s),
This term may also be normalized across the batch in order to stabilize learning. The second term, LBC, is most commonly defined as the distance d between the target policy and dataset action, being the mean squared error for deterministic policies or log- pee-robbbbiltyy for stochastic policies. However, some methods use advantage weighted regularization (AWR), which further weights this loss by the clipped and exponentiated advantage of the behaviour policy action order clone only positive actions within the dataset. Therefore, this term has the following variants:
(a: à²)² - log (u))s))
{
d
d =
CBC =
(10)
d min (Amax, e
where 7) and Amax are the temperature and maximum value for exponential advantage.
29

---

1.3 Dynamics Modelling
We use an ensemble of dynamics models D, = {D]. D3..... D.., where each D is trained to predict state transitions and rewards. Following MOPÓ, we penalize the agent for going to states where the ensemble disagreement is higher, as measured by the standard deviation of the model's predictions. This used to penalize the reward during policy optimization with pessimism coefficient 7
M Σ m=
R(s, 0r)
01) - 1) o a.).
(11)
M
where a (Dg (st, at)) represents the standard deviation across the models' state-change predictions and R8 (8;; 02) is reward prediction of the m-th ensemble member.
J Complete TD3-AWR Results
hopper-madium42
helfcheeta bollcheetb--meelunnnpprrrr
105
100
60
90
60
pen-humaneel
pen-cloned-er
pen-expert-v]
70
50-
Mean
130
125
ontmazo-Jagee mntzaaz-laarg-divveee...
maza26-large-vi
kitchen -i
200
50
150
10
100
20
0
10a
10'
10
Number of policy evaluations
Algorithm REDRAC
JQL
TD3-AWR
Figure 14: Full comparison of TD3-AWR to prior model-basce methods across all datasets.
30

---

K Complete MoBRAC Results
hopper-medium-22
w
100
80
80
60
80
60
40
20
10P
10'
10a
10"
10'
10'
10"
10
10"
pen buman:v1
pen-cloned-v1
pen expeepertvvl
100
00
75-
Mean
20
20
50
0-
-arge
masae2-laagge-v1
kitchen-mixed--d
0.020
15
10
0.010
0.005
0
10P
10!
10a
10"
10'
10a
10"
10'
10
Number of policy evaluations
Algorithm
Figure 15: Full comparison of MoBRAC to prior model-based methods across all datasets.
31

---

L Full Related Work
In this section, we describe the prior work related our evaluation procedure, implementation, and unified algorithm. We implement a comprehensive selection of offline RL algorithms, for which more information can be found in Appendix A.
L.1 Evaluation Regimes for Offline RL
The challenge of hyperparameter tuning in RL spans various domains. Wang et al. [36] discuss offline tuning and the practical risks of deploying policies of unknown quality in the real world, whilst Paine et al. [4] directly tackle this issue, estimating the zero-shot performance of offline-trained policies without any prior online interactions. Their evaluation is limited to behavioural cloning [31, BC] and two critic-based methods, which have since been outperformed by modern algorithms. Konyushova al. [7] extend this procedure with an online phase, using a UCB-based bandit to investigate policy selection over multiple online evaluations. Further highlighting these challenges, Smith et al. [10] propose procedure where offline evaluation methods are first calibrated using policies of known quality, evaluating on D4RL [20] locomotion tasks. Unlike their work, we evaluate across the D4RL suite and introduce a procedure that eliminates the need for reference policies or additional hyperparameters. Matsushima et al. [8] present variant of offline RI that uses limited number of online deployments to update the dataset and iteratively train offline to match the performance of online methods, introducing an online deployment frequency hyperparameter. Kurenkov and Kolesnikov [9] address the practice of unreported online evaluations for hyperparameter tuning, demonstrating how the performance of each algorithm changes with the number of online evaluations. Unlike our procedure, they assume a low-variance estimate of a policy's 's true performance each evaluation but still conclude that BC outperforms all baselines.
L.2 Open-Source Implementations
Offline RL Inspiring our implementation, Clean Offline RL [17, CORL] provides single-file implementations of model-free offline RL methods in PyTorch. JAX-CORL [23] is a JAX-based port of CORL, albeit with a limited range of only model-free algorithms, slower training time than our implementations, and lacking our evaluation procedure and code consistency. OfflineRLiit [16] and d3ripy [37] implement a range of offline RL methods and feature both model based and model-free methods. Although the repository has transparent class inheritance and polymorphism, it lacks any further attempt at algorithmic unification.
Online RL StableBaselines3 [38] is a of reliable RL algorithm implementations in Py PoTcch with the aim of abstracting away training and deployment through an object-oriented interface. SpinningUp [39] is similar, education-orienter effort of jointly implementing various online RL algorithms. CleanRL. [24] follows a different design philosophy with method-focused, single-file implementations of online RI. algorithms in Py'Torch and JAX. PurelaxRL [35] also follows the
single-file approach and is implemented in JAX. Rejax [40] is a popular multi-file JAX-based implementation of PureJaxRL with extensive logging integration and a selection of SOTA methods.
CleanRL, CORL, and JAX-CORL provide clear and accessible logs their final runs, a standard of reproducibility we plan to uphold throughout every release of our work.
L.3 Method Unification
Our unified algorithm, Unifloral, heavily inspired by prior work that also seeks to ablate and unify range of methods. Lu et al. [15] investigate the key components of model-based offline RL algorithms to find an optimized algorithm that outperforms all model-based baselines. Sikchi et al. [27] cast multiple offline RL methods in the same dual optimization framework and use this unification to categorize them in regularized policy learning and pessimistic value learning. Prudencio et al. [6] provide a survey of offline RL, focused on elucidating the taxonomy and disambiguating the contributions of each algorithm. In online RL, Hessel et al. [25] combine independent components of Deep Q networrk algorithms into unified algorithm, Rainbow, reaching SOTA in the Atari 2600 benchmark. Muesli [26] examines the combination of policy optimization and model-based methods.
32


Reviews:
[{"review_id": "zlepO34w65", "reviewer": "Reviewer_2NhV", "summary": "This paper identifies key issues in offline RL research, including ambiguous problem definitions, inconsistent evaluation practices, and overly complex implementations. To address these, the authors propose a transparent evaluation framework, release minimal single-file implementations of major offline RL methods, and introduce Unifloral, a unified algorithmic framework that spans prior approaches via a shared hyperparameter space. Using this framework, they develop two new algorithms—TD3-AWR and MoBRAC—that outperform standard baselines.", "questions": "1. Why is there no discounting in equation 1?\n2. The difference between post\n4. Why specifically do the authors recommend those particular datasets (line 164)? It seems like the argument is simply \"more is better\" \n5. My interpretation is that \"distractor policies\" are just policies with higher variance returns (perhaps because they have not yet converged to a nearly deterministic policy)?\n4. \n> we uniformly sample from the hyperparameter tuning ranges specified in each algorithm’s original paper or the 176 union of ranges when multiple are provided.\n\nCould the authors include these hyperparameter ranges in the appendix? Given that there are many algorithms evaluated, it would be a bit tedious for a reader to track down these values across many papers. This would also help answer one of my questions posed in weaknesses section.\n5. \n> either through pre-deployment interaction, which incurs no cost and occurs before evaluation, or through post-deployment adaptation, where performance includes the return during 107 these additional interactions.\n\nI'm a bit confused; how does pre-deployment interaction carry no cost? And what does it mean for performance to \"include\" the return during addition interaction?", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n1. The paper makes several fair critique of current practice in offline RL research that the community should seriously consider.\n1. The framework and empirical evaluation is thorough, capturing a wide range of popular offline RL algorithms.  One result I found particularly interesting that models-based algorithm results reported in prior works are overfitted to the task. \n1. The contributed codebase looks promising, and I anticipate it would indeed make it easier to understand what makes or breaks offline RL methods. (Algorithm implementations are indeed opaque. Anecdotally, I distinctly remember having quite a bit of trouble reproducing CQL results on AntMaze tasks years ago. I found that performance hinged on implementation-specific details like reward normalization.)\n\n**I personally feel that the contributions and call to action put forth by this paper outweighs shortcomings and lean accept.**\n\n# Weaknesses\n\n1. **Evaluation procedure.** The proposed evaluation procedure uses random hyperparameter sampling followed by UCB over evaluation rollouts. Why was random sampling chosen over the standard grid search? Additionally, the use of the mean score vs. number-of-evaluations curve may be less relevant in contexts where we care most about the final converged performance. Aren't we ultimately interested in the score these curves converge to? While it’s true that in practice we may be limited in the number of evaluations, doesn't the evaluation curve’s plateau reflects the best estimate of a method's performance? \n\n1. **Is it reasonable to report performance as the aggregate over many hyperparameter settings?** This paper takes the stance that we should factor in hyperparamter tuning as part of the empirical evaluation, which I agree with. But I'm not so sure we should be aggregating performance over hyperparamters during the UCB evaluation. For this to be appropriate, we need to be sure that the range of hyperparameters we use reflects a \"reasonable\" range [1] -- which is admittedly not well-defined. What happens during evaluation if we set the hyperparameters range to something obviously too wide? This would produce many very suboptimal arms that UCB would quickly eliminate, but we'd also have fewer high-scoring arms which would ultimately lower the score evaluation converges to after collecting a large number of trajectories, right? \n\n\n# Minor comments \n\n> Thus, the state-of-the-art remains ambiguous, with no method demonstrating uniformly strong performance across all datasets.\n\nThis phrasing suggest that if we evaluated algorithms fairly, one SOTA algorithm would rise to the top. But it's common for some methods to perform well on some tasks but not others. \n\n1. Line 186: Missing period/colon after bold header.\n\n---\n# References\n1. Patterson et al. \"Empirical Design in Reinforcement Learning.\" https://www.jmlr.org/papers/v25/23-0183.html", "was_revised": true, "final_justification": "As stated in my original review, I think this paper is useful and some of the key results are interesting to me. I believe this paper is a nice contribution to the RL community. Any issues with this paper are minor and easily addressable in the camera-ready.", "rebuttal": "Thank you for appreciating our work's contribution, the thoroughness of our evaluations and the value of our codebase. We address your questions and concerns below.\n\n## Weaknesses\n\n> The proposed evaluation procedure uses random hyperparameter sampling followed by UCB over evaluation rollouts. Why was random sampling chosen over the standard grid search?\n\nFor fair comparison via our bandit protocol, the number of policies trained with each algorithm should be constant. When using grid search however, the number of policies trained would be equal to the number of hyperparameter combinations. Thus, it is necessary to subsample the set of hyperparameter combinations, which we do with random search to remain unbiased.\n\n> Aren't we ultimately interested in the score these curves converge to? While it’s true that in practice we may be limited in the number of evaluations, doesn't the evaluation curve’s plateau reflects the best estimate of a method's performance?\n\nPerformance given a limited budget is of very high practical and theoretical significance. For example, zero-shot performance is a practically relevant metric, and siginifcant efforts have been made to maximize zero-shot performance based on offline analysis of trained policies (see e.g. the works referenced in our Related Work section, line 317).\nOur evaluation method gives the flexibility to compare methods at different budgets, **including** zero-shot and final, i.e. plateaued, performance. Thus it is more informative, and measures a method's performance in a more general sense.\n\n> \\[...\\] we need to be sure that the range of hyperparameters we use reflects a \"reasonable\" range \\[1\\] -- which is admittedly not well-defined. What happens during evaluation if we set the hyperparameters range to something obviously too wide? This... would ultimately lower the score evaluation converges to after collecting a large number of trajectories, right?\n\nThat is correct - methods with wide hyperparameter ranges, that require a large tuning budget to perform well, will perform poorly under our metric. Our evaluation was designed precisely to demonstrate this point, as prior work considers only the maximum performance within the hyperparameter range, disregarding tuning cost. It is also for this reason that we define offline RL methods (Section 3.1) as including the hyperparameter range, so an improved range over the same hyperparameters is viewed as a superior method.\n\n## Minor Comments\n> > Thus, the state-of-the-art remains ambiguous, with no method demonstrating uniformly strong performance across all datasets.\n>\n> This phrasing suggest that if we evaluated algorithms fairly, one SOTA algorithm would rise to the top. But it's common for some methods to perform well on some tasks but not others.\n\nWe stand by our assessment that SOTA remains ambiguous, since no algorithm has risen to the top. The existance of a single SOTA algorithm is not to be taken for granted, as you have pointed out.\n\n> Line 186: Missing period/colon after bold header\n\nWe have fixed that for the camera ready, thank you for pointing it out.\n\n## Questions \n\n> 1. Why is there no discounting in equation 1?\n\nWe consider finite-horizon MDPs, which do not require discounted return in their mathematical formulation as the episodic return is finite. Despite this, many methods still include a discount factor to aid credit assignment, but this is not theoretically necessary.\n\n> 2. The difference between post\n\nWe are making an educated guess that this refers to the post and pre-deployment setting; we are happy to clarify further if you were referring to something else.\n\nWe use *deployment* as the point in time at which we stop training and start evaluating our agent. In offline RL, an agent has no access to the environment before deployment. However, some methods relax this strict separation, allowing one (or both) of:\n\n- *pre-deployment interaction*, in which the agent may interact with the environment prior to deployment -- for instance, to select one of several policies to deploy based on online performance. Hyperparameter tuning needs pre-deployment interaction, since policies' returns are used to sample new hyperparameters to test, but the final performance metric is the performance *after* tuning, i.e. after deployment. \n\n- *post-deployment adaptation* allows the agent to continue learning after deployment, and the performance metric includes all returns collected after deployment. For example, when training a recommender system offline and finetuning it with live user data. \n\nWe believe that this clarification addresses most of the reviewers' doubts, and have improved the wording in the paper accordingly (including the usage of the overloaded word \"cost\" mentioned in line 106 of the paper).\n\n\n> 3. Why specifically do the authors recommend those particular datasets (line 164)? It seems like the argument is simply \"more is better\"\n\nWe select a fixed set of datasets to make future evaluations consistent and immune to cherry-picking. Of these, the individual datasets were selected to provide coverage over the maximum number of environments and behavior policies, as well as focusing *only* on datasets where existing methods do not achieve trivial performance (contantly nil or perfect), providing meaningful signal for future algorithm comparison. We note that for halfcheetah and walker datasets, the bandit evaluations converged to a clear final ranking that is different from other intermediate points throughout the bandit's score trajectory. This further demonstrates the additional insight our proposed evaluation procedure provides.\n\n> 4. My interpretation is that \"distractor policies\" are just policies with higher variance returns (perhaps because they have not yet converged to a nearly deterministic policy)?\n\nYou're correct that distractor policies have higher variance, however, it is typically assumed that this is due to an increased number of low-performance episodes. Instead, we show that this is also due to a *higher maximum performance*, which consistently high-performing policies fail to achieve. This is a presents challenges for bandit sampling, causing bandits to initially *prefer* distractor policies, as shown in Appendix D.\n\n> 5. Could the authors include these hyperparameter ranges in the appendix? Given that there are many algorithms evaluated, it would be a bit tedious for a reader to track down these values across many papers. This would also help answer one of my questions posed in weaknesses section.\n\nWe use ranges from each algorithms's original paper, as described in line 175.\nOur algorithm's hyperparameter ranges can be found in appendix H. This presents in the unified space of hyperparameters, allowing for direct comparison of hyperparameters across methods, a feature missing from previous work.\n\n> 6. I'm a bit confused; how does pre-deployment interaction carry no cost? And what does it mean for performance to \"include\" the return during addition interaction?\n\nThe performance of an agent before deployment does not affect the final evaluation metric. For example, you might test a few policies on a robot, while only measuring the performance of the best one (which you deploy).\nWe have changed wording in the camera ready version to correctly use the word \"cost\" in the context of rolling out episodes, rephrasing the relevant sentence."}, {"review_id": "Mv9PtgNIAw", "reviewer": "Reviewer_rtSN", "summary": "This work focuses on two primary problems in offline RL: Ambiguous Problem Setting, and Opaque Algorithmic Design. To address these problems, this work introduces a taxonomy of existing offline RL and proposes a unified high-quality implementation of popular offline RL methods. In addition, a combination of algorithmic components is implemented with flexible configuration, called Unifloral, out of which two new methods (TD3-AWR and MoBRAC) are demonstrated to perform well on selected MuJoCo and Adroit tasks from the D4RL suite.", "questions": "1. How to understand the “no cost” in “either through pre-deployment interaction, which incurs no cost and occurs before evaluation”? Are the interactions with the environment not counted as a type of cost?\n2. How the authors think about the empirical evaluation for zero-shot offline RL (and Offline RL with post-deployment online policy selection)?\n3. Will decision transformer and its follow-up methods be scheduled to add to the code base?", "limitations": "The limitations are discussed and acknowledged in the checklist.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "###Strengthes:\n\n- The revisiting presented in this paper with a new taxonomy and a unified code implementation will be beneficial to RL community.\n- The proposed unified code implementation shows an appealing speed-up compared with CORL. The combination implementation Unifloral will be a useful code base for offline RL practitioners to develop upon.\n- The simulated pre-deployment policy selection under a given budget offers an effective approach to evaluate offline RL methods in this setting.\n- The evaluation results in Figure 3 provide a useful reference. The proposed two new methods are valuable to practitioners.\n\n&nbsp;\n\n###Weaknesses:\n\n- After reading the paper and re-checking the sentences, I am still a bit confused about the difference between pre-deployment interaction and post-deployment adaptation.\n    - In Line 105-108, “some methods relax this strict separation by allowing limited interaction with the environment—either through pre-deployment interaction, which incurs no cost and occurs before evaluation, or through post-deployment adaptation, where performance includes the return during these additional interactions.” It seems that the difference is whether the return (i.e., evaluation) is included. But I think it is not the case after reading the following content of the paper. In addition, how to understand the “no cost” in the sentence above?\n    - And I found it seems contradictory between “either through pre-deployment interaction, which incurs no cost and occurs before evaluation” (Line 106) and  “Train a set of policies offline, select the best policy based on $N$ online evaluations before deployment” (2a. of the taxonomy). I guess the latter should be the correct one. And I think this confusing expression should be clarified.\n- As to the taxonomy,\n    - It is also not clear and confusing when reading “(2a.) … select the best policy based on $N$ online evaluations before deployment” and “(2b.) … then deploy one of them. After deployment, select which policy to use based on online performance.” I have no idea about what the exact principle to separate the two categories.\n    - Is the “Limited pre-deployment interaction” necessary for Offline-to-Offline RL?\n    - Moreover, I suggest the authors make a table to present existing offline RL methods by using this taxonomy.\n- I found the “fixed sampling range of each hyperparameter” in “A Definition of Offline RL Methods” somewhat problematic, as the method that uses different ranges can be converted to use a large and fixed range, which seems to fit this definition well.\n- For the evaluation and experiments,\n    - It seems that only the 2a. in the taxonomy is evaluated for existing methods. I think the pre-deployment selection (with online evaluations) is not necessarily practically feasible in many offline RL problems (This setting suits in-context learning more). Therefore, zero-shot evaluation is still an important category.\n    - Line 136, the authors mentioned, “We measure this budget in terms of the number of evaluation episodes”, however, episodes can have different lengths at least in general (i.e., number of timesteps). This does not seem to be a proper measure in this sense. I think this is fine since the considered environments have a pre-defined fixed horizon.\n\n&nbsp;\n\n###Minors\n\n- There are inconsistent formats in reference, e.g., arXiv papers.", "was_revised": true, "final_justification": "I've read the authors' responses and the other reviewers' comments.\n\nAfter rethinking the contribution of this paper, I think the taxonomy of evaluation protocols and the unified code implementation provided by this work will be useful to the audience in DRL community, especially for those who are interested in offline RL.\n\nI increased my rating accordingly.", "rebuttal": "We thank reviewer rtSN for their questions and criticism of our paper, and we would like to clarify on their remarks. \n\n## Weaknesses\n\n> After reading the paper and re-checking the sentences, I am still a bit confused about the difference between pre-deployment interaction and post-deployment adaptation.\n\nRecall that we use *deployment* as the point in time at which we stop training and start evaluating our agent. In offline RL, an agent has no access to the environment before deployment. However, some methods relax this strict separation, allowing one (or both) of:\n\n- *pre-deployment interaction*, in which the agent may interact with the environment prior to deployment -- for instance, to select one of several policies to deploy based on online performance. Hyperparameter tuning needs pre-deployment interaction, since policies' returns are used to sample new hyperparameters to test, but the final performance metric is the performance *after* tuning, i.e. after deployment. \n\n- *post-deployment adaptation* allows the agent to continue learning after deployment, and the performance metric includes all returns collected after deployment. For example, when training a recommender system offline and finetuning it with live user data. \n\nWe believe that this clarification should address most of your doubts, and have improved the wording in the paper accordingly (including removal of the overloaded word \"cost\" mentioned in line 106 of the paper).\n\n> As to the taxonomy, \\[...\\] I have no idea about what the exact principle to separate the two categories \\[(2a) and (2b)\\].\n\nThe answer above should make the difference between categories (2a) and (2b) of the taxonomy clear: (2a) allows for pre-deployment interaction only; (2b) allows for post-deployment adaptation only. The difference is if returns achieved are measured during or after training, i.e. before or after deployment.\n\n> Is the \"Limited pre-deployment interaction\" necessary for Offline-to-Offline RL?\n\nTo answer your question, could you elaborate on what you mean by offline-to-offline RL?\n\n> I found the \"fixed sampling range of each hyperparameter\" in \"A Definition of Offline RL Methods\" somewhat problematic, as the method that uses different ranges can be converted to use a large and fixed range, which seems to fit this definition well.\n\nWe agree that specifying different hyperparameter ranges for different environments, especially without justification, is problematic. In fact, we explicitly criticise this in line 123 of our paper. Instead, the hyperparameters are part of a method, which should not be environment-specific. \nFor fair comparison between methods, we do convert methods that specify multiple ranges into one that specifies one range by taking the ranges' union (line 175).\n\n> For the evaluation and experiments, \\[...\\] zero-shot evaluation is still an important category.\n\nWe note that you can easily compare zero-shot performance by looking at the first datapoint in each of our evaluation plots, as it represents performance after 0 episodes of pre-deployment policy selection.\n\n> episodes can have different lengths at least in general. This does not seem to be a proper measure in this sense. \n\nWe argue that both episodes and steps are valid units for budget, both with advantages and drawbacks:\n- Performance in RL is usually measured as episodic return, which makes episodes a natural fit.\n- In practice the cost for evaluating robotics applications is usually dominated by *resetting* the environment (think - placing objects or the robot itself back to their initial positions), which is done once per episode.\n- When setting a budget in steps, one might observe a different number of episodes between policies, and has to compare performances with different confidence levels.\n\n## Answers to Questions\n\n> How the authors think about the empirical evaluation for zero-shot offline RL (and Offline RL with post-deployment online policy selection)?\n\nAs mentioned above, zero-shot evaluation is a special case of our protocol, namely zero pre-deployment interaction episodes.\nWe believe that our method can be extended to offline RL with post-deployment by modifying the policy selection via a bandit, but this was beyond the scope of our paper.\n\n> Will decision transformer and its follow-up methods be scheduled to add to the code base?\n\nWe have already implemented the decision transformer and present some preliminary results as D4RL normalized scores on three seeds each:\n\n| Dataset | Mean | Stdev |\n| --- | --- | ---  |\n| Hopper-medium-v2  | 5.5 (unstable, max 59.6) | 16.3 |\n| Hopper-medium-expert-v2  | 10.9 (unstable, max 111.05) | 33.1 |\n| Halfcheetah-medium-v2  | 42.4 | 0.5 |\n| Halfcheetah-medium-expert-v2  | 74.6 | 23.1 |\n| Walker2d-medium-expert-v2  | 83.4 | 15.1 |\n\nThank you for your valuable feedback. Given our response and clarifications to your questions and concerns, as well as the new preliminary results, we kindly ask you to consider increasing your score."}, {"review_id": "tIpbXXn8Gh", "reviewer": "Reviewer_7aG4", "summary": "This paper tackles two core challenges in offline RL: ambiguous problem definitions and opaque algorithmic implementations. The authors propose a rigorous taxonomy of offline-RL variants and a transparent evaluation procedure that quantifies online tuning budgets via a simulated bandit over collected policy scores. Second, they deliver minimal, single-file JAX implementations of a broad spectrum of model-free and model-based offline-RL methods. Building on these foundations, they introduce a unified framework named Unifloral that unify the hyperparameter space across different algorithms and provide modular implementation of existing Offline-RL methods. Based on this framework, the authors instantiate two new algorithms, TD3-AWR (model-free) and MoBRAC (model-based), and achieve great performance.", "questions": "I don't quite understand the difference between pre-deployment and post-deployment Offline-RL (2a and 2b). They both have online evaluations and then policy selection, right? If so, what is the difference? It would be great to have examples to in the supplementary material to give more detailed explanation.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 2, "strengths_and_weaknesses": "Strengths:\n - The explicit taxonomy and simulated-bandit evlaulation expose hidden online tuning costs, enabling fair comparisons across Offline-RL algorithms\n - The single-file JAX reimplementations alleviate the discrepancies among algorithms, providing a unified way for follow-up researchers to perform fair and meaningful baseline comparisons.\n - The novel algorithms, TD3-AWR and MoBRAC show significant gains in model-free and model-based settings.\n\nWeakness:\n - The impact of individual Unifloral components (e.g., different critic objective terms, ensemble sizes, entropy weights) on algorithm performancne is not clear in this paper. I suggest the authors make systematical analysis on these components which may bring more insights.\n - It lacks analysis of the hyperparameter space itself. It would be great to have visualizations of the hyperparameter landscape, e.g., sensitivity heatmaps to show the component interactions.", "was_revised": true, "final_justification": "I'm generally satisfied with the paper. The discussion of pre-deployment interaction and post-deployment adaptation in response to reviewer rtSN is comprehensive. While the authors did not include a hyperparameter landscape analysis, I accept their explanation that this is not a standard practice in the field. That said, I believe a more comprehensive analysis of hyperparameter impacts could substantially strengthen the work, as also noted by other reviewers. Overall, I maintain my original score.", "rebuttal": "We thank the reviewer appreciating both our novel algorithms, the implementation and taxonomy. \n\n## Weaknesses\n\n> The impact of individual Unifloral components (e.g., different critic objective terms, ensemble sizes, entropy weights) on algorithm performancne is not clear in this paper. I suggest the authors make systematical analysis on these components which may bring more insights. \n\nWe believe that the strong empirical performance of our algorithms demonstrates a clear benefit of combining components that have been independently proposed. Additionally, the algorithms Unifloral is based on represent ablations of Unifloral itself. For example, ReBRAC is an ablation of TD3-AWR (no Adwantage Weighted Regression), and IQL is an ablation of TD3-AWR (basic BC loss instead of TD3-BC value loss). Therefore figure 6 and 7 explicitly analyse the impact of individual Unifloral components.\n\n> It lacks analysis of the hyperparameter space itself. It would be great to have visualizations of the hyperparameter landscape, e.g., sensitivity heatmaps to show the component interactions.\n\nWe decided to focus our limited experiment budget on a more comprehensive and complete comparison of methods. Analyses of hyperparameter landscapes are not standard in the field, and we are not aware that any paper we cite or implement visualises sensitivity heatmaps. Therefore, we respectfully disagree that this is a weakness of the paper.\n\n\n\n## Questions\n\n> I don't quite understand the difference between pre-deployment and post-deployment Offline-RL \n\nThe difference is if returns achieved are measured during or after training, i.e. before (as in 2a) or after (as in 2b) deployment.\n\nWhen tuning hyperparameters, one may only be interested in the performance after tuning. Evaluating policies during tuning is pre-deployment interaction, before the final policy is deployed and evaluated.\n\nWhen training a recommender system, one deploys after offline training, and starts evaluating. After deployment, one finetunes the system using live user data, requiring post-deployment adaptation."}, {"review_id": "cZubzcy8aM", "reviewer": "Reviewer_eMD6", "summary": "The paper identifies a key challenge in offline RL research, the reliance on online evaluations and hyperparameter tuning, which undermines the premise of offline learning. The authors argue that many state-of-the-art algorithms function as black boxes due to their opaque design. To address this, they provide clean, single-file implementations of several well-known offline RL algorithms, offering significant speed-ups over existing libraries. Additionally, they propose two novel algorithmic paradigms—TD3-AWR and MoBRAC—and provide extensive empirical evaluations.", "questions": "1. The authors attribute speed-ups relative to CORL to their use of JAX. Could they elaborate on why their implementation outperforms JAX-CORL? Also, could they provide a comparison with d3rlpy, another widely used offline RL library?\n2. In Taxonomy point 2b, if only one policy is eventually deployed, how is the selection carried out based solely on evaluation? \n3. Which hyperparameter selections are the authors targetting through online finetuning?  What is the range of these hyperparamters? Since hyperparameters often have continuous domains, this could result in a prohibitively large search space. A table of algorithms with hyperparameters, ranges and online finetuning steps and best hyperparameter found will be helpful for the community.\n4. How many iterations does the bandit algorithm require for convergence in practice? Can convergence be quantified or benchmarked?\n5. Why is the Decision Transformer (DT) not included in Figure 3? Also how does MoBRAC handle the overfiiting observed by model based RL since it uses MOPO to learn the world dynamics.\n6. Why does MoBRAC exhibit such high variance in the AntMaze task (Fig. 15)? Is this due to instability in model-based rollouts or something else?\n7. How sensitive are the results to random seeds or initialization?\n\nPlease also refer to the weaknesses.", "limitations": "Yes discussed.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Weaknesses\n\n\n1. The evaluation is limited to primarily robotic benchmarks. It would be beneficial to include results on additional datasets like those in neorl/neorl2  or other diverse domains [4].\n2. The taxonomy omits a class of techniques that tune offline RL algorithms based on Q-values without requiring online interaction. Please consider referencing [1].\n3. Minor: The notation in Equation 1 is slightly inconsistent. T is both used a transition probability and as a time subscript in Eq 1. Also in Eq 1 s1:H∼H should this be s1:H∼T as states are generated from transition dynamics?\n4. Several recent offline RL methods, such as Sparse Q-Learning (SQL) and Exponential Q-Learning (EQL) [2], as well as A2PR [3], are not discussed or compared.\n\n\nStrengths\n\n\n1. The provision of clean implementations and a unified benchmarking framework is an impactful contribution to the offline RL community, promoting transparency and reproducibility.\n2. The empirical evaluations are thorough and compelling. I particularly appreciated the finding that “no algorithm consistently performs well across all datasets,” and the revelation of overfitting in model-based algorithms such as MOPO, MOReL, and COMBO.\n3. Two new algorithm designs TD3-AWR and MoBRAC—that leverage the unified framework to demonstrate superior performance across various benchmarks.\n\n\n[1] Kumar, Aviral, et al. \"A workflow for offline model-free robotic reinforcement learning.\" arXiv preprint arXiv:2109.10813 (2021).\n[2] Xu, Haoran, et al. \"Offline rl with no ood actions: In-sample learning via implicit value regularization.\" arXiv preprint arXiv:2303.15810 (2023).\n[3] Liu, Tenglong, et al. \"Adaptive advantage-guided policy regularization for offline reinforcement learning.\" arXiv preprint arXiv:2405.19909 (2024).\n[4] Additional Benchmarks : https://github.com/polixir/NeoRL , https://github.com/polixir/NeoRL2, RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learning", "was_revised": true, "final_justification": "A clean offline RL library is useful for the community. The authors addressed some of my concerns in the rebuttal.", "rebuttal": "Thank you for your recognition of our extensive empirical studies and constructive criticism of our work.\n\n## Weaknesses\n\n> It would be beneficial to include results on additional datasets like those in neorl/neorl2 or other diverse domains.\n\nWe believe that our robotics benchmarks already capture significant diversity: Across 7 environments and 21 datasets, we include locomotion, manipulation and maze solving. We also evaluate our methods on *all datasets* of the three most common mujoco environments.\nWe also agree that neorl2 is an interesting benchmark and are currently looking into adding it for the camera-ready version.\n\n> The taxonomy omits a class of techniques that tune offline RL algorithms based on Q-values without requiring online interaction.\n\nThese methods fall under our taxonomie's category (1), since (as the reviewer noted) they use no online interaction before deployment. We will cite the reviewer's suggested reference in the next iteration of our paper.\nFurthermore, we already cite two references that estimate policy performance offline (line 317) and elaborate in more detail in section L.1 of the Extended Related Work in the Appendix.\n\n> Minor: The notation in Equation 1 is slightly inconsistent.\n\nWe thank the reviewer for their keen eye and have fixed this inconsistency.\n\n> Several recent offline RL methods, such as Sparse Q-Learning (SQL) and Exponential Q-Learning (EQL) [2], as well as A2PR [3], are not discussed or compared.\n\nWe are citing 21 RL methods and implementing 10 algorithms as well as 2 novel algorithms. This includes all offline-only algorithms in (Jax-)CORL and almost all algorithms in OfflineRLKit. We aimed to provide one of the most comprehensive offline RL libraries that currently exist, and argue that we have succeeded.\nStill, we are open to implementing the suggested algorithms and are planning to add them for the camera-ready version. Our work is also intended to be a scaffolding for the community to seemlessly add concise implementation of different methods.\n\n## Answers to Questions\n\n> Could \\[the authors\\] elaborate on why their implementation outperforms JAX-CORL?\n\nProfiling the runtime of jax-corl was outside the scope of our work. From looking at their codebase, it seems like they are not using `lax`-based (GPU-compilable) control flow, which can lead to significant performance decreases\n\n> In Taxonomy point 2b, if only one policy is eventually deployed, how is the selection carried out based solely on evaluation?\n\nThis depends on the specific offline RL method. For example, Active Offline Policy Selection (Konyushkova et al., 2022, [https://arxiv.org/abs/2106.10251XiV](https://arxiv.org/abs/2106.10251)) uses Bayesian optimization over a Gaussian process of policy values. We elaborate in Appendix L.1 (line 960).\n\n> Which hyperparameter selections are the authors targetting through online finetuning? What is the range of these hyperparamters?\n\nWhen selecting policies via a bandit, we select between policies trained with different hyperparameter settings. We use the hyperparameter ranges from the respective methods' original papers, as described in line 175. \nOur algorithm's hyperparameter ranges can be found in Appendix H. \n\n> How many iterations does the bandit algorithm require for convergence in practice? Can convergence be quantified or benchmarked?\n\nThis is highly dependent on the HP robustness of the evaluated algorithm. For example, if the algorithm produces high-performing policies for a wide range of hyperparameter settings, convergence is fast.\n\n> Why is the Decision Transformer (DT) not included in Figure 3? \n\nWe include it in the camera-ready version and present results below. We did not include it in the initial set of algorithms as the Decision Transformer is not a traditional RL algorithm, even though it is technically trained offline on the same datasets. Furthermore, as done in CORL [1] and the original DT paper [2], the sequential nature of the evaluation function is different from the other algorithms. We will report the results from the same DT evaluation setup used by CORL [1]. \n\n#### Preliminary DT results\n\nWe ran our implementation of the DT algorithm for three seeds each. \n\n| Dataset | Mean | Stdev |\n| --- | --- | ---  |\n| Hopper-medium-v2  | 5.5 (unstable, max 59.6) | 16.3 |\n| Hopper-medium-expert-v2  | 10.9 (unstable, max 111.05) | 33.1 |\n| Halfcheetah-medium-v2  | 42.4 | 0.5 |\n| Halfcheetah-medium-expert-v2  | 74.6 | 23.1 |\n| Walker2d-medium-expert-v2  | 83.4 | 15.1 |\n\n\n> Also how does MoBRAC handle the overfiiting observed by model based RL since it uses MOPO to learn the world dynamics.\n\nMoBRAC uses the *trained* MOPO models as they are. We propose the algorithm as an improvement over current approaches and demonstrate it by intentionally using the same models. MoBRAC is an ablation in itself, showcasing that behavior regularization leads to agents that are more robust, despite being trained using samples from somewhat flawed dynamics models. \n\n> Why does MoBRAC exhibit such high variance in the AntMaze task (Fig. 15)?\n\nThe variance is very low in practice, but might appear high due to the extremely small scale of the y axis. Most likely, a few policies have recieved a reward by sheer luck, hence the performance is ever-so-slightly above zero.\n\n> How sensitive are the results to random seeds or initialization?\n\nWe did not explicitly investigate this. However, every hyperparameter sample also has a different randomly sampled seed, so our metric evaluates an even stronger form of robustness (which includes hyperparameters).\n\n[1] Tarasov et al. *CORL: Research-oriented Deep Offline Reinforcement Learning Library* (2022)\n\n[2] Chet et al. *Decision Transformer: Reinforcement Learning via Sequence Modeling* (2021)\n\nThank you for your valuable feedback. Given our responses and clarifications to your questions and weaknesses, we kindly ask you to consider increasing your score."}]

Author rebuttal:

Meta-review: This work makes a contribution to the research practice in offline RL by identifying several problematic issues across empirical offline RL papers: ambiguous problem definitions, inconsistent algorithm designs and implementations, and weak, biased evaluations. The paper addresses these by introducing a clean taxonomy, evaluation protocol, and, crucially, implementations of a variety of RL algorithms, as well as a unifying algorithmic framework, Unifloral. The paper also presents two algorithms within this framework that outperform the alternatives.

The metareviewer and the reviewers believe that this is an elegant work that does a lot for making offline RL research results more reliable and comparable to each other, which promises to have a major positive impact on the field.
Decision: Accept (oral)