A Minimalist Approach to Offline Reinforcement Learning
Authors: Scott Fujimoto
Shixiang Gu
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: Deep Reinforcement Learning
Offline Reinforcement Learning

Abstract: Offline reinforcement learning (RL) defines the task of learning from a fixed batch of data. Due to errors in value estimation from out-of-distribution actions, most offline RL algorithms take the approach of constraining or regularizing the policy with the actions contained in the dataset. Built on pre-existing RL algorithms, modifications to make an RL algorithm work offline comes at the cost of additional complexity. Offline RL algorithms introduce new hyperparameters and often leverage secondary components such as generative models, while adjusting the underlying RL algorithm. In this paper we aim to make a deep RL algorithm work while making minimal changes. We find that we can match the performance of state-of-the-art offline RL algorithms by simply adding a behavior cloning term to the policy update of an online RL algorithm and normalizing the data. The resulting algorithm is a simple to implement and tune baseline, while more than halving the overall run time by removing the additional computational overheads of previous methods. 

A Minimalist Approach to Offline Reinforcement Learning
Scott Fujimoto 1.2
Shixiang Shane Gu²
Mila,, McGill University "Gooll Research, Brain Team
scott fujimott@mailll www.:momooomaammmlllll..oo
Abstract
Offline reinforcement learning (RL) defines the task of learning from fixed batch data. Due to errors value estimation from out-of-distribution actions, most offfine RL algorithms take the approach of constraining or regularizing the policy with the actions contained in the dataset. Built on pre-existing RL algorithms, modifications to make RL algorithm work offline comes the cost of additional complexity. Offline RL algorithms introduce new hyperparameters and often leverage secondary components such generative models, while adjusting the underlying RI. algorithm. In this paper we aim to make a deep RI. algorithm work while making minimal changes. We find that state-of-the-arr offline RL. algorithms by simply adding a behavior cloning term the policy update of an online RL algorithm and normalizing the data. The resulting algorithm is simple to implement and tune baseline, while more than halving the overall run time by removing the additional computational overheads of previous methods.
match the performance of
Introduction
Traditionally, reinforcement learning (RL) is thought of as a paradigm for online learning, where the interaction between the RL agent and its environment is of fundamental concern for how the agent learns. In offline RL (historically known as batch RL), the agent learns from a fixed-sized dataset, collected by some arbitrary and possibly unknown process (Lange al.) 2012) Eliminating the need to interact with the environment is noteworthy as data collection can often be expensive, risky, or otherwise challenging, particularly in real-world applications. Consequently, offline RL enables the use of previously logged data or leveraging an expert, such as human operator, without any of the risk associated with an untrained RL agent.
Unfortunately, the main benefit of offline RL, the lack of environment interaction, is also what makes it challenging task. While most oof-policy RL algorithms applicable in the offline setting, they tend to under-perfone due to "extrapolation error": an error in policy evaluation, where agents tend to poorly estimate the value of state-action pairs not contained in the dataset. This in turn affects policy improvement, where agents learn to prefer value has been overestimated, resulting in poor performance [Fujimoto al. 2019b]. The solution class for this problem revolves around the idea that the learned policy should d be kept close to the data-gem satuageneraaing process (or behavior policy), and has been given variety of names (such batch- constrained (Fujimoto et al. 2019b), KL-control [Jaques et al. 2019)] behavior-regularize
actions whose
[Wu et al.) 2019), or policy constraint Levine al. 2020 depending on how this "closeness" is chosen be implemented.
While there are many proposed approaches to offline RL, remark that few truly "simple", and even the algorithms which claim to work with minor additions to an underlying online RL
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

algorithm make a significant number of implementatiio--lvel adjustments. In other cases, there are unmentioned hyperparameters, or secondary components, such as generative models, which make offline RL algorithms difficult to reproduce, and even more challenging to tune. Additionally, such mixture of details slow down the run times of the algorithms, and make causal attributions of performance gains and transfers of techniques across algorithms difficult, as in the case for many online RL algorithms [Henderson et al. 2017 Tucker et al. 2018, [ngstrom et al. 20200] Andrychowicz et al. 2021 Furuta et al. 2021 This motivates the need for more minimalist approaches in offline RL.
In this paper, we ask: can we make deep RL algorithm work offtine with minimal changes? We find that we can match the performance of saate-of-the-arsd art offline RL algorithms with single adjustment to the policy update step of the TD3 algorithm [Fujimoto et al. 2018). TD3's policy π is updated with the deterministic policy gradient (Silver et al. 2014]:
argmax ((s,))=(Q((s, (()))
(1)
Our proposed change, TD3+BC, is simply add behavior cloning term to regularize the policy:
argmax (s))) [AQ(s, (()) (T(() - =a)
(2)
with single hyperparameter À to control the strength of the regularizer. This modification can be made by adjusting only a single line of code. Additionally, we remark that normalizing the states the dataset, such that they have mean 0 and standard deviation 1, improves the stability of the learned policy. Importantly, these are the only changes made to the underlying deep RL algorithm. To accommodate reproduciblity, all of our code is open-sourced
We evaluate our minimal changes to the TD3 algorithm on the D4RL benchmark of continuous control tasks [Fu et al. 2020]. We find that our algorithm compares favorably against many offline RL algorithms, while being significantly easier to implement and more than halving the required computation cost. The surprising effectiveness of our minimalist approach suggests that in the context of offline RI, simpler approaches have been left underexplored in favor of more elaborate algorithmic contributions.
2 Related Work
Although to the best of our knowledge, we are the first to use TD3 with behavior cloning (BC) for the purpose of offline RI., remark that combining RL with BC, and other imitation learning approaches, has been previously considered by many authors.
RL BC. With the aim of accelerating reinforcement learning from examples (known learning from demonstrations (Atkeson and Schaal 1997), BC has been used as regularization for policy
optimization with DDPG [Lillicrap et al. 2015 Nair et al.) 2018 Goecks et al. 2020) and the natural policy gradient [Kakade] 2001| Rajeswaran et al. [017]. but with additional sophistication through modified replay buffers and pre-trainin. The most similar work to own is SAC+BC
baseline (Haarnoja et af. 2018 from Nair et al. (020) and an unpublished course project [Booher combining PPO (Schulman al. 2017] with BC.
RL Imitation. Other than directly using BC with RL, imitation learning has been combined with RL in variety of manners, such mixed with adversarial methods (Zhu et al. 2018 Kang et al. 2018). used for pre-training [Pfeiffer et al., 20188. modifying the replay buffer Večerík et al. 2017 reward shaping Judah et al. 2014 Wu et al. 2021]. In all cases, these methods use demonstrations as a method for overcoming challenges in exploration or improving the learning speed of the RL agent.
Gulcehre et al. 2020), adjusting the value function [Kim et al. 2013] Hester et al.) 2017]
Offline RL. As aforementioned. offline RL methods generally rely on some approach for "staying close" to the data. This may be implemented using an estimate of the behavior policy and then
defining an explicit policy parameterization (Fujimoto et al., 201 9b] Ghasemipour et al.) (0200) or by using divergence regularization (aques et al. | Kumar et al. 2019 Wu et al.) 2019 Siegel et al.)
2020 Guo et al.) 1021 Kostrikov et al.| 202 Other approaches use a weighted version of BC to
favor high advantage actions [Wang al. 2018 Peng et al. 2019 Siegel et al. 2020 Wang et al.
2

---

2020 Nair et al. 2020 or perform BC over an explicit subset of the data [Chen et al. 2020]. Some methods have modified the set of valid actions based on counts [Laroche et al., 2012 9 or the learned
behavior policy [Fujimoto et al.) 2019a]. Another direction is to implement divergence regularization as a form of pessimism into the value estimate (Nachum et al. 2019 Kumar et al.) 2020 Buckman et al.
2020].
Meta Analyses of RL Algorithms. There are a substantial amount of meta analysis works on online RL algorithms. While some focus on inadequacies in the experimental protocols [Henderson
al. 2017 Osband et al.,
2019]. others study the roles of subtle implementation details in algo-
rithms [Tucker et al., 2018 Engstrom et al. [2020, Andrychowicz et al.) [021] Furuta et al. 2021]. 2018). Engstrom et al. [2000] identified that superior performances of certain algorithms were more dependent on, or even accidentally due to, minor implementation rather than algorithmic differences. Furuta et al. [2021] study two broad families of off-policy algorithms, which most offline algorithms are based on, and find that few subtle implementation details are strongly co-adapted and critical to specific algorithms, making attributions of performance gains difficult. Recent offline research also follows a similar trend, where number of implementation modifications are necessary for high algorithmic performances (see Table I). In contrast, we derive our algorithm by modifying the existing TD3 with only a few lines of codes. Our results suggest the community could also learn from careful explorations of simpler alternatives, besides emphasizing algorithmic novelties and complexities.
For example, Tucker al.)
3 Background
RL. Reinforcement learning (RL) is a framework aimed to deal with tasks of sequential nature. Typi- cally, the problem is defined by Markov decision process (MDP) (8, A, R. 7), with state space S. action space A, scalar reward function R, transition dynamics p. and discount factor y [Sutton] and Barto [998]. The behavior of an RL agent is determined by policy x which maps states to actions (deterministic policy), or states to a probability distribution over actions (stochastic policy). The objective of an RL agent to maximize the expected discounted return E, <>> r'lllal- which is the expected cumulative sum of rewards when following the policy in the MDP, where the importance of the horizon is determined by the discount factor. We measure this objective by value function, which measures the expected discounted return after taking the action in state s: Q (s. a) = E, E 7're+1 = 8, ag a).
BC. Another approach for training policies is through imitation of an expert or behavior policy. Behavior cloning (BC) is an approach for imitation learning (Pomerleau 1991), where the policy is trained with supervised learning to directly imitate the actions of provided dataset. Unlike RL, this process is highly dependent on the performance of the data-collecting process.
Offline RL. Offline RL breaks the assumption that the agent can interact with the environment. Instead, the agent is provided a fixed dataset which has been collected by some unknown data- generating process (such as a collection of behavior policies). This setting may be considered more challenging as the agent loses the opportunity explore the MDP according to its current beliefs, and instead from infer good behavior from only the provided data.
One challenge for offline RL is the problem of extrapolation error (Fujimoto et al., 2019b). which is generalization error the approximate value function, induced by selecting actions not contained in the dataset. Simply put, it is difficult to evaluate the expected value of a policy which is sufficienlly different from the behavior policy. Consequently, algorithms have taken the approach of constraining regularizing the policy to stay near to the actions in the dataset [Levine et al. 2020
4 Challenges in Offline RL
In this section, we identify key open challenges in offline RL through analyzing and evaluating prior algorithms. We believe these challenges highlight the importance of minimalist approaches, where performance can be easily attributed to algorithmic contributions, rather than entangled with the specifies of implementation.
Implementation and Tuning Complexities. RL algorithms are notoriously difficult to implement and tune [Henderson et al. 2017 Engstrom et al. 2020 Furuta et al. 2021 where minor code-level
3

---

COL
Fisher-BRC
TD3+BC
Kumar al. 2020
Kostrikov 2021
(Ours) Add BC term'
Add regularizer to critic Approximate logsumexp with sampling*
Train a generative model
Algorithmic Adjustments
Replace critic with offset function Gradient penalty on offset function?
Architecture it
Architecture
Normalize states
Implementation Adjustments
Actor learning rate
Reward bonus"
Pre-training actor
Remove SAC entropy term
Remove SAC entropy term
Max over sampled actions"
Table Implementation changes offline algorithms make the underlying base RL algorithm. corresponds details that add additional hyperparameter(S), and corresponds ones a computational cost.
CQL
Fisher-BRC
102
Percent Difference
100
HD
HC m
HC-I..
NC
Hope
Hop-rn
Hos
Wen
We
HCC HCm HK Hone
hop
Hop
War Wart wine
Figure 1: Percent difference of performance of offline RI. algorithms and their simplified versions which remove implementation adjustments to their underlying algorithm. HC = HalfCheetah, Hop Hopper, W = Walker, random, m medium, medium-replay, medium-expert, e expert. Huge drops in performances show that the implementation complexities are crucial achieving best results in these prior algorithms.
optimizations and hyperparameters have non-trivial impact of performance and stability. This problem may be additionally amplified in the context of offline RL, where evaluating changes to implementation and hyperparameters is counter-intuitive the nature of offline RL, which explicitly aims to eliminate environment interactions [Paine et al. 2020 Yang et al. 2020
Most offline RL algorithms are built explicitly on top of an existing off-policy deep RL algorithm, such as TD3 [Fujimoto et al., 2018 or SAC (Haarnoja al. 2018), but then further modify the implementation changes, such modifications to network architecture, learning rates, or pre-training the actor network. We remark that a desirable property of an offline RL algorithm would be to minimally modify the underlying algorithm. to reduce the space of possible adjustments required to achieve a strong performance.
underlying algorithm with inmpgrrriiiiii"n"ieeenniii
In Table I we examine the particular modifications made by two recent offline RL algorithms, CQL [Kumar et al., 2020) and Fisher-BRe BRC [Kostrikov al, 202T). On top of algorithmic changes, CQL also adds pre- training phase where the actor is only trained with imitation learning and selects the max action over sampled of actions from the policy during evaluation. Fisher-BRC adds constant reward bonus to every transition. Both methods modify SAC by removing the entropy term the target update and modify the default network architecture. These changes add supplementary hyperparameters which may need to be tuned or increase computational costs.
In the online setting, these changes are relatively inconsequential and could be validated with some simple experimentation. However, in the offline setting, where we cannot interact with the environment, making additional adjustments to the underlying algorithm should be considered as more costly as validating their effectiveness is no longer a trivial additional step. This is also problematic because unlike the algorithmic changes proposed by these papers, these implementation details are not well justified, meaning there is much less intuition as to when to include these details, or how to adjust them with minimal experimentation.
the case of the D4RL benchmark on MuloCo tasks [ooorov et al. 2012 Fu et al. 2020), we have strong prior that our base deep RL. algorithm performs well, as SAC/TD3 are considered state-of-the-art (or nearly) in these domains. If additional changes necessary, then suggests the algorithmic contributions alone are insufficient.

---

BC
COL
Fisher-BRC
TD3+BC
To3
Percent Differencs
-20
HCB
HCm
HO Noo
Hap
Hoper
Fob ni
We
W
40
Hap
Win
HC
Hop-te
tope
Wint Mine Wite
H00
Figure 2: Percent difference of the worst episode during the 10 evaluation episodes at the last evaluation. This measures the deviations in performance single point in time. HC HalfCheetah, Hop Hopper. W Walker, r= random, medium, mr = medium-replay, me medium-expert, e = expert. While online algorithms (TD3) typically have small episode variances per trained policy (as they should at convergence), all offline algorithms have surprisingly high episodic variances for trained policies.
BC
COL
Fisher-BRC
BD
(onlin)
Percent Difference
-20
-40
-80
HCH
HC-0)
Hope
no
Hopp
His mi
Hop no
HCC
HD oe
Hove
Hop
Hap Tipee
wit
NCH
Hopel
HC
Figure 3: Percent difference of the worst evaluation during the last 10 evaluations. This measures the deviations in performance over a period of time. HC HalfCheetah, Hop = Hopper, W Walker, random, medium, mr medium-replay, me medium-expert expert, = expert. Similarly the result in Figure 2 all offiine-trained policies have significant variances near the final stage of training that are absent in the online setting.
In Figure we examine the percent difference in performance when removing the implementation changes in CQL and Fisher-BRC. There is a significant drop in performance in many of the tasks, This is not necessarily a death knell to these algorithms, as certainly possible these changes could be kept without tuning when attempting new datasets and domains. However, since neither paper make mention of training/validationn split, we can only assume these changes were made with their evaluation datasets in mind (D4RL, in this instance), and remark there is insufficient evidence that these changes may be universal. Ultimately, we make this point not to suggest a fundamental flaw with pre-existing algorithms, but to suggest that there should be preference for making minimal adjustments to the underlying RL algorithm, to reduce the need for hyperparameter tuning.
Extra Computation Requirement. A secondary motivating factor for minimalism is avoiding the additional computational costs associated with modifying the underlying algorithm (in particular architecture) and more complex algorithmic ideas. In Table 3 (contained later in (Section 6), we examine the run time of offline RL algorithms, as well as the change in cost over the underlying algorithm, and find their is significant computational cost associated with these modifications. On top of the architecture changes, for CQL this is largely due to the costs of logsumexp over multiple sampled actions, and for Fisher-BRC, the costs associated with training the independent generative model. Since an objective of offline RL to take advantage of existing, potentially large, datasets, there should be preference for scalable and efficient solutions. Of course, run time should not come at the cost of performance, but as we will later demonstrate, there exists a simple, and computationally-ficet approach for offline RL which matches the performance of current state-of-the-an algorithms.
Instability of Trained Policies. In analyzing the final trained policies of prior offline algorithms, we learned of tangential, and open, challenge in the form of instability. In online RL, the current policy is unsatisfactory, we can use checkpoints of previous iterations of the policy, or to simply continue training. However, offline RL, the evaluation should only occur once by definition, greatly increasing the importance of the single policy at evaluation time. We highlight two versions of instability in offiine RI. Figure 2 shows that in contrast to the online-trained policy, which converges to robust low-variance policy, the offlinee- policy exhibits huge variances in performance
5

---

during single evaluation. Therefore, even if the average performance is reasonable. the agent may still perform poorly on some episodes. Figure shoows instability over the set of evaluations, which means the performance of the agent may be dependent on the specific stopping point chosen for evaluation. This questions the empirical effectiveness of offline RI. for safety-critical real-world use cases [Mandel et al.) [0144] Gottesman et al. 20188] Gauci et al. 2018 Jaques et al.) 22019) Matsushimal ctt al., 2020] as well as the current trend of reporting only the mean-value of the final policy in offline benchmarking [uu et al. 2020].
episodes.Fiiuure
Such variances are not likely caused by high policy entropies (e.g. TD3+BC trains a deterministic policy), but rather our hypothesis is that such problem is due to distributional shifts issues and poor generalizations across unobserved states caused by offline nature of training [Ross et al.) 2011]. where the optimized policy is never allowed to execute in the environment, similarly as in BC. This trait of offline RI. algorithms appears to be consistent across all offline algorithms we evaluated, even for our minimalistic TD3+BC that is only a few lines change from TD3. While we could not solve this challenge sufficiently within the scope of this work, the fact that this reproducible even in the minimalistic variant proves that this fundamental problem shared by all offline training settings, and is a critical problem for the community to study in the future.
5 A Minimalist Offline RL Algorithm
key problem in offline RL, extrapolation error, can be summarized as the inability to properly evaluate distriiiittibttionnn actions. Consequently, there has been a variety of different approaches to limiting, or regularizing, action selection such that the learned policy is easier to evaluate with the given dataset. We remark that while minimizing say. KL divergence, is a both logical and valid approach for reducing extrapolation error, there is no fundamental argument why minimizing one divergence or distance metric should be better than another. Thus, rather than derive an entirely new approach, we focus on simplicity, and present an offline RL algorithm which requires minimal modifications to pre-existing deep RL algorithm. As discussed in (Section a minimalist approach has a variety of benefits, such as reducing the number of hyperparameters to tune, increasing scalability by reducing computational costs, and providing an avenue for analyzing problems by disentangling algorithmic contributions from implementation details.
We now describe such approach to offline RL. Our algorithm builds top of TD3 FFjjiotoo et al. 2018), making only two straightforward changes. Firstly, we add a behavior cloning regularization term to the standard policy update step of TD3, to push the policy towards favoring actions contained in the dataset D:
= argmax EDED [Q(s, (s))]] + π argmax E (s at
A ((ss - (Λ(s) - -a)²
(3)
Secondly. we normalize the features of every state in the provided dataset. Let be the ith feature of the state s, let μı ag be the mean and standard deviation, respectively, of the ith feature across the dataset:
H4
(4)
O, +
where e is a small normalization constant (we use 10 10-3). While we remark this is a commonly used implementation detail in many deep RL algorithms [Raffin et al. 2019). we highlight it as (1) we want complete transparency about all implementation changes and (2) normalizing provides a non-trivial performance benefit in offline RL, where it is particularly well-suited the dataset remains fixed.
While the choice of X in Equation (3)is ultimately just a hyperparameter, we observe that the balance between RL (in maximizing Q) and imitation (in minimizing the BC term), is highly susceptible to the scale of Q. If we assume an action range of [-1, 1]. the BC term is at most 4, however the range of Q will be a function of the scale of the reward. Consequently, we can add a normalization term into A. Given the dataset of N transitions (s,, as). we define the scalar as:
a
A Σ
(5)
(()aa) a.)
N
This is simply normalization term based on the average absolute value of Q. In practice, we estimate this mean term over mini batche, rather than the entire dataset. Although this term includes Q, it is

---

BC
BRAC.p
AWAC
CQL
Fisher-BRe
TD3+BC
Random
HalfCheetah
2.0 +0.1
23.5
2.2
21.7 +0,9
32.2 +2.2
10.2 +
Hopper
9.5 10.1
11.1
9.6
10.7 L0.1
11.4 10.2
11.0 10.1
Walker2d
1.2 :0.2
0.8
5.1
2.7 11.2
0.6 10.6
1.4 11.
Medium
HalfCheetah
36.6 #:066
44.0
37.4
37.2 4:0.3
41.3 +0.5
42.8 +..3
Hopper
30.0 10.5
31.2
72.0
44.2 110.8
99.4 10.4
99.5 11.0
Walker2d
16.33
72.7
30.1
57.5 18.3
79.5 +1.0
79.7 118
Medium Replay
HalfCheetah
34.7 1188
45.6
41.9 +1.1
43.3 10.9
43.3 +0.55
Hopper
19.7 15.9
0.7
28.6 10.9
35.6 12.5
31.4 13.0
Walker2d
8.3 1.5
-0.3
15.8 +2.6
42.6 17.0
25.2 #5.1
Medium Expert
HalfCheetah
67.6 133.2
43.8
36.8
27.1 +3.9
96.1 +9.5
97.9 #4.4
Hopper
89.6 #:2.6
1.1
80.9
111.4 +1.2
90.6 +-3.3
112.2 +0.2
Walker2d
12.0 +:5.8
-0.3
42.7
68.1 1:3.1
103.6 4:4.6
101.1
+9.3
Expert
HalfChectah
105.2 1.7
3.8
78.5
82.4 +7,4
106.8 +3.0
105.7 = 1.9
Hopper
111.5 1.3
6.6
85.2
111.2 +2.1
112.3
+0.2
112.2 +0.2
Walker2d
56.0 +24.9
-0.2
57.0
103.8
#7.6
79.9
#32.4
105.7
=2.7
Total
595.3 +91.
284.1
764.3 +61.5
974.6 108.33
979.3
+33.4
Table Average normalized score over the final evaluations and seeds. highest performing scores are highlighted. CQL and Fisher-BBl BRC are re-run using author-pre uuthor-provided implementations to ensure an identical evaluation process, while BRAC and AWAC use previously reported results. + captures the standard deviation over seeds. TD3+BC achieves effectively the same performances the state-of-the-art Fisher-BRC, despite heing much simpler implement and tune and more than halving the computation
not differentiated over, and is simply used to scale the loss. This formulation has the added benefit of normalizing the learning rate across tasks, as the gradient V.Q(s, a) will also be dependent on the scale of Q. We use = 2.5 in experiments.
This completes the description of TD3+BC. The Equations (3]
summarizes the entirety of (5)
our changes to TD3, and can be implemented by modifying only handful of lines in most codebases.
6 Experiments
We evaluate proposed approach the D4RL benchmark of OpenAl gym MuJoCo tasks (Todorov
et al. 2012 Brockman et al. 2016 Fu et al. 2020]. which encompasses a variety of dataset settings state-of-thh-arr algorithms, CQL [Kumar et al. 2020 and Fisher-BRC [Kostrikov et al. 2021]. as well as BRAC [Wu et al. 2019 and AWAC [Nar al. 2020 due their algorithmic simplicity.
and domains. Our offline RL baselines include
To ensure fair and identical experimental evaluation across algorithms, we re-run the state-of-the. art algorithms CQL and Fisher-BRC using the author-providee implementation We train each algorithm for million time steps and evaluate every 5000 time steps. Each evaluation consists of 10 episodes. Results for BRAC are obtained from the D4RL benchmark and from the CQL paper, while the AWAC results are obtained directly from the paper. BC results were obtained using our own implementation.
We report the final performance results in Table and display the learning curves Figure Although our method is very simplistic in nature, it surpasses, or matches, the performance of the current satte-of-the-aee offline RI. algorithms in most tasks. Only Fisher- BRch exhibits a comparable
D4RL.

---

BC
CQL
Fisher-BRe
TD3+BC
HarChheeshh- Randomm
HarChhesah--Acce Medium
HairD hoatahoetanioo Medium
HalfCheosah Modium Excort
HattChestah Expert
Nomalized Scare
8
86
0.
92
04
00
1.0
0.5
0.2
0.4
86
00
1.0
50
02
54
85
60
0.
02
E4
06
01
0.
02
04
05
Randam
Hopour-Mediumi
Hopper-Medimn
Hoppa-Expart
Nomallzzd Score
8
8
8
02
04
9.6
00
1.5
0.5
02
24
50
40
16
59
02
54
10
0. Willar22WWedi
02
14
00
00
02 Walker25-Expert
130
Willa22dRRaddom
WAller/S-VMddum
Vakka22-MMeiu--Rgllyy
Normalized Score
8
02
04 34 08 14
02
5.4
50 08 16
68 5A
09
6a
00
09 03
04
Time Steps (1e6)
Time Steps (1e8)
Time Steps (1e8)
Time Steps (1e6)
Time Sleps (1e6)
Figure 4: Learning curves comparing the performance TD3+BC against offline RL baselines in the D4RL datasets. Curves are averaged over 5 seeds, with shaded area representing standard deviation across seeds. TD3+BC exhibits a similar learning speed and final performance as the state-of-the-art Fisher-BRC, without the need of pre-training generative model.
Run Time
4h 11m
CQL (GitHub)
Fisher-BRC (GitHub)
Fisher-BRC (Ours)
TD3+BC (Ours)
Implementation
25m
39m
15m
1s
Algorithmic
Ih 29m
33m
58m
5s
Total
11m
12m
2h 8m
39m
COL
WHWe-BRCFFooo RehenBBC TD1-3C
(Tersorfow) (yyere))
Table 3: Run time comparison of training each offline RL algorithm (does not include evaluation costs). (Left) Breakdown of the cost of the added implementation details (mainly architecture changes) and the algorithmic details by each method. (Right) Total training time of each algorithm. While CQL and Fisher- BRR have significantly increased computational costs over their base online RI. algorithm due to various added complexities (e.g. see Table I), TD3+BC has effectively no increase. This results in less than half of the computational cost of these prior state-of-the-art algorithms.
performance Examining the learning curves, we can see that our approach achieves similar learning speed and stability, without requiring any pre-training phase.
Run time. We evaluate run time of training each of the offline RL algorithms for I million time steps, using the provoodd implementations. Additionally, for fair comparison, we re-implement Fisher-BRC, allowing each method to be compared in the same framework (PyTorch [Paszke et al. 2019). The results are reported Table 3 Unsurprisingly, we find our approach compares favorably against previous methods in terms of wall-clock training time, effectively adding no cost to the underlying TD3 algorithm. All run time experiments were run with single GeForee GTX 1080 GPU and an Intel Core i7-6700K CPU at 4.00GHz.
perform ablation study over the components in method. As noted in previous work [Fujimoto et al., 2019b), without behavior cloning regularization, the RL algorithm alone insufficient achieve a high performance (except on some of the random data sets). We
Ablation. In Figure
As noted by previous papers Kostrikov et al. 2021 we remark the author-provide implementation of CQL achieves lower result than their reported results.

---

BC+Nom
(-8C)
TD3 (-Nor)
100
Porcent Difference
30
Figure 5: Percent difference of the performance of an ablation of our proposed approach, compared to the full algorithm. TD3+2+BC+Norm refers the complete algorithm, where Norm refers to the state feature normalization. HC HalfChectah, Hop = Hop=Hopper, W = Walker, r= random, me -eepium-expert. = expert. As expected, both and TD3 are necessary components to achieve a strong performance. While removing state normalization is not devastating to the performance of the algorithm, we remark it provides a boost in performance across many tasks, while being straightforward addition.
medium, mr medium-replay,
Porcem Diference
100
Figure 6: Percent difference of the performance of an ablation over compared to the full algorithm. Recall the form of the sole hyperparameter À TTQ where À weights the function during the policy update. = 2.5 is used by default. HC = HalfChectah, Hop = Hopper, W = Walker, = random, m medium, mr medium-replay, me medium-expert, remark that performance begins to degrade select tasks the algorithm begins to approach more RL (a = 1) or imitation (a = 1).
expert. While there is effectively performance difference
between a 2 and
also note that our algorithm never underperforms vanilla behavior cloning, even the expert tasks. Predictably, the removal of state normalization has the least significant impact, but it still provides some benefit across range of tasks while being minor adjustment. In Figure 6 we evaluate the sensitivity of the algorithm to the hyperparameter a, where the weighting
Ex [Q(,,o]] on the value function is determined by a. On many tasks, our approach is robust to this weighting factor, but note the performance on a subset of tasks begins to decrease as a begins to more heavily favor imitation (a 1) or RL (a 4).
Conclusion
Most recent advances in offline RL. center around the idea of regularizing policy actions to be close to the support within batch data, and yet many state-of-the-ard algorithms have significant complexities and additional modifications beyond base algorithms that lead to not only much slower time, but also intractable attributions for sources of performance gains. Instead of complexity we optimize for simplicity, and introduce minimalistic algorithm that achieves state-of-the-ane performance but is only a few lines of changes from the base TD3 algorithm, uses less than half the computation time of competing algorithms, and has only one additional hyperparameter.
Additionally, we highlight existing open challenges in offline RL research, including not only the extra implementation, computation, and Syperparameter-tuning complexities that we successfully address this work, but also call attention to the neglected problem of high episodic variance in filline-trained policies compared to online-trained (see Figures [ 3) that
the community should address in future works and benchmarking. Finally, we believe the sheer simplicity of our approach highlights possible overemphasis on algorithmic complexity made by the community, and we hope to inspire future work to revisit simpler alternatives which may have been overlooked.

---

References
Marcin Andrychowicz. Anton Raichuk, Piotr Stańczyk, Manu Orsini, Sertan Girgin, Raphaël Marinier, Leonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, Sylvain Gelly, and Olivier Bachem. What matters for on on-policy deep actor-critic methods? a large-scale study. In International Conference on Learning Representations, 2021.
Christopher G Atkeson and Stefan Schaal. Robot learning from demonstration. In Proceedings of the Fourteenth International Conference on Machine Learning, pages 12-20, 1997.
Jonathan Booher. Be+rl: Imitation learning from non-optimal demonstrations. StLLhttps: potttps//oi /wbb
staaffod. sstanfordeedu actu/"jaustinbpppperssss cttpapperrr//22..pdeeee
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym, 2016.
Jacob Buckman, Carles Gelada, and Mare G Bellemare. The importance of pessimism in fixed-dataset policy optimization. arXiv preprint wwXiv22009006799 2020.
Xinyue Chen, Zijian Zhou, Zheng Wang, Che Wang, Yanqiu Wu, and Keith Ross. Bail: Best- action imitation learning for batch deep reinforcement learning, Advances in Neural Information Processing Systems, 33, 2020.
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep rl: A case study on ppo and trpo. In International Conference Learning Representations, 2020.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep
data-driven reinforcement learning. 2020.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International Conference on Machine Learning, volume 80, pages 1587-1596. PMLR, 2018.
Scott Fujimoto, Edoardo Conti, Mohammad Ghavamzadeh, and Joelle Pineau. Benchmarking batch deep reinforcement learning algorithms. arXiv preprint arXiv:1910.01708 2019a.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pages 2052-2066 2019b.
Hiroki Furuta, Tadashi Kozuno, Tatsuya Matsushima, Yutaka Matsuo, and Shixiang Shane Gu. Iden- tifying co-adaptation of algorithmic and implementational innovations in deep reinforcement learn- ing: A taxonomy and case study of inference-based algorithms. arXiv preprint arXiv:2103. 17258, 2021.
Jason Gauci, Edoardo Conti, Yitao Liang, Kittipat Virochsiri, Yuchen He, Zachary Kaden, Vivek Narayanan, Xiaohui Ye, Zhengxing Chen, and Scott Fujimoto. Horizon: Facebook's 's open source applied reinforcement learning platform. arXiv preprint arXiv:1811.00266 2018.
Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, and Shixiang Shane Gu. Emaq: Expected-max q-learning operator for simple yet effective offline and online rl. arXiv preprint arXiv:2007.11091. 2020.
Vinicius G Goecks, Gregory Gremillion, Vernon J Lawhern, John Valasek, and Nicholas R Waytowich. Integrating behavior cloning and reinforcement learning for improved performance in dense and sparse reward environments. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems, pages 465-473, 2020.
Omer Gottesman, Fredrik Johansson, Joshua Meier, Jack Dent, Donghun Lee, Srivatsan Srinivasan, Linying Zhang, Yi Ding, David Wihl, Xuefeng Peng, et al. Evaluating reinforcement learning algorithms observational health settings. arXiv preprint arXiv:1805.122 12298, 2018.
10

---

Caglar Gulcehre, Tom Le Paine. Bobak Shahriari, Misha Denil, Matt Hoffman, Hubert Soyer, Richard Tanburn, Steven Kapturowski, Neil Rabinowitz, Duncan Williams, Gabriel Marth-Maron, Ziyu Wang, Nando de Freitas, and Worlds Team. Making efficient use of demonstrations to solve hard exploration problems. International Conference on Learning Representations, 2020.
Yijie Guo, Shengyu Feng, Nicolas Le Roux, Ed Chi, Honglak Lee, and Minmin Chen. Batch reinforcement learning through continuation method. In International Conference on Learning Representations, 2021.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International Conference on Machine Learning, volume 80, pages 1861-1870. PMLR, 2018.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In AAAI Conference on Artificial Intelligence, 2017.
Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Gabriel Dulac-Amold, et al. Deep q-learning from demonstrations. arXiv preprint FrXvvv1704.03722 2017.
Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforeement learning of implicit human preferences in dialog. arXiv preprint 1rxiv:1907.004530 2019.
Kshitij Judah, Alan Fern, Prasad Tadepalli, and Robby Goetschalekx. Imitation learning with demon- strations and shaping rewards. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 28, 2014.
Sham Kakade. A natural policy gradient. In Proceedings of the 14th International Conference on Neural Information Processing Systems: Natural and Synthetic, pages 1531-1538, 2001.
Bingyi Kang, Zequn Jie, and Jiashi Feng. Policy optimization with demonstrations. International Conference on Machine Learning, pages 2469-2478. PMLR, 2018
Beomjoon Kim, Amir-massoud Farahmand, Joelle Pineau, and Doina Precup. Learning from limited demonstrations. In Advances in Neural Information Processing Systems, pages 2859-2867, 2013.
Ilya Kostrikov, Jonathan Tompson, Rob Fergus, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. arXiv preprint arXiv:2103.0800 08050, 2021.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy
q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems, pages 11784-11794, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint wxXiv:2006.04799, 2020.
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement Jearning. In Reinforcement learning, pages 43-73. Springer, 2012.
Romain Laroche, Paul Trichelair, and Remi Tachet Des Combes. Safe policy improvement with baseline bootstrapping. International Conference on Machine Learning, pages 3652-3661. PMLR, 2019.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint 20t.:200550/6443 2020.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.022 02971, 2015.
Travis Mandel, Yun-En Liu, Sergey Levine, Emma Brunskill, and Zoran Popovic. Offline policy eval- uation across representations with applications to educational games. In International Conference on Autonomous Agents and Multiagent Systems, 2014.

---

Tatsuya Matsushima, Hiroki Furuta, Yutaka Matsuo, Ofir Nachum, and Shixiang Gu. Deployment- efficient reinforcement learning via model-based offline optimization. arXiv preprint arXiv:2006.03647, 2020.
Ofir Nachum, Bo Dai, Ilya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans. Algaedice: Policy gradient from arbitrary experience. arXiv preprint arXiv:1912.02072. 2019.
Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojciech Zaremba, and Picter Abbeel. Over- coming exploration in reinforcement learning with demonstrations. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 6292-6299. IEEE, 2018.
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09355) 2020.
Ian Osband, Yotam Doron, Matteo Hessel, John Aslanides, Eren Sezener, Andre Saraiva, Katrina McKinney, Tor Lattimore, Csaba Szepesvari, Satinder Singh, et al. Behaviour suite for reinforce- ment learning. arXiv preprint arXiv: / 908. 03568, 2019.
Tom Le Paine, Cosmin Paduraru, Andrea Michi, Caglar Gulcehre, Konrad Zolna, Alexander Novikov, Ziyu Wang, and Nando de Freitas. Hyperparameter selection for offline reinforcement learning. arXiv preprint (xXi::0077000555 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Advances Neural Information Processing Systems, pages 8024-8035, 2019.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighter regression. Simple and scalable off-policy reinforcement learning. arXiv preprint 1rXiv:1910.00117,,202 2019.
Mark Pfeiffer, Samarth Shukla, Matteo Turchetta, Cesar Cadena, Andreas Krause, Roland Siegwart, and Juan Nieto. Reinforeed imitation: Sample efficient deep reinforcement learning for mapless navigation by leveraging prior demonstrations. IEEE Robotics and Automation Letters, 3(4): 4423-4430, 2018.
Dean A Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural computation, 3(1):88-97, 1991.
Antonin Raffin, Ashley Hill, Maximilian Ernestus, Adam Gleave Anssi Kanervisto, and Noah Dormann. Stable baselines3. https:///ithuu. http://gittuuu..o//........ RM/ -ta.DDR--RR//aaaaaa-aaeeeleeelea ines3 2019.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv: / 1799. 10087, 2017.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction no-regret online learning. In Proceedings of the fourteenth international conference artificial intelligence and statistics, pages 627-635. JMLR Workshop and Conference Proceedings, 2011.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 17X1v:1707.06337, 2017.
Noah Siegel, Jost Tobias Springenberg, Felix Berkenkamp. Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavior modelling priors for offline reinforcement learning. In International Conference on Learning Representations, 2020.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International Conference on Machine Learning, pages 387-395, 2014.
Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.
12

---

Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5026-5033. IEEE, 2012.
George Tucker, Surya Bhupatiraju, Shixiang Gu, Richard Turner, Zoubin Ghahramani, and Sergey Levine. The mirage of action-dependent baselines reinforcement learning. In International Conference Machine Learning, 2018.
Matej Večerík, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nicolas Heess, Thomas Rothörl, Thomas Lampe, and Martin Riedmiller. Leveraging demonstrations for deep reinforcement learning roboties problems with sparse rewards. arXiv preprini wxXiv:177.0088) 2017.
Qing Wang, Jiechao Xiong, Lei Han, Peng Sun, Han Liu, and Tong Zhang. Exponentially weighted imitation learning for batched historical data. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 6291-6300 2018.
Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenberg, Scott E Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, et al. Critic regularized regression. Advances in Neural Information Processing Systems, 33, 2020.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint 19 111611111,1111111111111111111111 2019.
Yuchen Wu, Melissa Mozifian, and Florian Shkurti. Shaping rewards for reinforcement learning with imperfect demonstrations using generative models. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 6628-6634, IEEE, 2021.
Mengjiao Yang, Bo Dai, Ofir Nachum, George Tucker, and Dale Schuurmans. Offline policy selection under uncertainty. arXiv preprint arXiv:20/2.06919. 2020.
Yuke Zhu, Ziyu Wang, Josh Merel, Andrei Rusu, Tom Erez, Serkan Cabi, Saran Tunyasuvunakool, János Kramár, Raia Hadsell, Nando de Freitas, al. Reinforcement and imitation learning for diverse visuomotor skills. arXiv preprint arXiv:1802.09564 2018
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you descrihe the limitations of your work? [Yes] See the discussion on instability in Section 4.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] In the supplementary material.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] Included with the supplementary material.
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] In the suppicmentary material.
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes] See Figure 4.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [No] Partially. Run time experiments include compute description in Section 5.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] the supplementary material.
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "UKwS39l_2xO", "reviewer": "Reviewer_H8Cu", "summary": "This paper proposes a new, simple approach to offline reinforcement learning, based on two modifications on top of a standard online RL algorithm (TD3) - introducing a BC loss to the policy loss, and normalizing the state features in the datasets. The authors show that these modifications achieve state-of-the-art performance for offline RL in the MuJoCo domain, even when compared to prior methods which involve many more moving pieces.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The originality of the work is small, as the proposed method is a small change on top of prior work. However, the authors recognize this, and I can see that the proposed method (TD3+BC) will be a strong but simple to implement baseline for all future offline RL algorithms going forward, so I view the contribution of this paper as potentially valuable and significant. The fact that such a simple modification results in near state-of-the-art results is surprising and potentally means there is still a lot of work for improving offline RL algorithms going forward. \n\nThe writing of the paper is good and easy to understand. It is clear what the motivation of the paper (to propose a simple, easy-to-implement but competitive offline RL method) and I believe that the work meets those claims. \n\nFor experiments, I believe that among model-free methods, these are the strongest or close to state-of-the-art results, which is fairly surprising given the simplicity of the method. One concern I have with the results is that they are primarily done in the MuJoCo domain, which are some of the easier tasks contained within the D4RL benchmark. I would be curious to see what the performance of the method is on a task such as the AntMaze navigation task, since in principle the task requires one to use dynamic programming to stitch together multiple incomplete trajectories. This would be a more interesting excercise for an offline RL algorithm where BC would not likely be as beneficial as it is for the MuJoCo tasks.\n\nOverall, I think this is a good paper, mainly on the premise that it a) seems to be an overlooked method by the community, and b) achieves state-of-the-art results by a small margin on the domains reported. I think the paper could be stronger if there was more analysis of the method given that it is so simple (such as a theoretical analysis of the modified objective - does it still result in a convergent algorithm in simplified cases such as tabular, does adding BC loss prevent the method from achieving optimal returns, etc.). However, I still think the current experimental results are in general satisfactory.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors discuss some limitations of the method with regards to stability of the learned policies, and potential negative consequences such as negative applications of offline RL and data bias.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "SpA3irJIhVY", "reviewer": "Reviewer_RqWM", "summary": "This paper diagnoses some pitfalls with complex models in offline RL and proposes a very simple baseline model which attains competitive performance. This minimalist approach involves adding a behavior cloning term to the TD3 objective and normalizing the states and makes its hyperparameter easier to tune by normalizing by the value.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper attempts to cut through the often-confusing tangle of current algorithms for offline RL by providing a more straightforward alternative. The proposed method is in fact simple and easy to implement. The scaling rule for its hyperparameter is the least intuitive component but remains much simpler than the methods it compares to.\n\nAblating the state of the art algorithms by removing their implementation adjustments is a sensible thing to do. This emphasizes the need for \"test environments\" which researchers do not have access to for hyperparameter tuning; this is especially pressing in offline RL as access to the environment should be considered expensive and methods may overfit to the specific benchmark dataset. This need remains future work.\n\nOverall this is the kind of paper that I'm glad someone wrote. It does not have sensational results or a flashy method, but it points out how much of current offline RL work is light with no heat. It provides a baseline that is easy and fast to train and which should probably be the bar for new research going forward.\n\n### Notes to the authors\n* Missing \"which\" or other connector at the end of line 188.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "Ct7PNODT2q", "reviewer": "Reviewer_63kT", "summary": "This paper presents a simple offline RL method that only requires a few lines of changes in code. The authors add a BC loss as the regularization to the standard policy update objective while also normalizing the states. The authors also normalize the coefficient of the BC loss with the average Q values in the dataset to balance the policy update objective and the BC loss. Through experiments, the proposed method appears to perform comparably to the best-performing prior offline RL approaches despite being simple and requiring lower tuning complexity.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written and easy to understand. The proposed approach focuses on simplicity yet it seems to perform comparably to baselines in the empirical evaluations. The proposed method is also easier to tune and therefore makes it less costly to evaluate the train policies online after offline training. I think the method is of reasonable significance to the field of offline RL.\n\nHowever, I do have a few concerns, which I will discuss as follows.\n\nFirst, it seems that the novelty of the method is a bit limited. The authors seem to directly adapt RL+BC to the offline setting except that they add the state normalization, which is also not new. The authors also didn't theoretically justify the approach. For example, the authors should show that the method can guarantee safe policy improvement and moreover enjoys comparable or better policy improvement guarantees w.r.t. prior approaches. Without the theoretical justification and given the current form of the method, I think the method is a bit incremental.\n\nMoreover, the empirical evaluations are not thorough. The authors only evaluated the approach in simple mujoco environments in D4RL. It is unclear if the method can perform well more undirected multitask datasets such as antmaze and kitchen as well as more complicated manipulation tasks such as adroit in D4RL. It also seems that the method does not perform well on random datasets. Is this a major limitation? I also think the authors should add state normalization to all baselines to ensure fair comparison as the state normalization is not a new technique in RL.\n\nFinally, I think the comparisons are not complete. The authors should also compare the method to more recent model-free offline RL approaches such as [1] and model-based methods such as [2,3], which attains better performance on random and medium-replay datasets.\n\nOverall, given the above comments, I would vote for a weak reject.\n\n[1] Sinha, S., & Garg, A. (2021). S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning. arXiv preprint arXiv:2103.06326.\n\n[2] R. Kidambi, A. Rajeswaran, P. Netrapalli, and T. Joachims. Morel: Model-based offline reinforcement learning. arXiv preprint arXiv:2005.05951, 2020.\n\n[3] T. Yu, G. Thomas, L. Yu, S. Ermon, J. Zou, S. Levine, C. Finn, and T. Ma. Mopo: Model-based offline policy optimization. arXiv preprint arXiv:2005.13239, 2020.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "5am1mJ7nBLB", "reviewer": "Reviewer_6fNL", "summary": "The authors revisit addition of a behavior regularization term for offline reinforcement learning but with minimal change to existing RL approach, namely TD3. They showcase their approach in D4RL gym environments. The results are surprisingly good for such a small change and  are competitive with SOTA for the most part. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n**Strengths:**\n\nS1: Overall, the proposed method is very straightforward, fairly intuitive, and the performance seems comparable with the presented baselines. \n\nS3: The experimental results present a compelling case for simplicity in offline RL and calls into attention the problem of high variance in offline-trained policies. \n\nS2: With a few exceptions (discussed below), the presentation quality is high. Figures are and tables clear/readable. Writing is clear.\n\nS3: The codebase accompanying the publication is minimal by design and supports high reproducibility. \n\t\n**Weaknesses:**\n\nW1: No free Lunch - The proposed algorithm while being simple seems vulnerable to noisy datasets with a large number of sub-optimal trajectories. It would be nice to see some experiments where for the given datasets of  \"random\" and \"replay\" tags are combined with \"expert\" datasets. This will reveal the effect of regularization in a dataset collected from  diverse set of sub-optimal policies. Datasets with noisy action spaces would also be an interesting addition. The paper fails to showcase these edge cases where the current simple algorithm might fail over the more complex offline RL approaches. \n\t\nW2: The choice of hyperparameter does seem to affect the performance of the algorithm, However the initial choice of 2.5 seems to be arbitrary. This is because even if the actions are normalized the regularization term is dependent on dimension of the action space. Moreover a simple addition of random dimension in the action space might affect the performance of the solution. Hence it is still not clear how to heuristically choose the value of lambda for a new problem. \n\n\n**Minor Clarity Issues/ Comments:**\n\nCL1: It would also be interesting to see if the performance holds up for challenging tasks such as Ant-Maze. \n\nCL2: In figure 5 It would be easy for the reader if it is explicitly mentioned that the full algorithm equates to\" TD3 + BC + NORM + lambda\" and replacing \"- Norm\" with \" TD3 + BC + lambda\". (If I understood the figure correctly.) ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Overall this paper makes a nice contribution and is arguably an example of the type of work our fields need more of. 

My only reservation is that the authors should have done more to highlight deficiencies of the approach, which may not be present for some of the other approaches. In particular, the approach obviously should suffer when data has coverage of good policies, but may not correspond to a good policy that one should aim to imitate. The authors are aware of this and will ideally make this point more strongly in the revision. Regardless, the paper still makes a nice contribution. 


Decision: Accept (Spotlight)