Behavior Regularized Offline Reinforcement Learning
Authors: Yifan Wu
George Tucker
Ofir Nachum
Venue:  (2020)
Keywords: reinforcement learning
offline RL
batch RL

Abstract: In reinforcement learning (RL) research, it is common to assume access to direct online interactions with the environment. However in many real-world applications, access to the environment is limited to a fixed offline dataset of logged experience. In such settings, standard RL algorithms have been shown to diverge or otherwise yield poor performance.  Accordingly, much recent work has suggested a number of remedies to these issues.  In this work, we introduce a general framework, behavior regularized actor critic (BRAC), to empirically evaluate recently proposed methods  as well as a number of simple baselines across a variety of offline continuous control tasks. Surprisingly, we find that many of the technical complexities introduced in recent methods are unnecessary to achieve strong performance. Additional ablations provide insights into which design choices matter most in the offline RL setting.

Under review as a conference paper at ICLR 2020
BEHAVIOR REGULARIZED OFFLINE REINFORCEMENT LEARNING
Anonymous authors Paper under double-bling review
ABSTRACT
In reinforcement learning (RL) research, is common to assume access to direct online interactions with the environment. However in many real-world applications, access to the environment is limited to fixed offline dataset of logged experience. In such settings, standard RL algorithms have been shown to diverge or otherwise yield poor performance. Accordingly, much recent work has suggested number of remedies to these issues. In this work, introduce general framework. behavior regularized actor critic (BRAC), to empirically evaluate recently proposed methods well number of simple baselines across variety of offline continuous control tasks. Surprisingly, we find that many of the technical complexities introduced in recent methods are unnecessary achieve strong performance. Additional ablations provide insights into which design choices matter most in the offline RL setting.
INTRODUCTION
Offline reinforcement learning (RL) describes the setting in which a learner has access to only fixed dataset of experience. contrast to online RL, additional interactions with the environment during learning are not permitted. This setting is of particular interest for applications in which deploying a policy is costly or there is a safety concern with updating the policy online (Li et al. 2015). For example, for recommendation systems (Li al., 201 I Covington et al. (016)) health applications (Murphy et al.] 200T), deploying a new policy may only be done at low frequency after extensive testing and evaluation. In these cases, the offline dataset is often very large, potentially encompassing years of logged experience. Nevertheless, the inability to interact with the environment directly poses a challenge to modern RI. algorithms.
Issues with RL algorithms in the offline setting typically arise in cases where state and actions spaces are large or continuous, necessitating the use of function approximation. While off- off-policy (deep) RL algorithms such as DQN (Mnih et al., 2013). DDPG (Liliccrap et al. 2015). and SAC (Haarnoja et al., 2018) may be run directly on offline datasets learn policy, the performance of these algorithms has been shown to be sensitive to the experience dataset distribution, even in the online setting when using a replay buffer (Van Hasselt et al.) 2018 Fu et al. 2019). Morcover, Fujimoto et al. (2018a) and and]aumar et al. (219)) empirically confirm that in the offline setting, DDPG fails to learn good policy, even when the dataset collected by single behavior policy, with or without noise added to the behavior policy. These failure cases are hypothesized to be caused by erroneous generalization of the state-action value function (Q-value function) learned with function approximators, as suggested by /Sutton| (1995): Baird (1995): Tsitsiklis & Van Roy (1997); Van Hasselt et al. (2018). To remedy this issue, two types of approaches have been proposed recently: Agargal et al. (2019) proposes to apply random ensemble of Q-value targets to stabilize the learned f-fuctioon, 2) Fujimoto et al.) ( [Jauus et al. (2019); Laroche & Trichelair (2017) propose to regularize the learned policy towards the behavior policy based on the intuition that unseen state-action pairs are more likely to receive overestimated Q-values. These proposed remedies have been shown to improve upon DQN or DDPG at performing policy improvement based on offline data. Still, each proposal makes several modifications to the building components of baseline off-policy RL algorithms, and each modification may be implemented in various ways. So a natural question to ask is, which of the design choices in these offline RL algorithms necessary to achieve good performance? For example, to estimate the target Q-value when minimizing the Bellman error, Fujimoto et al. (2018a) uses soft combination of two target Q-values, which is different from TD3
(2018)): Kumar et al.] (2019);

---

Under review as a conference paper at ICLR 2020
(Fujimoto et al. 2018b), where the minimum of two target Q-values is used. This soft combination is maintained by/ Kumar et al. (2019), while further increasing the number of Q-networks from two to four. As another example, when regularizing towards the behavior policy, Jaques et al. (2019)
uses Kullhack-Leebler (KI.) divergence with a fixed regularization weight while Kumar et al. 2019) proposes use Maximum Mean Discrepancy (MMD) with an adaptively trained regularization
weight, Are these design choices crucial to success in offline settings? Or are they simply the result of multiple, human-directed iterations of research?
In this work, we aim at evaluating the importance of different algorithmic building components as well as comparing different design choices in offline RL approaches. We focus on behavior regularized approaches applied to continuous action domains, encompassing many of the recently demonstrated successes (Fujimoto et al., 2018a Kumar et al.) 2019). We introduce behavior regularized actor critic (BRAC), general a algorithmic framework which covers existing approaches while enabling us to compare the performance of different variants in modular way. We find that many simple variants of the behavior regularized approach can yield good performance, while previously suggested sophisticated techniques such as weighted Q-ensembles and adaptive regularization weights are not crucial. Experimental ablations reveal further insights into how different design choices affect the performance and robustness of the behavior regularized approach in the offline RL setting.
2 BACKGROUND
2.1 MARKOV DECISION PROCESSES
We consider the standard fully-observed Markov Decision Process (MDP) setting (Puterman, 1990). An MDP can be represented M = (S, A, R, 7) where S is the state space, A is the action space, P(.|s,a a) is the transition probability distribution function, R(s, a) is the reward function and 7 is the discount factor. The goal is to find policy (-1ss) that maximizes the cumulative discounted reward starting from any state E S. Let (*((|s) denote the induced transition distri- bution for policy æ. For later convenience, we also introduce the notion of multi-step transition distributions as P where P (-[s) denotes the distribution over the state space after rolling out ps for steps starting from state s. For example, PJ (-[s) is the Dirac delta function at s and (-(())) = (*s(ss). We use RT (s) to denote the expected reward at state when following policy n, i.e. (s) -Eana(-1)) [R(s, a)). The state value function (a.k.a. value function) is defined by (5) = ~ T'ENEE
(s:).. The action-value function (a.k.a. Q-function) can be written as Q (s) = R(s,0) a) + y. P(-laa)))) [V (s')). The optimal policy is defined as the policy x² that maximizes V=" (s) at all states E S. In the commonly used actor critic paradigm, one optimizes a policy (-(ss) by alternatively learning a Q-value function Q₂ to minimize Bellman errors over single step transitions (s, a, r,, s'). Earooo(te)) [ (r+>2(s', 7Q(s'' aa)) - Q (s, "))²]. where Q denotes a target Q function; e.g., it is common use slowly-updated target parameter set v/ to determine the target Q function as Qwr (s', a'). Then, the policy is updated to maximize the Q-values, Ea-x(-le) [Q (s, a)].
2.2 OFFLINE REINFORCEMENT LEARNING
Offline RL (also known as batch RL (Lagge et al. 2012) 2 considers the problem of learning policy m
from a fixed dataset D consisting of single-step transitions (s, a, s'). Slightly abusing the notion of "behavior", we define the behavior policy Ty (a|s) as the conditional distribution p(a]s) observed in the dataset distribution D. Under this definition, such behavior policy my is always well-defined even if the dataset was collected by multiple, distinct behavior policies. Because we do not assume direct access to Tb, it is common in previous work to approximate this behavior policy with max- max-likelinne -ix-likeldood over D:
# == argmax (sann)))> [log (a]s))]
(1)
We denote the learned policy as मn and refer to it as the "cloned policy" to distinguish it from the true behavior policy.
In this work, we focus on the offline RL problem for complex continuous domains. We briefly review two recently proposed approaches, BEAR (Kumar et al. 2019) and BCQ (Fujimoto et al. 8a).
2

---

Under review as a conference paper at ICLR 2020
BEAR Motivated by the hypothesis that deep RL algorithms generalize poorly to actions outside the support of the behavior policy, Kumar et al. (2019) propose BEAR, which learns a policy to maximize Q-values while penalizing it from diverging from behavior policy support. BEAR measures divergence from the behavior policy using kernel MMD (Gretton et al. 2007): :
(= mo(-s)) (--1))(--1))
[x
21E (1s))
[K (zz y)]
E v.0' -
(v.,3)) (2)
gomma (11s
where K is a kernel function. Furthermore, to avoid overestimation in the Q-values, the target Q-value function Q is calculated as.
Q(s == 0.75 min Quy (s', 0.25
max Quy (s',
(3)
j=I....,
J=I....k
where v// is denotes soft-updated ensemble of target Q functions. In BEAR's implementation, this ensemble is of size & 4. BEAR also penalizes target Q-values by an ensemble variance term. However, their empirical results show that there no clear benefit to doing so, thus we omit this term.
BCQ BCQ enforces to be close to m. with specific parameterization of n Te(als) argmax Qu + Ea(s, a:)) for a, TT(als), i= 1....,,N N, (xt()()
(4)
where &o is function approximator with bounded ouptput in |- Φ where 4 is hyperparameter. N an additional hyperparameter used during evaluation to compute Ty and during training for Q-value updates. The target Q-value function Q is calculated as in Equation 3 but with l = 2.
3 BEHAVIOR REGULARIZED ACTOR CRITIC
Encouraging the learned policy to close to the behavior policy is a common theme in previous approaches offline RL. To evaluate the effect of different behavior policy regularizers, we introduce behavior regularized actor critic (BRAC), an algorithmic framework which generalizes existing approaches while providing more implementation options.
There are two common ways to incorporate regularization to a specific policy: through a penalty in the value function or as a penalty solely on the policy. We begin by introducing the former, value penalty (vp). Similar to SAC (Haarnoja et al. (0188) which adds an entropy term to the target Q-value calculation, we add a term to the target Q-value calculation that regularizes the learned policy i towards the behavior policy Ty. Specifically, we define the penalized value function as
V5()) - 1= " (PF (R*
a (T(-)s:), mT(-|s:))]
(5)
where D is a divergence function between distributions over actions (e.g., MMD or KL divergence). Following the typical actor critic framework, the Q-value objective is given by,
[(+
-D ( (Q(((()))))) - aD (mo(-Is'), mo((ss')) - Qu(s,a)))
min E (eanss)) Qo a' a'ws)ottss")
(6)
where Q again denotes a target Q function and D denotes sample-based estimate of the divergence function D. The policy learning objective can be written as,
max Teg
(s,,,,,,')>>
E (urrma((le)) Qu a")] aD (TT)))s)), To mT((ss))]
Accordingly, one performs alternating gradient updates based on (6) and (7). This algorithm is equiv- alent to SAC when using single-sample estimate of the entropy for D; i.e., ((To(-ls'), mT(-ls²))= = log (a'lss for a' ~ T--ss').
The second way to add the regularizer is to only regularize the policy during policy optimization. That is, we use the same objectives Equations and but use a = 0 the Q update while using a non-zero a in the policy update. We call this variant policy regularization (pr). This proposal is similar to the regularization employed in A3C (Mnih et al. 2016). if one uses the entropy of अe to compute D.
In addition to the choice of value penalty or policy regularization, the choice of D and how to perform
sample estimation of Đ) is key design choice of BRAC:
3

---

Under review as a conference paper at ICLR 2020
We can compute sample based estimate of kernel MMD (Equation 2) by drawing samples from both Te and Tr. Because we do not have access to multiple samples from Tb. this requires pre- pre-estimated cloned policy Tt.
Kernel MMD
KL Divergence
With KL Divergence, the behavior regularizer can be written as
DEL (To(- To (-ls) (a-1x((1s) [log mo (a(s)) log To(a|s)]
Directly estimating DKL via samples requires having access to the density of both and mat as in MMD, the cloned it can be used in place of иı. Alternatively, by using the dual form of the KI.-divergence. Specifically, any f-divergence (Csiszár 1964) has a dual form (Nowozin al. 2016) given by,
avoid estimating as explicitly,
Dy(p, = Ez-p [S (a(<)/p(z)) = max Enng [g(x)] - Ezzp (+()))
g:Xx>sdm((-")
where f is the Fenchel dual of f. In this case, one no longer needs to estimate a cloned policy & but instead needs to learn a discriminator function with minimax optimization as in Nowozin jet al. (2016). This sample based dual estimation can be applied to any f-divergence, In the case of KL-divergence, f(x) log and f (t) -log(-))
Wasserstein Distance One may also use the Wassertein distance as the divergence D. For sample- based estimation, one may its dual form.
(p..) = sup Egep [g(x)) - Eezing [x(x)]
s1lallas1
and maintain a discriminator 9 as in Gulrajani et al. 2017).
Now discuss how existing approaches can be instantiated under the framework of BRAC.
BEAR To re-create BEAR with BRAC, one uses policy regularization with the sample-hased kernel MMD for Ô and uses min-max ensemble estimate for Q (Equation 3) Furthermore, BEAR adaptively trains the regularization weight as Lagriagian multiplier: sets threshold > 0 for the kernel MMD distance and increases a if the current average divergence is above the threshold and decreases or if below the threshold.
BCQ The BCQ algorithm does not use any regularizers (i.e. a 0 for both value and policy objectives). Still, the algorithm may be realized by BRAC if one restricts the policy optimization in Equation to be over parameterized policies based Equation
KI.-Control There has been rich set work which investigates regularizing the learned policy through KL-divergence with respect to another policy, e.g. Abdolmaleki et al. (2018): Kakade (2002); Peters et al. [Shl0)][Shhulman (Schulman et al. (2015) Nachum et al. (2017). Notably, Jaques et al. (2019) apply this idea offline RL in discrete action domains by introducing KL value penalty in the Q-value definition. clear that BRAC can realize this algorithm as well.
To summarize, can instantiate the behavior regularized actor critic framework with different design choices, including how to estimate the target Q value, which divergence to use, whether to learn adaptively, whether to
value penalty in the Q function objective (6) just policy regularization in (7) and so on. In the next section, we empirically evaluate set of these different design choices to provide insights into what actually matters when approaching the offline RL problem.
4 EXPERIMENTS
The BRAC framework encompasses several previously proposed methods depending on specific design choices (e.g., whether to use value penalty policy regularization, how to compute the target Q-value, and how to impose the behavior regularization). For practitioner, key questions are: How should these design choices be made? Which variations among these different algorithms actually matter? To answer these questions, we perform a systematic evaluation of BRAC under different design choices.
4

---

Under review as a conference paper at ICLR 2020
Following Kumar et al. (2019), we evaluate performance on four Mujoco Todorov et al. 2012 continuous control environments in OpenAl Gym (Brockman et al., 2016): Ant-v2, HalfCheetah- v2, Hopper-v2, and 22 In many real- real-world applications of RL, one has logged data from sub-optimal policies (e.g., robotic control and recommendation systems). To simulate this scenario, we collect the offline dataset with sub-optimal policy perturbed by additional noise. To obtain a partially trained policy, we train policy with SAC and online interactions until the policy performance achieves a performance threshold (1000, 4000, 1000, 1000 for Ant-v2, HalfCheetahh-2,, Hopper-v2, Walker2d-v2, respectively, similar to the protocol established by Kumar et al. (2019)). Then, we perturb the partially trained policy with noise (Gaussian noise or c-greedy at different levels) to simulate different exploration strategies resulting in five noisy behavior policies. We collect I million transitions according to each behavior policy resulting in five datasets for each environment (see Appendix for implementation details). We evaluate offline RI. algorithms by training on these fixed datasets and evaluating the learned policies on the real environments.
In preliminary experiments, we found that policy learning rate and regularization strength have significant effect on performance. As a result, for each variant of BRAC and each environment, we do grid search over policy learning rate and regularization strength. For policy learning rate, we search over six values, ranging from 3 10 to 0.001. The regularization strength is controlled differently in different algorithms. In the simplest case, the regularization weight a is fixed; in BEAR the regularization weight is adaptively trained with dual gradient ascent based on a divergence constraint that is tuned as hyperparameter; in BCQ the corresponding tuning is for the perturbation range 4. For each of these options, we search over five values (see Appendix for details). For existing algorithms such as BEAR and BCQ, the reported hyperparameters in their papers (Kumar al.
2019 Fujimoto et al. 2018a) are included in this search range, We select the best hyperparameters according to the average performance over all five datasets.
Currently, BEAR (Kumar et al. 2019) provides sfaeeoof-hhe-a performance on these tasks, so to understand the effect of variations under our BRAC framework, we start by implementing BEAR in BRAC and run series of comparisons by varying different design choices: adaptive vs. fixed regu- larization, different ensembles for estimating target Q-values, value penalty vs. policy regularization and divergence choice for the regularizer. We then evaluate BCQ, which has a different design in the BRAC framework, and compare it to other BRAC variants as well as several baseline algorithms.
4.1 FIXED V.S. ADAPTIVE REGULARIZATION WEIGHTS
In BEAR, regularization is controlled by threshold c, which is used for adaptively training the Lagrangian multiplier a, whereas typically (e.g., in KL-control) one uses fixed a. In our initial experiments with BEAR, we found that when using the recommended value of e, the learned value of consistently increased during training, implying that the MMD constraint between To and T. was almost never satisfied. This suggests that BEAR is effectively performing policy regularization with a large rather than constrained optimization. This led us to question if adaptively training is better than using fixed a. To investigate this question, we evaluate the performance of both approaches (with appropriate hyperparameter tuning for each, over either both approaches learn a policy that is much better than the partially trained policy"] although we do observe a consistent modest advantage when using fixed a. Because using a fixed a simpler and performs better than adaptive training, we use this approach in subsequent experiments.
e) in Figure []] On most datasets,
4.2 ENSEMBLE FOR TARGET Q-VALUES
Another important design choice in BRAC is how to compute the target Q-value, and specifically, whether one should use the sophisticated ensemble strategies employed by BEAR and BCQ. Both BEAR and BCQ use a weighted mixture of the minimum and maximum among multiple learned Q-functions (compared to TD3 which simply uses the minimum of two). BEAR further increases the number of O-functions from 2 to 4. To investigate these design choices, we first experiment with different number of Q-functions show that using two Q-functions provides significant improvements in online RI.; similarly, we find that using k = 1 sometimes fails to learn a good policy (e.g., Walker2d) in the offline setting.
{1. 2, 4). Results are shown in Figure 2 Fujimoto et al. (2018b)
'The partially trained policy is the policy used to collect data without injected noise. The true behavior policy and behavior cloning will usually get worse performance due injected noise when collecting the data.
5

---

Under review as a conference paper at ICLR 2020
trained alpha
fixed. alpha
HoPCCheetah-22 v2
Hazpers22
kerke
2300
2500
2500
2000
3300
1250
1030
colosets
colos4t
Figure/ 1: Comparing fixed a with adaptively trained a. Black dashed lines are the performance the partially trained policies (distinct from the behavior policies which have injected noise). We report the mean over the last 10 evaluation points (during training) averaged over 5 different random seeds. Each evaluation point is the return averaged over 20 episodes. We report the performance 0 if it is negative.
Using k = 4 has a small advantage compared to k = 2 except in Hopper. Both A = 2 and 1: 4 significantly improve over the partially trained policy baseline. general, increasing the value of in ensemble will lead to more stable or better performance, but requires more computation cost. On these domains we found that k = 4 only gives marginal improvement over k 2, so we use our remaining experiments.
in
Regarding whether using a weighed mixture of Q-values or the minimum, we compare these two options under A 2. Results are shown in Figure 3 We find that taking the minimum performs slightly better than taking a mixture except in Hopper, and both successfully outperform the partially trained policy in all cases. Due to the simplicity and strong performance of taking the minimum of two Q-functions, we use this approach in subsequent experiments.
k-1
k-2
k-4
PCheeeahh2
Hospers2
Y000
2000
2500
19/00
50000
1200
12:0
calosets
culooots
dalasots
Figure 2: Comparing different number of Q-functions for target Q-value ensemble. We use weighted mixture to compute the target value for all of these variants. As expected, we find that using an ensemble (k > 1) is better than using a single Q-function.
mix
min
2500-
2006
35/00
1500
1500
1000
500
CaOiett
Calonats
Figure 3: Comparing taking the minimum v.s. weighted mixture in Q-value ensemble. We find that simply taking the minimum is usually slightly better, except Hopper-v2.
4.3 VALUE PENALTY OR POLICY REGULARIZATION
So far, we have evaluated variations in regularization weights and ensemble of Q-values. We found that the technical complexity introduced in recent works not always necessary to achieve of the-art performance. With these simplifications, we now evaluate major variation of design choices in BRAC- -
using value penalty or policy regularization. We follow our simplified version BEAR: MMD policy regularization, fixed and computation of target Q-values based on the minimum of k 2 ensemble. We compare this instantiation of BRAC to its value penalty version, with results shown in Figure 4 While both variants outperform the partially trained policy, we find that value penalty performs slightly better than policy regularization in most cases. We consistently observed this advantage with other divergence choices (see Appendix Figure 8 for full comparison).

---

Under review as a conference paper at ICLR 2020
mmdd pr
mnn_ypp vp
HoPCCheetah-22
kerkz
2300
3500
colosets
colosots
Figure 4: Comparing policy regularization (pr) v.s. value penalty (vp) with MMD. The use of value penalty is usually slightly better.
4.4 DIVERGENCES FOR REGULARIZATION
We evaluated four choices of divergences used as the regularizer D: (a) MMD (as in BEAR). (b) KL in the primal form with estimated behavior policy (as in KL-control), and (c) KL and (d) Wasserstein in their dual forms without estimating a behavior policy. As shown in Figure we do not find any specific divergence performing consistently better or worse than the others. All variants are able to learn a policy that significantly improves over the behavior policy in all cases.
In contrast, Kumar al. (2019) argue that sampled MMD superior to KL based on the idea that it is better to regularize the support of the learned policy distribution to be within the support of the behavior policy rather than forcing the two distributions to similar. While conceptually reasonable. we do not find support for that argument in our experiments: (i) we find that KI. and Wassertein can perform similarly well to MMD even though they are not designed for support matching; (ii) we briefly tried divergences that are explicitly designed for support matching (the relaxed KL and relaxed Wasserstein distances proposed by Wu et al. (2019), but did not observe clear benefit to the additional complexity. We conjecture that this is because even if one uses noisy or multiple behavior policies to collect data, the noise is reflected more in the diversity of states rather than the diversity of actions a single state (due to the nature of environment dynamics). However, expect this support matching vs. distribution matching distinction may matter in other scenarios such as smaller state spaces contextual bandits, which potential direction for future work.
mmc_pr
wupr
A1tv2
HoPCChetahh v2
Haapen 12
NJ kerzz +2
1300
20000
S2000-
500
mind_yp
KLyp
R_duell
WLYP
3004
Antv2
1:000
harChustan--e
3500
Mosper-22
4000
40000
200C
1500-
4000
2800
1900
Comparing different divergences under both policy regularization (top row) and value penalty (bottom row). All variants yield similar performance, which is significantly better than the partially trained policy.
Figure 5:
4.5 COMPARISON TO BCQ AND OTHER BASELINES
We now compare one of our best performing algorithms so far, kl_vp (value penalty with KL. divergence in the primal form), to BCQ, BEAR, and two other baselines: vanilla SAC (which uses adaptive entropy regularization) and behavior cloning. F the comparison. We find that vanilla SAC only works in the HalfCheetah environment and fails in the other three environments. Behavior cloning never learns a better policy than the partially trained policy used to collect the data. Although BCQ consistently learns policy that is better than the partially trained policy, its performance is always clearly worse than kl_ vl_ (and other variants whose performance is similar to kl_ vp, according our previous experiments). We conclude that BCQ is less favorable than explicitly
7

---

Under review as a conference paper at ICLR 2020
using& divergence for behavior regularization (BEAR and kl_ vp). Although, tuning additional hyperparameters beyond & for BCQ may improve performance.
SBC
bc
bcq
bear
up
Hazpero2
Neler2ca2
2500
2006
1500
1000
10000
500
CMbret
ciitasets
datosets
bcq
becar
vp
PCheerah uf
3100
Happersy2
3100
Welkerada2
2000
2500
2000
4000m
Training Rops 21651
traniig Stpp 23651
training steps 13:50
cranking steps (ee5)
Figure 6: Comparing value penalty with KL divergence (kl_vp) vp) to vanilla SAC, behavior cloning (bc), BCQ and BEAR. Bottom row shows sampled training curves with out of the 5 datasets. See Appendix for training curves on all datasets.
4.6 HYPERPARAMETER SENSITIVITY
In our experiments, we find that many simple algorithmic designs achieve good performance under the framework of BRAC. For example, all of the 4 divergences we tried perform similarly well when used for regularization. In these experiments, we allowed for appropriate hyperparameter tuming over policy learning rate and regularization weight, as we initially found that not doing so can lead to premature and incorrect conclusions. P However, some design choices may be more robust to hyperparameters than others. To investigate this, we also analyzed the sensitivity to hyperparameters for all algorithmic variants (Appendix Figures and 10). To summarize, we found that (i) MMD and KL Divergence are similar in terms of sensitivity to hyperparameters, (ii) using the dual form of divergences (e.g. KL dual, Wasserstein) appears to be more sensitive to hyperparameterss, possibly because of the more complex training procedure (optimizing a minimax objective), and (iii) value penalty is slightly more sensitive to hyperparameters than policy regularization despite its more favorable performance under the best hyperparameters.
ABt/Z
Happoree2
Weller2l22
2500
2000
1.20
Marmed V00
feamedd 02 1al
leareed
1earned 108/20
Figure 7: Correlation between learned Q-values and performance. x-axis is the average of learned Qs(s, a) over the last 500 training batches. y-axis is the average performance over the last 10 evaluation points. Each plot corresponds to a (environment, algorithm. dataset) tuple. Different points in each plot correspond to different hyperparameters and different random seeds.
Although we utilized hyperparameter searches in our results, in pure offline RL settings, testing on the real environment is infeasible. Thus, natural question is how to select the best hyperparameter or the best learned policy among many without direct testing. As a preliminary attempt, we evaluated whether the Q-values learned during training can be used proxy for hyperparameter selection. Specifically, we look at the correlation between the average learned Q-values (in mini-batches) and the true performance. Figure 7 shows sampled visualizations of these Q-values. We find that the learned Q-values are not a good indicator of the performance, even when they are within a reasonable range (i.e., not diverging during training). A more formal direction for doing hyperparameter selection is to do off-policy evaluation. However, off-policy evaluation is an open research problem with limited
2 For example, taking the optimal hyperparameters from one design choice and then applying them to a different design choice (e.g., MMD vs KI. divergence) can lead to incorrect conclusions (specifically, that using KI. is worse than using MMD, only because one transferred the hyperparameters used for MMD to KI.).
8

---

Under review as a conference paper at ICLR 2020
success complex continuous control tasks (see Liu al. (2018); Nachum et al. (2019); Irpan et al. (2019) for recent attempts). we leave hyperparameter selection as future work and encourage more researchers to investigate this direction.
5 CONCLUSION
In this work, we introduced behavior regularized actor critic (BRAC), algorithmic framework, which generalizes existing approaches to solve the offline RL problem by regularizing to the behavior policy. In our experiments, we showed that many sophisticated training techniques, such weighted target Q-value ensembles and adaptive regularization coefficients are not necessary in order to achieve state-of-the-an performance. We found that the use of value penalty is slightly better than policy regularization, while many possible divergences (KL, MMD, Wasserstein) can achieve similar performance. Perhaps the most important differentiator these offline settings whether proper hyperparameters are used. Although some variants of AC are more robust to hyperparameters than others, every variant relies on a suitable set of hyperparameters to train well. Off-policy evaluation without interacting with the environment is challenging open problem. While previous off off-ppolcy evaluation work focuses on reducing mean- mean-squared gquured-errorr to the expected return, in our problem, we only require ranking of policies. This relaxation may allow novel solutions, and we encourage more researchers to investigate this direction in the pursuit of truly offline RL. Another potential direction is to look the situations when the dataset is much smaller. Our preliminary observations on smaller datasets is that it is hard to get a hyperparameter that works consistently well on multiple runs with different random seeds. So we conjecture that smaller datasets may need either more careful hyperparameter search or (more interestingly) a better algorithm. We leave an extensive study of this setting to future work.
REFERENCES
Abbas Abdolmaleki, Jost Tobias Springenberg. Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum a posteriori policy optimisation. arXiv preprint arXiv: 1o0::08069990 2018.
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. Striving for simplicity in off-policy deep reinforcement learning. arXiv preprini
2019.
Leemon Baird. Residual algorithms: Reinforcement learning with function approximation. In Machine Learning Proceedings 1995, pp. 30-37. Elsevier, 1995.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540. 2016.
Paul Covington, Jay Adams, and Emre Sargin. Deep neural networks for youtube recommendations. In Proceedings of the 10th ACM conference recommender systems, pp. 191-198. ACM, 2016.
Imre Csiszár. Eine informationsshheoretshhhe ungleichung und ihre anwendung auf beweis der ergodizitaet von markoffschen ketten. Magyer Tud. Akad. Mat. Kutato Int. Koezl., 8:85-108, 1964.
Justin Fu, Aviral Kumar, Matthew Soh, and Sergey Levine. Diagnosing bottlenecks in deep q-learning algorithms. arXiv preprini arXiv: 19Xiv::900./// 002.000250 2019.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without 2018a.
exploration. arXiv preprint arXiv:
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. arXiv preprint arXiv: 080200947777 2018b.
Arthur Gretton, Karsten M Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel approach to comparing distributions. In Proceedings of the National Conference on Artificial Intelligence, volume 22, pp. 1637. Menlo Park, CA; Cambridge, MA; London; AAAI Press; MIT Press; 1999, 2007.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. In Advances in neural information processing systems, pp. 5767-5777, 2017.

---

Under review as a conference paper at ICLR 2020
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor critrccritic: Off-policy maxi- mum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXivv1801.0122 2018.
Alex Irpan, Kanishka Rao, Konstantinos Bousmalis, Chris Harris, Julian Ibarz, and Sergey Levine. Off-policy evaluation via off- ooficy classification. arXiv preprint arXiv: 01624, 2019.
Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of 2019.
implicit human preferences in dialog. arXiv preprint
Sham M Kakade. A natural policy gradient. In Advances in neural information processing systems, pp. 1531-1538, 2002.
Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off off-policy q-learning via bootstrapping error reduction. arXiv preprint arXiv: 1900//000000 1906.00949, 2019.
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforcement learning, pp. 45-73. Springer, 2012.
Romain Laroche and Paul Trichelair. Safe policy improvement with baseline bootstrapping. arXiv preprint arXiv:1712.0660
2017.
Lihong Li, Wei Chu, John Langford, and Xuanhui Wang. Unbiased offline evaluation of contextual- bandit-based news article recommendation algorithms. In Proceedings of the fourth ACM interna- lional conference Web search and data mining, pp. 297-306. ACM, 2011.
Lihong Li, Rémi Munos, and Csaba Szepesvári. Toward minimax off-policy value estimation. 2015.
Timothy Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:/509.022 029.0,9991100. 2015.
Qiang Liu, Lihong Li, Ziyang Tang, and Dengyong Zhou, Breaking the curse of horizon: Infinite- horizon off-policy estimation. In Advances in Neural Information Processing Systems, pp. 5356- 5366, 2018
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Toannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprini arXiv:1312.5602, 2013.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference machine learning, pp. 1928-19999999 1938,1937,000 2016.
Susan A Murphy, Mark van der Laan, James M Robins, and Conduct Problems Prevention Re- search Group. Marginal mean models for dynamic regimes. Journal of the American Statistical Association, 14145)))4410-1433 2001.
Ofrr Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Trust-pel: An off-policy trust region method for continuous control. arXiv preprint arXiv: 2017.
Ofir Nachum, Yinlam Chow, Bo Dai. and Lihong Li. Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections. arXiv preprint arXiv:1906.0473: 04733, 2019.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. In Advances in neural information processing systems, pp. 271-279, 2016.
Jan Peters, Katharina Mulling, and Yasemin Altun. Relative entropy policy search. In Twenty-Fourth AAAI Conference on Artificial Intelligence, 2010.
Martin L Puterman. Markov decision processes. Handbooks in operations research and management science, 2:331-434 1990.
10

---

Under review as a conference paper at ICLR 2020
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897, 2015.
Richard S Sutton. On the virtues of linear learning and trajectory distributions. In Proceedings of the Workshop Value Function Approximation, Machine Learning Conference, pp. 85, 1995.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
John Tsitsiklis and Benjamin Van Roy. Analysis of temporal-ddffferrenc learning with function approximation. In Advances in neural information processing systems, pp. 1075-1081, 1997.
Hado Van Hasselt, Yotam Doron, Florian Strub, Matteo Hessel, Nicolas Sonnerat, and Joseph Modayil. Deep reinforcement learning and the deadly triad. arXiv preprint arXiv:1812.026455 2018.
Yifan Wu, Ezra Winston, Divyansh Kaushik, and Zachary Lipton. Domain adaptation with asymmerrically-relxxed distribution alignment. In International Conference on Machine Learning, pp. 6872-6881, 2019.

---

Under review as a conference paper at ICLR 2020
A
ADDITIONAL EXPERIMENT RESULTS
A.1 ADDITIONAL EXPERIMENT DETAILS
Dataset collection For each environment, we collect five datasets: [no-noise, eps-0.1, eps-0.3, gauss-O.1, gauss-0.3) using partially trained policy и. Each dataset contains 1 million transitions. Different datasets are collected with different injected noise, corresponding to different levels and strategies of exploration. The specific noise configurations are shown below:
no-noise: The dataset is collected by purely executing the partially trained policy x without adding noise.
eps-0.1: We make epsilon greedy policy x' with 0.1 probability. That is, at each step, x' has 0.1 probability to take uniformly random action, otherwise takes the action sampled from x. The final dataset is a mixture of three parts: 40% transitions are collected by n', 40% transitions are collected by purely executing иТ, the remaining 20% are collected by random walk policy which takes uniformly random action at every step. This mixture motivated by that one may only want to perform exploration in only a portion of episodes when deploying a policy.
eps-0.3: x' is epsilon greedy policy with 0.3 probability take random action. We do the same mixture as in eps-0.1.
gauss-0.1: x' is taken as adding an independent, . (0, 0.12) Gaussian noise to each action sampled from w. We do the same mixture as in eps-0.
gauss-0.3; x' is taken as adding an independent N(0,, 0.3²) Gaussian noise to each action sampled from æ. We do the same mixture as in eps-0.1.
Hyperparameter search As we mentioned in main text, for each variant of BRAC and each environment, do a grid search over policy learning rate and regularization strength. For policy learning rate, we search over six values: {3 10 105, 3 10 0.0001, 0.0003, 0.001 The regularization strength controlled differently different algorithms:
In BCQ, we search for the perturbation range & f0.005, 0.015, 0.05, 0.15, 0.5). 0.05 is the reported value by its paper (Fujimoto et al. 20188)).
In BEAR the regularization weight is adaptively trained with dual gradient ascent based on a divergence constraint that is tuned as hyperparameter. We search for E {0.015, 0.05, 0.15, 0.5, 1.5). 0.05 is the reported value by its paper (Kumar al. 2019
When MMD used with a fixed o, we search for a E {3, 10,30, 30, 100, 300}.
When KI. divergence used with a fixed a (both KI. and KIl_dual), we search for a {0.1, 0.3, 1.0, 3.0, 10.0).
When Wasserstein distance is used with a fixed a, we search for a E (0.3, 3.0, 10.0,,0.00.
in BEAR the regularization weight is adaptively trained with dual gradient ascent based on divergence constraint that tuned as hyperparameter;
In the simplest case, the regularization weight a is fixed; in BCQ the corresponding tuning is for the perturbation range &. For each of these options, we search over five values (see Appendix for details). For existing algorithms such as BEAR and BCQ, the reported hyperparameters in their papers (Kumar et al. (2019: Fujimoto et al. (018a) are included this search range, We select best hyperparameters according to the average performance over all five datasets.
Implementation details All experiments are implemented with Tensorflow and executed on CPUs. For all function approximators, we use fully connected neural networks with RELU activations. For policy networks, we use tanh (Ganssiaa) on outputs following BEAR (Kumar et al. 2019), except for BCQ where we follow their open soureed implementation. For BEAR and BCQ we follow the network sizes as in their papers. For other variants of BRAC, we shrink the policy networks from (400, 300) to (200, 200) and Q-networks from (400,300) to (300, 300) for saving computation
12

---

Under review as a conference paper at ICLR 2020
time without losing performance. Q-function learning rate is always 0.001. As in other deep RL algorithms, we maintain source and target Q-functions with an update rate 0,005 per iteration. For MMD we use Laplacian kernels with bandwidth reported by Fujimoto et al. (2018ap. For divergences in the dual form (both KI__dual and Wasserstein), we training a (300, 300) fully connected network the critic in the minimax objective. Gradient penalty (one sided version of the penalty in Gulrajani ett al. (2017) with coefficient 5.0) is applied to both KL and Wasserstein dual training. In each training iteration, the dual critic updated for 3 steps (which we find better than only step) with learning rate 0.0001. We use Adam for all optimizers. Each agent is trained for 0.5 million steps with batch size 256 (except for BCQ we use 100 according their open sourced implementation). At time we follow Kumar et al. (2019) and Fujiimto et al. (2018a) by sampling 10 actions from at each step and take the one with highest learned Q-value.
A.2 VALUE PENALTY V.S. POLICY REGULARIZATION
mmmd_pr
mnd_yp
2500
1000
500
caltunets
HoRCheetahi12
3500
cilssets
calasots
catosets
M_dually
Antu2
Hasper-22
2500
2000-
son
201
8:000
Holl HoltCheetthhe V2
XXCE
Houper V2
Wo kerze 22
2000
23000
1800
500
cllssets
dltasets
Figure 8: Comparing policy regularization (pr) v.s. value penalty (vp) with all four divergences. The use of value penalty is usually slightly better.
13

---

Under review as a conference paper at ICLR 2020
A.3 FULL PERFORMANCE RESULTS UNDER DIFFERENT HYPERPARAMETERS
M.P reed
144
550
250
070
1850
114
380
134
pelicy learning roto
pelcy learning 1000
loarning
loarning rote
repoeeu M.
Werkerzae Ap
me
508
18.0
pia
policy Icaming 1052
policy lcaming r0ts
Icarning rots
policy learinng rate
azbny dusl
hearshur dul
r0caeeou dall up
waltersdeo the
vou
me
50mm
policy learnin rute
palicy learnnng
learning rata
pulicy learning
NurCherahhh
Wallert v2 km
1010
icm
use
36.
polkcy Icamiig 1922
policy lcaring rots
policy lcarning rats
policy learting rote
100
590
2013
cole Bood
w-th 0001 EG0 3XT
with 1-1 coin
with colet
palicy pairinn ruke
pylicy learning rañe
pol by learning rale
pol leamning rate
rxppaner
11
03
mm
nm
408
na
M todk leik odlc odac 603)
tace Siic 0132 coan com
sede te-e 0392 o18ck 603
Se-08 S-00 0 0.00k 0:01
policy learnin rute
policy leamnngg ruta
policy learning ruts
policy ming
Har vectallli2 x2 dad
Hopper qudl
Weltordd 22 dud
Ail
23
L0n0 Bil MORO 003)
18CO
B410 01 LOO 1ut policy lcamiing
policy Icamiig 1952
policyy Icomning rotz
policy (oaming
Antvo
Mile
Iroppe -d
500
maakerdd x2
3800
118
250
1166
300
18C0 341 GOBL S0B3 003)
LACO 2413 0 0000 6GB
2HD 2-10 0 03R3 030 SACO 2010 020 0
polacy Icaming 1002
policy Icamiing
policy lconinng rot2
policy ming
Figure 9: Visualization of performance under different hyperparameters. The performance averaged over all five datasets.
14

---

Under review as a conference paper at ICLR 2020
401-42 Ace
Resperd bep
60%
2002
070
141
644
potccy leariigg rale
polcy sarring rale
pel by learrang rile
poliyy leatring rabe
1:ppens
3303
1000
3305
3403
500
8415
600
mm
201 W-. 260 VAR 5000 AA1
2013 Hec AO VAR C00 XX1
INC 3e0 cAL cUNs AK1 palry
(ntlly leiningg
palry Weaning
reaid med andl
J.weesalB med crs)
Popeena2 mnd ees
AocBer22.0 rend and)
2000
2000
2000
2000
polcy waating rale
policy Searning rale
(cl Searting rale
policy learning rabe
0.0
5000
606
609
1800
1000
1800
1800
408
240 Wi. Sodua todl XA2
2013 W.
2010 COR1 6000 AA1
2010 JiC AND SURL LOO XX1
2010 SALL LUNG ARL palry
(dlry weining
palry leaning
wc ueed mc
144
900
100
11
200
pelicy leariing rite
pelccy learnng
rate
learinng
polcy leamning rote
Figure 10: Visualization of performance under different hyperparameters.
15

---

Under review as a conference paper at ICLR 2020
A.4
ADDITIONAL TRAINING CURVES
Loar
Pcheesah-uz
Happer-og
Walber
2030
2.05
2000
2000
10:0
+2000
1000
(ranerg D sings (1+3)
transro aiaps (3:3)
Crainiing alapps (lun))
Crandingg staze
Hasperso
Welkerradup
2000
6030
-2000
traneng saggs (ta5)
transig alaps (363)
Craininn shapp (lam)
Crainingg sheza
Hopperssp
-1.0
2030
training staps Ste5]
traiiing shepo
trainingg steps (2e5) Happerse
training steas (2e5) Welter/idep
2030
1000
L000
training staps (ta5)
trannin stapp (3:3)
trainiing stepss (2a5)
trainiing sheap
PChhershuu2
3100
Walkeriade2
2000
2500
2000
Training Rops 11651
traniig Stpp 13451
training steps (355)
tranking steps De5i
Figure 11: Training curves on all five datasets when comparing kl. kl_vp to other baselines.
16

---

Under review as a conference paper at ICLR 2020
mmdd jr
Happer-ig
WAllerS
2000
2000
cumdee
20W
15.0
300
graking steps
traniig stapo
Dranking steps
Cranking stepu
2004
Hasper-ug
20.00
We 2ackerZees
corulativ
2000
craiiin stepe (1a))
training Stpp (te5: (3651
triining stepe
trriniig steppe
10130
2000
-1000
Dreating alogo (Lub)
trannro alaps (365)
Duiining slago (Lub)
Davining stape (145))
Welter2da22
25000
1000
1000
trainig stepo (Lub)
stags (343)
stepps Eueb0 Ho poppprry
training steaa Faes!
Antrr
1200
00:00
Tumdeette reaeere
2000
1200
1000
50c
trakiing Stepe
Figure 12: Training curves when comparing different divergences with policy regularization. All divergences perform similarly.
17

---

Under review as a conference paper at ICLR 2020
201
Hoppert 1/2
C3TT 140001
60.00
2500
1590
3000
201.2
Ho Ho-sper-g2
Welbbrr
2906
10:00
1000
Cransing stops (145)
transin disps (303)
truitinng stapp (145)
tranniingg shups
rtheezahhhhe x2
2000
1000
Dreating slogo (Lub)
trannrn siagss 2365)
Cuatniigg stagp TlaD
Loaniiigg staps [AACI)
PCheerah +22
wyltrrdds2
2590
curdletve
2000
1000
stepo CLub)
Mtags 2343)
triiniing stepps Eueb0
staps 1245)
1000
10.00
walloride
00:00
2000
11:0
12:0
(1:50)
(145)
Figure 13: Training curves when comparing different divergences with value penalty, All divergences perform similarly.
18

---

Under review as a conference paper at ICLR 2020
A.5
FULL PERFORMANCE RESULTS UNDER THE BEST HYPERPARAMETERS
Environment: Ant-v2
Partially trained policy: 1241
dataset
no-noise
eps-0.1
eps-0.3
gauss-0.1
gauss-0.3
SAC
0
-1109
-911
-1071
-1498
BC
1235
1300
1278
1203
1240
BCQ
1921
1864
1504
1731
1887
BEAR
2100
1897
2008
2054
2018
MMD_
2839
2672
2602
2667
2640
KL_vp
2514
2530
2484
2615
2661
KL d_auu_lppp
2626
2334
2256
2404
2433
W W_vp
2646
2417
2409
2474
2487
MMD_PI
2583
2280
2285
2477
2435
KL pr
2241
2247
2181
2263
2233
KI_ XL__dal_ppr
2218
1984
2144
2215
2201
W W_pr
2241
2186
2284
2365
2344
Table Evaluation results with tuned hyperparameters. performance means overflow encountered during training due to diverging Q-functions.
Environment:
HallCheetah-v2
Partially trained policy: 4206
dataset
no-noise
eps-0.1
eps-0.3
gauss-0.1
gauss-0.3
SAC
5093
6174
5978
6082
6090
BC
4465
3206
3751
4084
4033
BCQ
5064
5693
5588
5614
5837
BEAR
5325
5435
5149
5394
5329
MMD_vP
6207
6307
6263
6323
6400
KL_vp vp
6104
6212
6104
6219
6206
KL_ dual
6209
6087
6359
5972
6340
W vp
5957
6014
6001
5939
6025
MMD
5936
6242
6166
6200
6294
KL
6032
6116
6035
5969
6219
KL_ dual_pr
5944
6183
6207
5789
6050
W pr
5897
5923
5970
5894
6031
Table 2: Evaluation results with tuned hyperparameters.
19

---

Under review as a conference paper at ICLR 2020
Environment: Hopper-v2
Partially trained policy: 1202
dataset
no-noise
eps-0.1
eps-0.3
gauss-0.1
gauss-0.3
SAC
0.2655
661.7
701
311.2
592.6
BC
1330
129.4
828.3
221.1
284.6
BCQ
1543
1652
1632
1599
1590
BEAR
0
1620
2213
1825
1720
MMD_vp
2291
2282
1892
2255
1458
VL_vp
2774
2360
2892
1851
2066
KL ____
1735
2121
2043
1770
1872
W vp
2292
2187
2178
1390
1739
MMD _r
2334
1688
1725
1666
2097
KL_
2574
1925
2064
1688
1947
KL dual pr
2053
1985
1719
1641
1551
W Wrr
2080
2089
2015
1635
2097
Table Evaluation results with tuned hyperparameters.
Environment: Walker-v2
Partially trained policy: 1439
dataset
no-noise
eps-0.1
eps-0.3
gauss-0.1
gauss-0..
SAC
131.7
213.5
127.1
119.3
109.3
BC
1334
1092
1263
1199
1137
BCQ
2095
1921
1953
2094
1734
BEAR
2646
2695
2608
2539
2194
MMD_vp
2694
3241
3255
2893
3368
KL_vp
2907
3175
2942
3193
3261
KL KL_duallvp
2575
3490
3236
3103
3333
W vp
2635
2863
2758
2856
2862
MMD _r
2670
2957
2897
2759
3004
KL_
2744
2990
2747
2837
2981
KL dual
2682
3109
3080
2357
3155
W W_pr
2667
3140
2928
1804
2907
Table Evaluation results with tuned hyperparameters.
20


Reviews:
[{"review_id": "rkxX9dh3YB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces a general framework for behavior regularized actor-critic methods, and empirically evaluates recent offline RL algorithms and different design choices. \n\nOverall, the paper is well written and easy to follow.  I appreciate the authors for their careful empirical study. I am leaning to accept the paper because (1) the experimental design is rigorous and the results provide several insights into how to design a behavior regularized algorithm for offline RL.\n\nThere are some comments for the experiments.\n1. Are the results significant (e.g. Figure 3 and 4)? Have you checked the error bars?\n2. Missing numbers: trained_alpha in dataset 0 of Hopper-v2 in Figure 1 and SAC in dataset 0 of Hopper-v2 in Figure 6? Are they negative so not reported in the figure or just missing? \n3. Do you think the conclusion will change if you use training datasets of different size (e.g. much less than 1 million)? ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "r1gZvxq1qB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a framework for evaluating offline reinforcement learning (RL) algorithms.  Results from a  thorough series of experiments are presented which suggest that certain details of recently proposed RL methods are not necessary for achieving strong performance.  These results suggests that some of the complexity in RL design can be ignored.  \n\nI commend the authors for performing a valuable test and comparison of existing offline RL methodology.\n\nThis paper could be improved by providing more clear insight and intuition about the deeper meaning of these results regarding the \"unnecessary\" technical complexities.  Could the authors suggest why certain complexities are unnecessary?  Clearly the authors of those previous works thought they were needed.  To really help researchers design better algorithms, we need to be guided by some insight about not only what doesn't work but why it doesn't work.\n\nAlso, the paper could be improved by being more clear about the nature of the evaluations.  The authors provide extensive results; but it wasn't clear whether these were \"apples-to-apples\" comparisons with the previous results in the papers that proposed the \"unnecessary\" technical complexities.  For example, I didn't see the authors say that they reproduced the results of previous works, only that they tested previous methods in certain tests.  Does the BRAC framework reproduce the results for previous papers?  If so, this should be made more clear and stated prominently in the paper so that the reader knows that BRAC is, in this reproduction of previous results sense, reliable.  If not, how if the reader to know that the \"unnecessary\" technical complexities, are truly unneccessary?\n\nFinally, the paper presents lots of results, but I did not see any mention of the statistical significance of these results.\n\nMinor issue:  In the Conclusion Section, the authors say, \"Unfortunately, off-policy ... is an challenging open problem.\"  Unfortunately?!  A challenging open problem is a good thing!  And I think the authors did a good job addressing a difficult problem.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Hyed4LRcFB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a unifying framework, BRAC, which summarizes the idea and evaluates the effectiveness of recently proposed offline reinforcement learning algorithms, specifically BEAR, BCQ, and KL control. The authors generalize existing offline RL approaches to an actor-critic algorithm that regularizes the learned policy such that it can stay close to the behavior policy. Based on prior work, the authors state that there are two variants of regularizations to the behavior policy, value penalty (vp) and policy regularization (pr) and three choices of divergence functions along with their sample estimate that measures the distance between the learned policy and the behavior policy. The paper empirically investigates the effectiveness of each regularization scheme as well as each divergence function and conclude that vp is slightly more effective than pr while all divergence functions have similar performances.\n\nOverall, this paper could be an interesting summary of prior works in offline RL and provide some empirical insights on the effectiveness of each building block in the previous approaches, though it neither offers theoretical explanations nor proposes a new offline RL algorithm that outperforms the existing methods under the BRAC framework. My score would be increased given some technical insights or some promising results in a relatively novel offline RL algorithm in the author’s response. \n\nSpecifically, the paper does a thorough ablation study on BEAR, BCQ, and KL-control within the BRAC framework. While the experimental results demonstrate some interesting phenomenons such as combining vp and the primal form of KL divergence achieving the best performance and taking minimum over Q functions outperforming using a mix of maximum and minimum over the Q functions, I believe the paper would be greatly improved if the authors can provide a new offline RL method based on the BRAC that can achieve better performance than current approaches and is less incremental than simply combining vp and KL divergence. \n\nThough BRAC summarizes offline RL methods in a neat way, it would be more technically sound if a general theoretical analysis/insights of offline RL algorithms can be offered in the paper, e.g. showing the reason that vp is outperforming pr through convergence analysis in the tabular case.\n\nMinor comment:\nError bars should be added to the all the bar plots.\n\n**UPDATE**\nAfter reading the rebuttal, I think my concerns are addressed and thus I updated my rating.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper is an empirical studies of methods to stabilize offline (ie, batch) RL methods where the dataset is available up front and not collected during learning. This can be an important setting in e.g. safety critical or production systems, where learned policies should not be applied on the real system until their performance and safety is verified. Since policies leave the area where training data is present, in such settings poor performance or divergence might result, unless divergence from the reference policy is regularized. This paper studies various methods to perform such regularization. 

The reviewers are all very happy about the thoroughness of the empirical work. The work only studies existing methods (and combination thereof), so the novelty is limited by design. The paper was also considered well written and easy to follow. The results were very similar between the considered regularizers, which somehow limits the usefulness of the paper as practical guideline (although at least now we know that perhaps we do not need to spend a lot of time choosing the best between these). Bigger differences were observed between "value penalties" versus "policy regularization". This seems to correspond to theoretical observations by Neu et al (https://arxiv.org/abs/1705.07798, 2017), which is not cited in the manuscript. Although unpublished, I think that work is highly relevant for the current manuscript, and I'd strongly recommend the authors to consider its content. Some minor comments about the paper are given below.

On the balance, the strong point of the paper is the empirical thoroughness and clarity, whereas novelty, significance, and theoretical analysis are weaker points. Due to the high selectivity of ICLR, I unfortunately have to recommend rejection for this manuscript.

I have some minor comments about the contents of the paper:
- The manuscript contains the line:  "Under this definition, such a behavior policy πb is always well-defined even
if the dataset was collected by multiple, distinct behavior policies". Wouldn't simply defining the behavior as a mixture of the underlying behavior policies (when known) work equally well?
- The paper mentions several earlier works that regularize policies update using the KL from a reference policy (or to a reference policy). The paper of Peters is cited in this context, although there the constraint is actually on the KL divergence between state-action distributions, resulting in a different type of regularization.
Decision: Reject