BRAC+: Going Deeper with Behavior Regularized Offline Reinforcement Learning
Authors: Chi Zhang
Sanmukh Rao Kuppannagari
Viktor Prasanna
Venue:  (2021)
Keywords: offline reinforcement learning
behavior regularization

Abstract: Online interactions with the environment to collect data samples for training a Reinforcement Learning agent is not always feasible due to economic and safety concerns. The goal of Offline Reinforcement Learning (RL) is to address this problem by learning effective policies using previously collected datasets. Standard off-policy RL algorithms are prone to overestimations of the values of out-of-distribution (less explored) actions and are hence unsuitable for Offline RL. Behavior regularization, which constraints the learned policy within the support set of the dataset, has been proposed to tackle the limitations of standard off-policy algorithms. In this paper, we improve the behavior regularized offline reinforcement learning and propose \emph{BRAC+}. We use an analytical upper bound on KL divergence as the behavior regularizor to reduce variance associated with sample based estimations. Additionally, we employ state-dependent Lagrange multipliers for the regularization term to avoid distributing KL divergence penalty across all states of the sampled batch. The proposed Lagrange multipliers allow more freedom of deviation to high probability (more explored) states leading to better rewards while simultaneously restricting low probability (less explored) states to prevent out-of-distribution actions. \textcolor{blue}{To prevent catastrophic performance degradation due to rare out-of-distribution actions, we add a gradient penalty term to the policy evaluation objective to penalize the gradient of the Q value w.r.t the out-of-distribution actions. By doing so, the Q values evaluated at the out-of-distribution actions are bounded.} On challenging offline RL benchmarks, BRAC+ outperforms the state-of-the-art model-free and model-based approaches.

Under review as a conference paper at ICLR 2021
BRAC+: GOING DEEPER WITH BEHAVIOR REGULAR- IZED OFFLINE REINFORCEMENT LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
Online interactions with the environment to collect data samples for training a Reinforcement Learning agent not always feasible due to economic and safety concerns. The goal of Offline Reinforcement Learning (RL.) is to address this problem by learning effective policies using previously collected datasets. Stan- dard off- off-policy RL algorithms prone to overestimations of the values of out- of-distribution (less explored) actions and are hence unsuitable for Offline RL. Behavior regularization, which constraints the learned policy within the support set of the dataset, has been proposed to tackle the limitations standard off- off-policy algorithms. In this paper, we improve the behavior regularized offline reinforce- ment learning and propose BRAC+. We use an analytical upper bound on KL divergence as the behavior regularizor reduce variance associated with sample based estimations. Additionally, we employ depende Lagrange multipli- ers for the regularization term to avoid distributing KL divergence penalty across all states of the sampled batch. The proposed Lagrange multipliers allow more freedom of deviation high probability (more explored) states leading to better rewards while simultancously restricting low probability (less explored) states prevent out-of-distribution actions. To prevent catastrophic performance degra- dation due to rare out-of-distribue of-distribution actions, we add gradient penalty term to the policy evaluation objective to penalize the gradient of the Q value w.r. the actions. By doing so, the Q values evaluated at the out-of- distribution actions are bounded. On challenging offline RL benchmarks, BRAC+ outperforms the oate-of-thhearr model-free and model-based approaches.
INTRODUCTION
Reinforcement Learning (RL) has shown great success in wide range of applications including
board games (Silver et al. 2016), strategy games (Vinyals et al. 2019). energy systems (Zhang et al. 2019), robotics (Lin 1992), recommendation systems (Choi et al. 2018),, etc. The success of RL relies heavily on extensive online interactions with the environment for exploration. However, this is not always feasible in the real world as it can be expensive dangerous (Levine al. 2020
Offline RI,, also known as batch RI., avoids online interactions with the environment by learning from static dataset that is collected an offline manner (Levine et al. 2020). While standard off-
policy RL algorithms (Mnih et al. 2013 Lillicrap et al. (0066) Haarnojj et al. 220 8a) can, in theory, be employed to learn from offline data, in practice, they perform poorly due distributional shift between the behavior policy (probability distribution of actions conditioned on states observed in the dataset) of the collected dataset and the learned policy (Levine et al., 2020). The distributional shift manifests itself in form of overestimation of the nut-of-distribution (OOD) actions leading to erroneous Bellman backups.
Prior works tackle this problem via behavior regularization (Fujimoto et al. 2018b Kumar et al.) 2019 Wu et al.) 2019 Siegel et al. (020). This ensures that the learned policy stays "close" to the behavior policy. This is achieved by adding regularization term that calculates the f-divergence
between the learned policy and the behavior policy. Kernel Maximum Mean Discrepancy (MMD) (Gretton al. 2007), Wasserstein distance and KL divergence
widely used (Wu et al. 2019 The regularization term is either fixed (Wu et al. 2011), or tuned via dual gradient descent (Kumar
et al. 2019), applied using trust region objective (Siegel et al. 2020).

---

Under review as a conference paper at ICLR 2021
In this paper, we propose improvements to the Behavior Regularized Actor Critic (BRAC) algorithm presented in (Wu et al. 2019). To obtain the same, we observe that sample based estimation of di- vergence measures computationally expensive and prone higher variance. Therefore, to reduce variance, we derive an analytical upper bound on the KI. divergence measure as the regularization term in the objective function. Moreover, we show that current works that apply the regularization term i.c. the divergence measure, on the entire batch, end up distributing the penalty over all states in the batch in amounts inversely proportional the state's probability of occurrence in the batch. This needlessly restricts the deviation of highly explored states while allowing less explored ones to deviate farther leading to OOD actions. To address the same, we employ state dependent Lagrange multipliers for the regularization terms and automatically tune their strength using state-wise dual gradient descent. In addition, the performance of the learned agent trained using prior methods often deteriorates over the course of training. We found that if the learned Q function generalizes such that the gradient of the Q function w.r.t the OOD actions is monotonically increasing, behavior regularization fails to keep such actions within the support set. To mitigate this issue, we penalize the gradient of the Q function w.r.t the OOD actions by adding a gradient penalty term to the pol- icy evaluation objective. This reduces the policy improvement at OOD actions the problem of minimizing the divergence between the learned policy and the behavior policy.
We call our improved algorithm BRAC+ following (Wu al. 2019). Our experiments suggest that BRAC+ outperforms existing state-of-the-art model-free and model-based offline RI. algorithms in various datasets on the D4RL benchmark (Fu et al. 2020
BACKGROUND
Markov Decision Process RL algorithms aim to solve Markov Decision Process (MDP) with unknown dynamics. A Markov decision process (Sutton & Barto 2018) is defined as tuple < S, A, R, P, и >, where S is the of states, A is the set of actions, R(s, a, s') S A S + R defines the intermediate reward when the agent transitions from state s to s' by taking action a, A S + [0. 1] defines the probability when the agent transitions from state s to s' by taking action l : S + [0, [] defines the starting state distribution. The objective of reinforcement learning is to select policy æ 1 - P(A) to maximize the following objective:
P(s'|s,a) a)
((T)
E s0 ( 844 P()
R(s;, at 8(+1]
1=0
Offline Reinforcement Leaairnn The goal of offline RL is to learn policy no from a fixed dataset D {ss, s.rr)) r consisting of single step transitions {(s,, ((, s',,r:)); The dataset is as- sumed to be collected using a behavior policy ny which denotes the conditional distribution p(a[s) observed in the dataset. Note that Ti may consist of multi-modal policy distribution. In principle, standard off-policy RL algorithms using a replay buffer (Mnih et al.) 2013) Lillicrap et al. 2016
Haarnoja et al. 2018aa 8a) can directly learn from D. The key challenge resides in the policy evaluation step:
Qu vt arg ([Q (r(s,a) + +E (s', (s,,''')) (policy evaluation) (2)
In this step, the target Q value depends on the learned policy. the learned policy distribution my diverges from the data distribution Ts, it results in evaluation of target Q values using out- of-distribution (OOD) actions. Such evaluations are prone to errors. Erroneous overestimation of values get exploited by the policy improvement, preventing the algorithm from learning useful policies. In order to avoid such cases, behavior regularization is adopted to force the learned policy
to stay "close" to the behavior policy (Fujimoto et al. 2018b Kumar et al. 2019 Wu et al. 2019
IMPROVING BEHAVIOR REGULARIZED OFFLINE REINFORCEMENT LEARNING
In this section, we discuss and propose three non-trivial improvements to the Behavior Regularized Actor Critic (BRAC) offline reinforcement learning (Wu et al. 2019). BRAC (Wu al. 2019)
2

---

Under review as a conference paper at ICLR 2021
augments either the policy improvement or the policy evaluation step with penalty term to constrain the policy within the support set of the behavior policy. For simplicity of illustration, we only consider the policy improvement step (Wu et al. 2019):
max [(saars))
- (D(()(-ss)
(3)
Here, 1 is selected f-divergence (Csisáár 1972) measure used for behavior regularization. Zoom- ing into the regularization term, we get:
(4)
#ES
In the equation above, S denotes the states that occur in single sampled batch and P. is the probability of state 8 in the sampled batch. The first two improvements that we propose are targeted towards the following two terms in Equation 4 (I) The divergence term D which for each fixed m determines the impact actions have the overall objective conditional the state, and (2) the term (P,a) which determines the impact the "divergence penalty" of each state will have on the overall objective function. Note that this term is dependent upon the probability of the occurrence of state in the sampled batch. For (1), we propose use an analytical upper bound of KI. divergence to reduce variance. For (2), we propose decouple the impact of "divergence penalty" of each state from the probability of its occurrence. Detailed explanation of the two improvements and the rationale behind them follows in the next two sub-scctions.
3.1 REGULARIZZATION METHOD
Behavior regularization is used to constrain the learned policy with the support set of the behavior policy. In other words, it ensures that the "difference" between the probability distributions of the learned policy ne (-1s)) and behavior policy a is small. The following divergence measures most widely used in the community:
Kernel MMD Kernel Maximum Mean Discrepancy (MMD) (Gretton al. 2007) was first intro-
duced in (Kumar et al. 2019) to penalize the policy from diverging from the behavior policy:
MMD (=()s), z.zl/
(x)))
F
+
F
[v.'')) (5)
(-1((1))
v.y/
where K is a kernel function. Symmetric kernel functions such as Laplacian and Gaussian kernels are typically used (Gretton et al. 2007).
KL divergence For two probability distributions P and Q on some probability space x, the KL- Divergence from Q P is given as
1 P(x) log P(z) dr Q(x)
(P.(P
- x-
(6)
KL divergence is assymetric and both DRI. (-olss, TT (-1s) and DKI. (To(-1s), To(-ss) are valid. However, DxI. for each may assign (als)) 0 when ma(ass) 0, producing out of distribution actions. Therefore, DKL (Te(-)))))) s), To (-(ss) is used for regularization.
(mo(-ss), (xI.(ms(-1))))
(-(1s)) is not suitable because it requires Th (als) * 0 when (ajs)) * O,
3.1.1 ANALYTICAL KL DIVERGENCE UPPER BOUND
All the existing behavior regularized methods estimate the divergence via samples (Kumar et al., 2019 Wu et al. 2.19). While in theory it produces unbiased estimator, it requires a large number of samples to reduce the variance.
To stabilize the performance, the key idea is have low variance estimator for KL divergence. We obtain this by deriving an upper bound on the KL divergence between the learned policy no and the behavior policy ne that be computed analytically.
3

---

Under review as a conference paper at ICLR 2021
Assume we learn Vs using latent variable model (e.g. VAE (Kingma & Welling 2014) with latent variable Z. According to the evidence lower bound (ELBO), we obtain log (3(a)s) 2 Ezwg(z) [log p(als, 2)] - DxL. (9<)))l((2) where q(2) is the approximated posterior distribution and p(z) is the prior. Then, the KL divergence is bounded by:
DKL (TT( Eawao [log To Ea [lo (a|s)]
I Eartr llog To [a]s] Ea [ ( 2)] (xL(a()) Ea (Te (Tl(((((1)))l((((s)))))) + Dxi (xx(
Thus, we choose p(-|s, 2) be a tractable distribution, Equation can be computed analytically. Note that although p()18,=) has special form (e.g. Gaussian), the latent variable model can in theory represent any probability distribution without sacrificing the expressiveness. practice, we choose p(-1s, z) to be Gaussian distribution.
3.2 STATE DEPENDENT LAGRANGE MULTIPLIER
From the term (P,a) in Equation 4 one can infer that the "penalty budget" divergence mea- sure gets distributed among the states in amounts inversely proportional to the probability of the occurrence of the states in the sampled batch. If we assume the batches are sampled uniformly, this implies that the states which are been explored more thoroughly are restricted from deviating from the behavior policy. Whereas the less numerous state that have had limited exploration enjoy undesirable as restricting highly explored states is overly conservative while allowing less explored states deviate can lead OOD actions.
numerous in the offline dataset D i.e. the states that have
freedom for deviation. This is
To address this we add state wise constraint for KL divergence as follows:
max Ea- (s) s.t. (Kz((e((ss) EKL, Vs (policy improvement)
This translates to assignment of state dependent Lagrange multipliers when solved using dual gra- dient descent:
max E.D (Fa a - a(s) (ma((s)))))))) =.) (-(+))))]]
(8)
Maintaining a Lagrange multiplier for each state is impractical for large datasets. In practice, we parameterize (s)) with a neural network.
3.3 GRADIENT PENALIZED POLICY EVALUATION
The fundamental challenge in offline reinforcement learning is to mitigate the impact of erroneous values that evaluated ditt---distribution actions and used in policy evaluation. Due the limited representation capacity of neural networks, such actions are unavoidable for large datasets, even with state-dependent behavior regularization. CQL (Kumar et al. 2020 resolves this problem by optimizing a conservative lower bound of the Q value. The key idea in this paper is to bound the Q value the out-of-distributtin actions such that their values are not greater than the Q value of in-distribution actions. We achieve this by augmenting the policy evaluation step with gradient penalty regularization term. To elaborate our approach, we first analyze the gradient of the policy improvement step:
V
Va Q () amm Vaza(als) smama - Vos:)Vo (To((T)(-)s) +=(+1s)) (9)
If the current policy produces in such a way that the gradient of Q network is monotonically increasing, this leads to the unbounded value of the Q network and the failure of the behavior regularization. We created toy example to illustrate this phenomenon in Appendix A The analysis suggests that if we penalize the gradient of the network with respect to the out-of-distribution actions such that they close zero,
actions and the Q network erroneously generalizes
4

---

Under review as a conference paper at ICLR 2021
Algorithm BRAC+: Improved Behavior Regularized Actor Critic
1: Train the behavior policy Tg on the offline dataset D = {(s,, as ro 8,) via maximum
likelihood estimation: xg arg max Zi=1 log (aiils))
N
2: Train initial policy: نe arg min elll (TL TT)
3: Train initial Q network: Qs arg min (Qu(s, (r(s, a) + FE (u
4: for 1: do
5: for = T
6:
Update network using Equation 10
7: Update the policy using Equation 7
8: Update via dual gradient descent: (( f- a(s) Ao (To((s)))))) =p((1)) oKL)
9:
Update 3 via dual gradient descent: B t B + As (s(H((T(((ss) Ho)
Update the target network v = TUh + (1 r)
11: end for
end for
performing policy improvement step simply reduces to minimizing the KL divergence. Inspired from (Gulaaaani et al. 2017). we add gradient penalty term to the policy evaluation step as:
min E (s. ((s.))) r(s,a) a) + t Qu (s', d'))
AF (1 QuV ("ll,""")lll =s())))))))))))))))))))))))))))) <GP)
(10)
where f is a non-decreasing function and EGP is the threshold for gradient penalty. In this paper, we set f to be the Heavis siavissid step function (or indicator). Other variants such as its soft version (sigmoid function) is left for future work. Setting EGP EKL is too conservative because it prevents generalization. In practice, we set EGP at epoch t to be PKL + OKL, where PKL and øиl the mean and standard deviation of the KI. divergence for all the states in the dataset epoch t 1. In addition, we found that À = 0.1 works well for all the tasks.
4 RELATED WORK
We briefly summarize prior works in offline RL and discuss their relationship with our approach. As discussed in Section D the fundamental challenge in learning from static data is to avoid out-of-distribution actions (Levine et al., 2020). This requires solving two problems:
estima- tion of behavior policy, 2) quantification of out-of-distribution actions. We follow BCQ (Fujimoto et al. 2018b), BEAR (Kumar al.) 2019) and BRAC (Wu al.] 2019) by learning the behavior policy using conditional VAE (Kingma & Welling 2014). To avoid out-of out-of-iistribution actions, BCQ generates actions in the target values by perturbing the behavior policy. However, this is over-pessimistic in most of the cases. BRAC (Wu et al. 2019) constrains the policy using vari- ous sample-based. f-divergence measures including MMD, Wasserstein distance and KL divergence with penalized policy improvement policy evaluation. BEAR (Kumar et al. 2019) is instance of BRAC with penalized policy improvement using MMD (Gretton et al.) 2007). Sample-based estimation is computationally expensive and suffers from high variance. In contrast, our method uses an analytical upper-bound of the KL divergence to constrain the distance between the learned policy and the behavior policy. is both computationally efficient and has low variance. (Siegel et al. 2020) solves trust-region objective instead of using penalty. CQL (Kumar et al.) (0200 avoids estimating the behavior policy by learning a conservative Q function that bound- value. Hyperparameter search is another challenging problem in offline RL. (Lee et al. 2020) uses a gradient-based optimization of the hyperparameter using held-out data. MOPO (Yu et al. (020) follows MBPO Janner et al. (2019) with additional reward penalty unreliable model-generated transitions. MBOP (Argenson & Dulac-Arnold 2020) learns the dynamics mode, the behavior policy and truncated value function perform online planning. (Kidambi al. 2020) learns surrogate MDP using the dataset, such that taking out-of-distribution actions transit to the terminal state. The aat-of-distributin actions are detected using the agreement of model ensembles.
true
5

---

Under review as a conference paper at ICLR 2021
5 EXPERIMENTS
Our experiments aim to answer the following questions: 1) How does the performance of our improvements compare with state-of-the-art model-free and model-based offline RL methods? 2) How does the use of analytical variational upper bound on KL divergence for regularization term compare with sampled-based MMD? 3) How does the state-dependent Lagrange multiplier based regularization (state-wite wise regularizor) compare with global policy regularization? 4) Does the gradi- ent penalized policy evaluation improve the stability during the training? To answer these questions, evaluate methods locomotion tasks (hopper, walker2d, and halfcheetah) and four types of datasets: 1) random (rand): collect the interactions of a run of random policy for IM steps create the dataset, 2) medium (med): collect the interactions of a run of medium quality policy for IM steps as the dataset, 3) medium-expert (med-exp): run medium quality policy and an expert quality policy for 1M steps, respectively, and combine their interactions to create the dataset, 4) mixed (medium-replay) train policy using SAC (Haarnoja et al. 2018a) until the performance of the learned policy exceeds pre-determined threshold, and take the replay buffer as the dataset. In addition, we consider more complex Adroit tasks (Rajeswaran et al. 2018) that requires controlling a 24-DoF robotic hand, using limited data from human demonstrations.
subset of the D4RL. (Fu et al., 2020) benchmark. We consider three
We compare against state-of-the-ee model-free and model-based baselines, including behavior cloning, BEAR (Kumar et al.) 2019) that constrains the learned policy within the support of the behavior policy using sampled MMD, BRAC-p/v (Wu et al.) 2019) that constrains the learned pol- icy within the support of the behavior policy using various sample-based f-divergences penalize either the policy improvement (p) or the policy evaluation (v), CQL((H) (Kumar et al. 2020) that learns Q function that lower-bounds its true value. We also compare against model-based ap- proaches including MOPO (Yu et al. (2020) that follows MBPO (Janner et al. (219)) with additional reward penalties and MBOP (Argenson Dulac-Armodd 2020 that learns an offline model to per- form online planning.
5.1
COMPARATIVE RESULTS
Table I: Results for OpenAl gym (Brockman al., 2016) environments in the D4RL (uu et al., 2020) datasets. For each task, we train for I million gradient steps and report the performance by running the policy obtained at the last epoch of the training for 100 episodes, averaged over 4 random seeds with standard deviation. Each number is the normalized score as proposed in (Fu et al. 2020 Please refer (Fu et 2020) for results on more baselines.
Model-Free
Model-Based
Task Name
BEAR
BRAC-p/v
CQL(H)
BRAC+ (Ours)
MOPO
MBOP
halfcheetah- rand
25.1
24.1/31.2
35,4
1..4
31.9
6.3:t4.0
walker2d-rand
7.3
-0.2/1.9
7.0
16.7: 2.3 12.5+H0.3
13.0:2..6
8.1 5.5
hopper-rand
11.4
11.0/12.2
10.8
13.3+-1.6
10.8 0.3
halfcheetah-mee
41.7
43.8/46.3
44.4
46,6110.6
40.24 2.7
44.610.8
walker2d-med
59.1
77.5/81.1
79.2
75.113..5
14.0=10.1
41.0 29.4
hopper-med
52.1
32,7/31.1
58.0
53.243.1
26.5+:3.7
48.8 + 26.8
halfcheetah-mmed-xx
53.4
44.2/41.9
62.4
61.2.1.88
57.91 24.8
105.91 17.8
walker2d-med-exp
40.1
76.9/81.6
98.7
95.31 5.9
55.0.L 19.1
70,2 1 36,2
hopper-med-exp
96.3
1.9/0.8
111.0
112.9+0.11
51.7=42.9
55.1 = 44.3
halfcheetah-mixxd
38.6
45.4/47.7
46.2
46.1
54.0.12.6
42,3 0.9
walker2d-minn mixed
19.2
-0.3/0.9
26.7
39.0
42.7+8.33
9.7 5.3
hopper-me mixed
33.7
0.6/0.6
48.6
72.7+18.9
92.5
12.4 5.8
Performance on multi-modal datasets We first compare the performance on multi-modal datasets i.e. med-exp and mixed datasets. Results shown in Table T that our method outper- forms various model-free baselines on most of the multi-modal datasets, especially on hopper-mix
Our code will be released at wtt.h/hs/gtthhbb.coo..

---

Under review as a conference paper at ICLR 2021
Table Results for Adroit tasks with human demonstrations in the D4RL Fu al. 2020) datasets. The numbers are reported by following the same procedure as in Table except we run the policy obtained the last epoch of training for 1000 episodes due large variance across different runs.
Task Name
BC
BEAR
BRAC-p/v
CQL(H) CQL(p)
BRAC+ (Ours)
pen-human
34.4
-1.0
8.1/0.6
37.5
55.8
64.9 1.6 3,9 + 0,9
hammer-human
1.5
0.3
0.3/0.2
4.4
2.1
door-human
0.5
-0.3
-0.3/-0.3
9.9
9.1
11.5# 1.2
relocate-human
0.0
-0.3
-0.3/-0.3
0.20
0.35
0.20 + 0.11
and walker2d-mix by up to 1.5x. Compared with BEAR (Kumar et al. 2019), our performance
improvement arises from the advantage of the KL divergence over the kernel MMD (see discussions in Appendix B). The choice of parameters in CQL(H) makes it too conservative to achieve higher performance.
Performance on single-modal datasets The performance of our method on single-modal (rand and med) dataset outperforms or matches with baseline methods evident from Table [ except halfchectab-rradong dataset. We observe that the performance is very sensitive to the choice of target policy entropy. We hypothesize that the our choice of target policy entropy in the halfcheetah- random task makes it hard to compose correct sub-optimal policies from random collected dataset.
1w12
pen
80
1.0
70
weo gP
660
so
AverggeQ
0.6
40
20
0.4
20
100
200
100
400
500
100
200
300
400
500
Epoch
Epoch
Figure I: Left: learning curve of pen-human-vie task. Right: the average Q value of the first ensem- ble over the course of training. Both curves are smoothed by a factor of 20.
Performance datasets with human demonstrations The performance Adroit tasks is shown Table These tasks are substantially harder than OpenAl gym tasks due to limited training data in a high dimensional observation and action space. Our method makes non-trivial improve- ment over the behavior cloning. Compared with the state-of-the-an approaches, our approach is superior half of the tasks and matches the asymptotic performance on the remaining ones. Fig- ure shows that the Q value is bounded when the gradient penalized policy evaluation technique is employed. On the contrary, the Q value without the gradient penalized policy evaluation increases exponentially, Note that we use MMD with Laplacian kernel for Adroit tasks. We observe that the Kl--based regularization struggles with datasets collected with narrow behavior distributions (have large density within a tiny space and almost zero density anywhere else). In such a case, the KI. divergence is sensitive to tiny policy changes, making gradient- based optimization hard to converge.
5.2 ABLATION STUDY
To answer question (2), (3) and (4), we conduct a thorough ablation study on BRAC+ on various tasks with different data collection policies (hopper-mixed, walker2d-medium--oon halfcheetah- medium).
Sampled based MMD analytical upper bound KL The results of using sampled-based MMD versus analytical upper bound KL are shown in Figure 2a The difference of the perfor- mance in the single-modal dataset saaffhheetah-mediing is negligible. However, the performance
7

---

Under review as a conference paper at ICLR 2021
hopper-mediune
walker2d medium expertivo
halfcheetah medium-v0
120
100
45
Testiphe
40
-35
20
20
25
mmd
-20
20
forward kid
100
200
400
200
400
500
1.00
200
100
500
Epoch
Epoch
(a) MMD vs. forward KL. Global Lagrange multiplier. No gradient penalty.
hopper rop v0
expert
100
m
global
100
glocal
45
Normalizedesstpnee
NormaliedTesilpee
CormaizedTesttatee
40
60
60
33
30
0
global
20
500
Epoch
Epoch
Epoch
(b) Global vs. state-wise Lagrange multiplier. Forward KL. No gradient penalty.
hopper-mediune uv-
wolker/o-medium--xppe..
hullcheetal-mediun--o
gp
vi gp
mic gp
wile go
NomalloedTette CamalloadTontsphee Tontepptte
Normalizedt Testipptt
80
-20
NormalassTerttooo
60
60
60
30
SP
100
200
300
400
500
100
200
300
400
500
100
200
300
400
500
Epoch
Epoch
Epoch
(c) Gradient penalty No gradient penalty. State-wise Lagrange multiplier. Forward KL.
Figure 2: Figures of ablation study. Each setting is repeated for random seeds. The curve is the mean and the shaded area the standard deviation. The curves smoothed by a factor of 20. The number of gradient steps per epoch is 2000. To make fair comparison, we only substitute KL diver- gence with MMD-based measurement with additional MMD-specifie hyperparameter tuning. The other design choices are different from (Kumar et al. 2019). Details can be found in Appendix D
on multi-modal datasets is varied. Our toy example in Appendix suggests that MMD and the backward KL divergence tends cover all the "modes" in the behavior policy while the forward KL divergence tends to seek one the "mode" in the behavior policy. Note that this argument is only valid if the learned policy is single-modal (e.g. Gaussian distribution). The superior performance when using forward KL regularization supports this argument. The
in the results in hopper-mixed task seems be contradictory. We hypothesize that since the analytical KL actions when available while in
has low variance, the policy quickly adopts the sample- based methods, such adoption is slower due to higher variance in the policy regularization.
State-win wise vs. global regularization The performance of using global and state- wise regulariza- tion on various tasks is shown Figure 2b It is noticeable that the state-wise regularization im- proves the performance in hopper-mixed and walker2d-medium expert task, while the performance halfcheetah salicheetah-meddum does not show much improvement. To understand the consequence of the state-wise wise regularization, plot the histogram of the KI. divergence of all the states in the dataset
8

---

Under review as a conference paper at ICLR 2021
hopper medium
walker2d medium expertivo
halfcheetah medium
0.25
0.5
0.4
Denaby
0.3
Denstyy
0:35
0.2
0.2
0.30
0.1
0.1
0.0
0.00
10
20
15
divergence
divvergencce
divergencee
Figure 3: Histogram of the KL divergence of all the states in the dataset with global and state-wise Lagrange multiplier.
in Figure g The KI. divergence histogram of using state-wise regularization more concentrated around the threshold while the global regularization is more sporadic. Strictly enforcing the policy regularization around the threshold helps avoid cutoo--distrbbution actions, that is often caused by a few states in the dataset.
Gradient penalty vs. no gradient penalty Even with policy regularization, evaluating the Q value the out-of-distribution actions can't be fully avoided. Thus, important to bound the Q value at the out-of-distribulie distribution actions. FigureZc|shovs the performance with and without gradient penalty in the policy evaluation step. While there is little difference in the hopper-mixed and the aificheetah-eedinne task, the performance policy evaluation. On the contrary, the performance deteriorates over time without gradient penalty. Please refer to Appendix D for more analysis.
actions such that their values are not greater than the Q value of in-
the
expert task stabilizes with the gradient penalized
DISCUSSIONS AND LIMITATIONS
There are several limitations of our approach. The discussion in Appendix D suggests that although KL-regularized offline policy optimization is good at combining sub-optimal policies. they may stuck at local optimums; and they are hard to escape. Another drawback is that the threshold value is hard set. In our experiments, we try few numbers that are above the minimum possible KI. threshold, which is obtained by training a policy that minimizes that KI. divergence.
Finally, we conjecture that the behavior-regularizz approach is not sufficient to tackle offline RL problems since it fully ignores the state distribution. To see this, we can create a dataset that only adds few trajectories from expert policy dataset collected by low-quality policy. If the low-quality policy doesn't 't visit the "good" states in the expert policy (can't combine sub-optimal policies), nehavior-regularized approach leads a policy that imitates the expert policy. Such imita- tion is likely to fail due to compounding errors (Ross et al. 2010)). The right approach for this dataset to completely ignore expert trajectories and combine sub-optimal policies in the low-quality re- gions. To achieve this, we need to consider the state distribution as the density of the "good" states is very low.
7 CONCLUSION
In this paper, we improved the behavior regularized offline reinforcement learning by proposing a low-variance upper bound of the KL divergence estimator to reduce variance, state-dependent Lagrange multiplier to allow more freedom of deviation to high probability states while restricting low probability states and gradient penalized policy evaluation such that the Q values of out-of- distribution actions are not greater than those of in-distribution actions. Our experimental results on challenging benchmarks illustrate the benefits of our improvements.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Abien Fred Agarap. Deep learning using rectified linear units (relu). ArXiv, abs/1803.08375. 2018.
Arthur Argenson and Gabriel Dulac-Arnold. Model-based offline planning. ArXiv, abs/2008.0553 2020.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym, 2016.
Sungwoon Choi, Heonseok Ha, Uiwon Hwang, Chanju Kim, Jung- Woo Ha, and S. Yoon. Reinforce- ment learning based recommender system using biclustering technique. ArXiv, abs/1801.05532. 2018.
I. Csiszár. A class of measures of informativity of observation channels. Periodica Mathematica Hungarica, 2(1):1912233, 1972. doi: 10. 10.1007/BF02018661 URL https:://60..org/11 10C7/BP02018661
R. Fox. Toward provably unbiased cemporal-difference value estimation. 2019.
Justin Fu, Aviral Kumar, Ofir Nachum, G. Tucker, and Sergey Levine. D4rl: Datasets for deep
data-driven reinforcement learning. ArXiv, abs/2004.07219, 2020.
Scott Fujimoto, H. V. Hoof, and David Meger. Addressing function approximation enror in actor- critic methods. ArXiv, abs/1 (bs//182.09977,, 1a.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. CoRR,
2018b. /R..ttt::/aaxx...or//aaaaa
02.000
Arthur Gretton, Karsten M. Borgwardt, Malte Rasch, Bernhard Scholkopf, and Alexander J.
Smola. A kernel approach to comparing distributions. In AAAI, pp. 1637-1641, 2007. URL
http:// htt:://wwwt http://ww.aaa.aa....... aaai.
hww.h/ararrrrraaaaaaaaaaaaaaa.aa http://AAA///0///aa// 07 26..-262.ppp
Ishaan Gulrajani, F. Ahmed, Martín Arjovsky, Vincent Dumoulin, and Aaron C. Courville. Improved training of wasserstein gans. In NIPS, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. CoRR, abs/1801 1101.01290.
8a.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algo- rithms and applications. CoRR, 1bs/1982..0595.. 2018b. URL http::/arxiv. wwwllivorrrreee org.abs.//......oooo... /
1812.05900
I. Higgins, Loic Matthey, A. Pal, C. Burgess, Xavier Glorot, M. Botvinick, S. Mohamed, and Alexander Lerehner. beta-vae: Learning basic visual concepts with constrained variational framework. In ICLR, 2017.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model- based policy optimization. CoRR, abs/1906.08253, 2019. URL htllht:tp///xxiv.rrgg /bs/ 1906. 08253
Liyiming Ke, M. Barnes, W. Sun, Gilwoo Lee, Choudhury, and S. Srinivasa. Imitation learning as f-divergence minimization. ArXiv, abs/1905. 12888, 2019.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel Model- based offline reinforcement learning, 2020.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980. 2015.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. CoRR, Abs/1311.66144 2014.
10

---

Under review as a conference paper at ICLR 2021
Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off aff-ppolicy q-learning via bootstrapping error reduction. CoRR, abs/1906.00949, 2019. URL http:: /tttt////aaxx.
Aviral Kumar, Aurick Zhou, G. Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. ArXiv, abs/2006.04779, 2020.
Byung-Jun Lec, Jongmin Lee, Peter Vrancx, DongHo Kim, and Kee-Eung Kim. Batch reinforce- ment learning with hyperparameter gradients. 2020.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. ArXiv, abs/2005.01643 2020.
Timothy P. Lillicrap. Jonathan Hunt, Alexander Pritzel, Nicolas Manfred Otto Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learn- ing. CoRR, abs//509 10111509009999 02911,991111. 2016.
Long-Ji Lin. Reinforcement Learning for Robots Using Neural Networks. PhD thesis, USA, 1992.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. CoRR, abs/1312.5602, 2013. http:/tp///errt..orrrraaaaaa///a
A. Rajeswaran, V. Kumar, Abhishek Gupta, John Schulman, E. Todorov, and S. Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. ArXiv, abs/1709. 10087, 2018.
Stéphane Ross, Geoffrey Gordon, and Andrew Bagnell. No-regret reductions for imitation learning and structured prediction. CoRR, abs/1011.0688 2010. URL /tt/ttp:: http::/aaxxv..rrz
0686
Noah Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, T. Lampe, Roland Hafner, and Martin A. Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. ArXiv, abs/2002.08896, 2020.
David Silver, Aja Huang, Christopher Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schritwieser,, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanetot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lilli- crap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 529:484-503 2016. htLhttp: htmll
//www. nature. com/ nat http:///./../..oooo//o//////////
Richard S.
Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. Bradford Book, Cambridge, MA, USA, 2018. ISBN 0262039249.
Oriol Vinyals, Igor Babuschkin, Junyoung Chung, Michael Mathieu, Max Jaderberg, Wojtek Czarnecki, Andrew Dudzik, Aja Huang, Petko Georgiev, Richard Powell, Timo Ewalds, Dan Horgan, Manuel Kroiss. Ivo Danihelka, John Agapiou, Junhyuk Oh, Valentin Dal- ibard, David Choi, Laurent Sifre, Yury Sulsky, Sasha Vezhnevets, James Molloy, Trevor Cai, David Budden, Tom Paine, Caglar Gulcehre, Ziyu Wang, Tobias Pfaff, Toby Pohlen, Dani Yogatama, Julia Cohen, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy Lil- licrap, Chris Apps, Koray Kavukcuoglu, Demis Hassabis, and David Silver. AlphaStar: Mastering the Real-Time Strategy Game StarCraft II. /tttps://deepmind.... ctm/blog//////////occ/oooooooocc stw.tssy-attttteeeeeeeee
a alphasttar-mattriing-- otw.://wwo....ee.ee.eeeeeeeeeeen stwrcraftiira/aaoaaaaaaa 2019.
Ziyu Wang, A. Novikov, Konrad Zolna, Jost Tobias Springenberg, Scott Reed, B. Shahriari, N. Siegel, Josh Merel, Caglar Gulcehre, Nicolas Heess, and N. D. Freitas. Critic regularized regression. ArXiv, abs/2006. 2020.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. 2019.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization, 2020.

---

Under review as a conference paper at ICLR 2021
Chi Zhang, Sanmukh R. Kuppannagari, Rajgopal Kannan, and Viktor K. Prasanna. Building hvac scheduling using reinforcement learning via neural network based model approximation. Pro- ceedings of the 6th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation, BuildSys '19, pp. 287-296, New York, NY, USA, 2019. Associa- tion for Computing Machinery. ISBN 9781450370059, doi: 00.1145/33603222333606 URL
12

---

Under review as a conference paper at ICLR 2021
V bain. 5
10.0
ground truth
7,5-
w/o gP
distribution, ttraiinn data:
5.0
w/ gP
Zero gradient
Erromeous genoralization
2.5
-25
Zero gradient
Errorneous generalization
-5.0
Correct generalization
-7.5
110.0-
1.0
-1.
0.0
1.0
Figure 4: Fitting a regression model with out of distribution data
A TOY EXAMPLE TO DEMONSTRATE BOUNDING THE VALUE OF THE OUT OF DISTRIBUTION INPUTS VIA GRADIENT PENALTY
To demonstrate the effectiveness of the gradient penalty to enforce the bound of the predicted values of regression model, we conduct experiments by fitting regression model with in-distribution data while minimizing the norm of the gradient at out of distribution inputs. Specifically, we generate dataset ((ri, yt) Ji 1-N=100 as r; U0-0.8, 0.8), y/ 5in F.r:) 6i, where E; N 0.5). In of distribution dataset { as f U(-2, 0.8) U (O.88, 2). We fit regression model f. represented as a two-layer feed-forward neural network. The size of the hidden layer 64 and the activation is RELU. We use Adam (Kingma & Bal (205) optimizer with learning rate 0.01. In addition to the standard MSE loss, we add a gradient penalty term inspired from (Gulrajani et al. (017)) such that the gradient at the out of distribution inputs penalized. The overall loss function 18:
addition, we generate
N [((o)) L N
M Σ J=1
vi)² A
(11)
M
In our experiments, set À 0.1. We fit with and without the gradient penalty term. Figure shows the predicted value at both in-distribution and that: 1) both models generate good prediction for in-distribution inputs ([-0.8, 0.8)). which is be- tween the two red lines 2) both models generalize well at the out-of-distribution regions between the red line and the blue line ([-1, -0.8) and [0.8, 1). 3) both models erroneously generalize be- yond the blue lines ([-2.-] and [1, 2)) However, the value of the model trained without gradient penalty keep on increasing or decreasing while the model trained with gradient penalty has zero gradient. This suggests that with proper out-of-distributtin regularization, the gradient direction at the out-of-dine out-of-disttibution inputs point to the in-distribution regions.
inputs. The results suggest
B VISUALIZATION OF VARIOUS PROBABILITY DIVERGENCE MEASUREMENT
To understand the impact of forward KL (DKI (PKa (T))) backward KL (Dxz, ( and MMD distance in constraining the learned policy within the support set defined by the behavior policy, we create toy example as shown in Figure [ The black curves in the two figures represent the behavior distribution TT. The orange, blue and green curves in the graph show forward KL, backward KL and MMD with Laplacian kernel between π and
. Nz, σ), respectively, where is variable between [-10, 10] and a fixed. If we draw a horizontal line (red), the area below this line defines the support set. In the left figure, the behavior policy is single-modal and there is little difference between the three metrics. However, when the behavior policy is multi-modal the forward KI. divergence contains two bottoms, each corresponds to the peak probability density of
13

---

Under review as a conference paper at ICLR 2021
forward_kid
backward_ _ld
mmd_ lan_llal
Divergence
10?-
0.4
0.2
0.2
100
0..0
-0.0
10
10
Figure 5: The black curve in the two figures represents the behavior distribution тτ. The orange, blue and green curves in the graph show forward KL, backward KL and MMD with Laplacian kernel between TT and ma = MNr,,)) respectively, where ar a variable between [-10, 10) and a is fixed.
the behavior policy. However, in backward KL and MMD distance, the policy with the smallest divergence actually has low probability density in the behavior policy. This because minimizing the forward KL leads to "mode seeking" while minimizing the backward KL and MMD leads to "mode covering". ". These two terms are heavily used when talking about generative models such VAES and GANs (Ke et al. 2019). When performing offline RL in multi-modal dataset, it is necessary to combine sub-optimal actions in the behavior policy, which is in fact "mode seeking". Mode covering typically leads to dfitt--diittiuution actions even when the divergence measurement is small (e.g. the area between the two modes are actually out-of cit-o--distribution) while mode seeking may be stuck at a local optimum (e.g. moving across different "modes" is hard).
In the left figure, Fe = (0, 1). In the right figure, is mixture of two Gaussian distributions: and 0.,0. The weight of each component is 0.3 and 0.7, respectively, a set to
0.2.
C MISSING BACKGROUND
C.I VARIATIONAL. AUTO-ENCODER
variational auto-en (VAE) (Kingma Welling
2014) is generative model that aims to learn
the data distribution p(X) given a set of observations {z; LN 1- While directly optimizing p(X) is
intractable, we can optimize its evidence lower-bound (ELBO):
p(X) El p(X|=)] DRL. (a())(())l(()))
(12)
where 2( is the variational distribution and p( Z) is the prior. In VAE, g(Z) is (Z so that an auto-encoder. Optimizing Equation [2] using gradient descent requires back-propagate the gradient through sample operator. Fortunately. if the latent variable is multivariate Gaussian dis- tribution, we can use re- re-parametrization trick. The tightness of the upper bound is the KL divergence between the approximated posterior distribution and the true posterior distribution.
D MISSING ABLATION STUDY
For fair comparison between MMD and KI. divergence in policy regularization, we only substitute KL divergence with MMD-based measurement with additional MMD-specific hyperparameter tun- ing. the original implementation (Kumar et al., 2019), the author uses ensembles of Q network and compute the Q value by convex combination of both the minimum of the ensembles and the maximum the ensembles. In our implementation, we follow the standard architecture in the online setting: only use 2 ensembles of Q networks and compute the Q value as their minimum. In this work, we only compare against MMD with Laplacian kernels. The MMD-specific hyper-parametent shown in Table 3
14

---

Under review as a conference paper at ICLR 2021
Difference in training the behavior policy (Kumar al. 2019) learns the behavior policy as a 3-VAE (Higgins et al. 2017) with MSE reconstruction loss. This is equivalent to maximizing the log probability of Gaussian distribution with fixed variance the decoder output. However, the variational lower bound does not hold in 3- V-V that breaks our derivation. this paper, we learn the behavior policy regular VAE. The decoder outputs Gaussian distribution with input- conditioned mean and variance.
Gradient penalized policy evaluation Figure suggests that the L.2 norm of the gradient at the outoof-distribution actions with the gradient penalty much lower than that without the gradient penalty.
gooppe--edium-eoe replay--v
walker2d-meddum expert-v0
hatlcheetah--edduu--
2.00
wits
0.5
Wio gp
1.4
w/o
1.75
1.2
1.0
Densty
1.25
Denaty
Denndyy
0.8
0.6
0.75
0.4
0.25
0.2
0.00
0.0
0.0
Gradient Norm
Gradient Norm
Gradient Norm
Figure 6: Histogram of the L2 norm of the gradients for fiftof-distribution actions
Out-of-distribution generalization Lastly, we discuss the out-of-distribution generalization. In offline RL, we rate the learned policy with three levels. Level policies are able to strictly follow the behavior policy. Level II policies policy. Level III policies are able to generalize to out-of-distribution actions. To visualize the out- of-distribution generalization, we plot the KL divergence of the learned policy the testing distribution in Figure 7 In the hopper-mixed and halfcheetah- suggesting that the agents visit states which are the training distribution test time. Although the results suggest that such generalization is correct, our approach is unable to explicitly quantify it. Thus, we leave it as future work.
able to combine sub-optimal policies in the bahevior
task, there are periodic spikes,
hopper-medum-rggee [c)=3)
walka/26-medlum vD (5)=5)
hatfcheetah medium-vO (Ea
22
200
600
600
400
000
30000
200
500
B00
1000
Figure 7: The KL divergence between the learned policy and the behavior policy the testing data distribution. The æ axis the step an episode.
Performance sensitivity to the choices of target entropy During our experiments, we found that
the performance using the KL divergence as the behavior regularization method very sensitive to the choice of the target entropy. Policies trained with larger entropy may not be able to capture the narrow expert distribution within the behavior policy distribution. This phenomenon is most typical in the wwwichettah-mddim--xpppeee dataset depicted in Figure
15

---

Under review as a conference paper at ICLR 2021
haifcheetah expert-vO
Hm-15
NarraibedTast
20
10
300
400
500
Figure 8: The performance of the halfcherrah-meddn-experr task on various choices of the target entropy.
Table 3: MMD-speciie Hyper-paramen parameters
Hlype-ppaameee
hopper-medium-replay
walker-mecdum-eeper
halfchectah--mddiug
o in Laplacian kernel
10
20
10
ÉMVD
0.2
0.2
0.3
Number of samples
10
10
10
E MISSING RELATED WORK
For KL regularized policy improvement with fixed temperature, the optimal policy has closed form solution [Wang el al. 2020). Although it avoids estimating the behavior policy, the issue with fixed temperature is that all the states are penalized equally. This leads to over-pessimistic for states with higher occurence in the dataset and over-optimistic for states with lower occurence. (Fox) 2019) presents closed-form expression for the regularization coefficient that completely eliminates value updates. However, the so stmaxx operator introduced by the
the bias in approach makes it hard use in continuous action space.
IMPLEMENTATION DETAILS
Computation of the analytical KL upper bound To reduce the variance when computing the analytical KL upper bound, we sample L latent variable and compute the average of the L KL. upper bounds. our experiments, we set = Note that it doesn't 'tt reduce the bias of the upper bound.
Reward scaling Any affine transformation of the reward function does not change the optimal policy of the MDP. In experiments, rescale the reward [0,
(r Tmin (r Tmin)
(13)
where mmar and Tiin is the maximum and the minimum reward in the dataset.
Entropy regularization The KL divergence is the sum of negative entropy of the learned policy plus the cross entropy between the learned policy and the behavior policy: (mo(-ss)) vT(+(s)) -H(mT(-Is)) + H- (-)s)) When the learned policy distribution violates the KL con- straints, the KL divergence between the policy distribution and the behavior distribution is decreased by the optimizer. This equivalent to increasing the entropy of the learned policy and decreasing the cross entropy between the learned policy and the behavior policy. In soft actor-critic (Haarnoja cet al., 2018a), the minimum entropy the learned policy is enforced to encourage exploration. However, due to the absence of exploration, stochastic policy with large entropy will sample out-of- distribution actions when computing the target Q values in Equation 2 If such values are overesti- mated, the policy will exploit the erroneous values when performing the policy improvement in Equation
& and lead to failure, which can't 't be corrected without more data. Thus, we maintain the maximum entropy of the learned policy using the technique proposed in (Haarnoja et al. 2018b).
16

---

Under review as a conference paper at ICLR 2021
Initialization If the dataset is collected using narrow policy distribution in high dimensional space (c.g. human demonstration), the constrained optimization problem using dual gradient descent finds it difficult to converge if random initialization is used for the policy network. To mitigate this issue, we start with a policy that has the minimum KI. divergence with the behavior policy: To arg min emlll DRI( πT), where IT represents a family of policy types. In this work, consider II as Gaussian policies. Correspondingly, we initialize the Q network to Q
Policy network Our policy network is a 3-layer feed-forward neural network. The size of each hidden layer is 512. We apply RELU activation (Agarap) 2018) after each hidden layer. Following (Haarnoja et al. 2018a), the output is Gaussian distribution with diagonal covariance matrix. We apply tanh to enforce the action bounds. The log-likelihood after applying the t tanh function has a simple closed form solution. We refer to (Haarnoja et al. 2018a) Appendix for more details.
Q network Following (Haarnoja al.) 2018a Fujimoto al. 2018a Wu al. 2019), train two independent Q network iQ viz to penalize uncertainty over the future states. We maintain target Q network {Qwr, , Qey J with the same architecture and update the target weights using weighted sum of the current Q network and the target Q network. When computing the target Q values, we simply take the minimum value of the two Q networks:
(s', a')
min 1.2
(6',(6'00)
(14)
Each Q network is a 3-layer feed-forward neural network. The size of each hidden layer is 256. We apply RELU activation (Agarap 2018) after each hidden layer.
Behavior policy network Following the previous work (Fujimoto et al.
201 8b; Kumar et al,,
2019), we learn conditional variational auto-encoder (Kingma & Welling 2014) as our behavior
policy network. The encoder takes a pair of states and actions, and outputs Gaussian latent variable Z. The decoder takes sampled latent code and states, and outputs a mixture of Gaussian distribu- tions. Both the architecture of the encoder and the decoder is 3-layer feed- ferwadd neural network. The size of each hidden layer is 512. The activation relu (Agarap) 2018). To avoid epistemic uncertainty, train B ensembles of behavior policy networks. At test time, randomly select one model to perform the calculations. We found B = is sufficient for all the experiments. We pre-train the the behavior policy network for 400k gradient steps.
The a network takes in a state and outputs the Lagrange multiplier for the state. The architecture of the a network 3-layer feed-forward neural network with relu (Agarap) 2018) activation. The size of each hidden layer is 256. We use softplu activation after the output to ensure that all the values are positive.
network
Table 4: Default
Hyper-parameter
Value (Gym/Adroit)
Optimizer
Adam Kingma Ba 2015)
Policy learning rate
18-6/5e-8
Q network learning rate
3e-4
learning rate batch size
1e 5/1e-7
100
Target update rate 7
1e-3
Discount factor y
0.99
Initial 3
10
B learning rate
1e-3
Steps per epoch T
2000
Number of epochs
500
17

---

Under review as a conference paper at ICLR 2021
Table 5: Task-sp aaper-parameters for OpenAl gym tasks.
Task name
KL divergence threshold EKI.
Maximum entropy Ho
halfchectah-rand
-3
walker2d-rand
0.1
hopper-rand
-3
halfcheetah-med
-12
walker2d-med
2.1
-9
hopper-med
2.4
-6
haffcheetah--med-exp
11.5
-24
walker2d-med-exp
5
-12
hopper-med-exp
2.6
-6
halfcheetah-mine mixed walker2d-mixe
-12
hopper-mixed
-3
Table 6: Task-specific hyper-parametedr for Adroit tasks with human demonstrations.
Task name
MMD threshold EMMD
Minimum entropy Ho
pen-human
0.06
-200
hammer-human
0.1
-60
door-human
0.1
-60
relocate-human
0.1
-60
18


Reviews:
[{"review_id": "pzfgB4iHJXQ", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a number of improvements to the standard behavior-regularized offline RL paradigm introduced in BRAC (Wu 2019). Specifically, the paper proposes (1) using an upper bound on the KL regularization to induce lower-variance updates, (2) state-dependent Lagrange multipliers for the regularization, (3) adaptive causal entropy regularization, (4) behavior-cloned policy initialization, and (5) policy evaluation masking. The paper combines these improvements to propose BRAC+ and presents empirical demonstrations showing favorable performance.\n\nStrengths:\n\n-- Given the many improvements over BRAC suggested, the paper is organized well, and generally easy to follow. \n\n-- The derivations used in improvement (1) are novel as far as I can tell. These seem useful, and I am not aware of previous RL work doing this.\n\n-- The reasoning and motivation behind improvement (2) is especially compelling.\n\nWeaknesses:\n\n-- The derivations for improvement (1) suggest that the behavior policy should be given in variational form (i.e., trained as a VAE). However, the \"Initialization\" section suggests the behavior policy is actually a Gaussian. So how is improvement (1) applied?\n\n-- The motivation behind improvement (1) is for variance reduction. However, if the aim in variance reduction, aren't there other (simpler?) ways to do this? For example, one can change the KL regularization to a different divergence that is analytically expressible. Or, if one is already using a Gaussian distribution for both behavior and learned policy, the KL is already analytically expressible. These and other alternatives should be considered in addition to the proposed method.\n\n-- I am not sure how the motivation for improvement (2) connects to the proposed solution (state-conditioned Lagrange multipliers). In fact, Sec 3.2 suggests that the KL limit (\\epsilon_KL) is constant for all states.  Using the reasoning of the paper, shouldn't this limit be larger for states which appear more in the dataset?\n\n-- Improvement (4), specifically initializing the learned policy as the behavior policy, is used by previous work. For example, https://arxiv.org/abs/2006.03647 I also know that it is used in the implementation of CQL (although I am not sure if the CQL paper mentions it).\n\n-- Improvement (5) seems drastic. Why should we remove all learning signal in areas where the KL is too high? Moreover, the underlying problem here appears to be an issue with the form of KL regularization. Perhaps a solution is to use a different regularizer? One which does not blow up like this?\n\n-- The experimental results are favorable for BRAC+, but they are not especially compelling. As the paper mentions, BRAC+ mostly improves on multi-modal datasets, where as performance of BRAC+ is worse than existing methods on other datasets.\n\n-- Moreover, experimentally, if the claim is that BRAC+ is better on multi-modal datasets, it would be nice to show performance on other (more multi-modal) datasets in D4RL. For example, pointmaze, antmaze (diverse), kitchen -- these all exhibit more multi-model datasets than the Gym Mujoco envs.", "extra_scores": {}, "extra_text": {}}, {"review_id": "GZOnKl_5MYg", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes two major improvements over BRAC to address the key challenge of offline RL: out-of-distribution action sampling. The first improvement is to replace the sample-based divergence calculation with an analytical upper-bound estimation. The second improvement is to use a state-dependent weight for divergence regularization, which is learned automatically via dual gradient descent. In addition, several small and practical enhancements are proposed. The method is evaluated on simulation benchmarks and demonstrates performance gains over several state-of-the-art baselines, especially on multi-modal datasets.\n\nThe paper addresses an important research direction: offline RL, which is crucial in the field where data collection is difficult and data is scarce. The paper is clearly written. Although the method is improvements of an existing approach (BRAC), the improvements are non trivial, novel, and show promising results.\n\nI have three minor questions and suggestions:\n1) Intuitively, I think that KL divergence between policies might not be the best measure to prevent OOD action samples.  The regularization of BRAC(++) is to minimize the KL divergence between the current policy and the behavior policy. The KL divergence only cares about the \"closeness\" between the policies, and totally ignores the actual density of (s, a) in the dataset. For example, although \\pi_\\beta(s, a) can have a high probability, which allows the \\pi_\\theta to sample action a frequently on state s, depending on how the behavior policy is sampled to create the dataset, the state s might not be visited often, and thus has a low density in the dataset.\n\n2) BRAC++ has less improvement on single-modal datasets. However, it is these single-modal datasets that improvements are more desired. In multi-modal datasets, a good policy already exists and is used to collect the dataset. So the use of offline RL in multi-modal datasets is not well motivated despite its better performance. To improve the paper quality, I suggest that more analysis and discussions about why BRAC++ did not work well on single-modal datasets are included.\n\n3) In the ablation study (Figure 1), I suggest that the paper adds more curves from different tasks so that it is clear that the observations and conclusions in ablation study are general across different tasks. If page limit is a concern, it is OK to add additional figures in the supplementary material.\n\nDespite my minor concerns, I still think that this is a strong paper that advances the field of offline RL. Thus, I recommend accepting this paper.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Eu3CFfKyxZe", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces several enhancements of BRAC (Behavior Regularized Actor Critic), an approach for offline reinforcement learning. The original idea of BRAC is to add a divergence term between behavior distribution, which was used to collect data, and training policy. In the original paper, this is implemented by fitting behavior policy with a density model and then using this density model to estimate a divergence between these distribution. This paper proposes to improve this approach in several ways: first, authors propose to increase the expressiveness of the behavior policy density model by using a mixture of Gaussian distributions; second, authors propose to adaptively tune the weight of the KL term by learning a state-dependent Lagrange multiplier; finally, authors add an additional entropy term, supervised pre-training and out-of-distribution action masking for target Q.\n\nIn overall, this paper introduces a simple and well motivated set of enhancements.  The paper is well written and easy to follow. The approach is evaluated on a subset of tasks from a widely used benchmark for offline RL, D4RL. The approach is competitive with the state-of-the-art methods and  outperforms the original BRAC and CQL, a recent state-of-the-art approach, on several tasks.\n\nIt is very impressive that such a simple set of enhancements provides significant performance gains. I like the paper but I have several concerns regarding the significance of the contribution and some experimental details:\n1) although the approach demonstrates impressive results, the novelty is incremental;\n2) the approach improves BRAC by moving from a single Gaussian distribution to a mixture of distributions. How sensitive is the approach is the choice of a number of mixture components?\n3) table 3 raises some concerns regarding generality of the approach since some of the crucial hyper-parameters are task-specific;\n4) each ablation in figure 1 is performed on a single task that raises concerns regarding generality of the conclusions. \n\nAt the moment, this is a borderline paper. However, if the aforementioned concerns are properly addressed, I will reconsider my rating. I looking forward to authors' response.", "extra_scores": {}, "extra_text": {}}, {"review_id": "3st-qc2uqnk", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary**: This paper proposes a few improvements to BRAC, a framework for offline (or batch) RL approaches. Namely, the authors propose using an analytical bound on the action KL to reduce variance, a state-dependent Lagrange multiplier (output by a neural network), an extra entropy term, and a mask on the Q-value updates for KL-constraint violating actions. The authors demonstrate the effectiveness of these techniques on a subset of the D4RL datasets, where they compare favorably on some tasks. Ablations demonstrate the benefit of each technique on a subset of tasks.\n\n**Strong Points**: The writing is clear, presenting each of the improvements well. For the most part, these improvements are also technically correct, or at least fairly well motivated.\n\nExperiments demonstrate that these techniques generally improve the performance of BRAC on the D4RL datasets. D4RL is a recent standardized dataset for offline RL, and results are reported using 4 random seeds, which is generally considered reasonable. Thus, this experimental support appears to be rigorous. The reported results are also competitive with other model-free and model-based offline RL approaches. In addition, the authors also perform ablation experiments to investigate each of the proposed improvements, generally demonstrating improved stability and final performance.\n\nThe approach is likely to be reproducible, considering that BRAC is a fairly straightforward framework and the proposed improvements are not incredibly complex. The authors provide network architectures and training details in the appendix. However, in practice, many RL algorithms are prone to small, important implementation details. Releasing code would help to further improve reproducibility.\n\nAlthough none of the improvements, on their own, are particularly novel, their combination appears to provide important practical advances for BRAC. These seemingly small improvements can make a significant difference in practice, e.g. SAC is substantially worse without the ensemble of Q-networks (from TD3). \n\n**Weak Points**: As mentioned above, this paper is not substantially novel; it proposes a few practical improvements for an existing framework/algorithm in an existing area (offline RL). While practical improvements are still worthwhile, this does place a greater requirement for either improved empirical performance or new insights into issues with existing algorithms (e.g. the TD3 paper).\n\nEmpirical performance improvements are demonstrated, however, most improvements appear to be on the hopper environment. It should also be noted that 3/4 of the ablation experiments are on the hopper environment as well. In other environments, the results (Table 1) are mixed. While the results are encouraging, I do not seem them as being in favor of BRAC+ across the board. Instead, I could see future researchers incorporating the new techniques into other offline RL approaches, e.g. BEAR+, MOPO+, etc. As such, the issues with previous works and the improvements from these techniques should be clearly described and demonstrated.\n\nWhile the authors do present ablation experiments for their techniques, the reader is left to infer that the performance improvement is due to the reasons given by the authors. This paper would be more convincing and impactful if these issues were shown directly, giving readers insight into the issues with existing methods. For instance, instead of reporting performance from different KL evaluation techniques, the authors could report the empirical variance of each technique, as well as the bias incurred by their bound.\n\nAnother weak point of this paper pertains to the experiment details / comparison. The authors appear to have implemented the basic BRAC framework along with their proposed improvements. The results for other methods in Table 1 are from Fu et al., 2020 and other recent papers. However, it’s unclear if these implementations are comparable. For instance, BEAR typically uses a larger ensemble of Q-networks, whereas the authors report using 2 networks. While these results can be useful for benchmarking across the field, they do not necessarily give insights to readers. I would have preferred to see the authors implement their improvements across multiple previously proposed offline RL algorithms (both model-free and model-based) to enable a more direct comparison and evaluation.\n\n**Accept / Reject**: I would lean slightly toward acceptance for this paper. The authors present several techniques for improving offline RL algorithms within the BRAC framework. These practical improvements are often overlooked, but they can have an outsized impact on the community. This is inline with papers like Flow++, TD3, etc. However, as mentioned above, there are aspects of this paper that could be substantially improved. Currently, the paper does not demonstrate clear insight into the issues that are being fixed; this is only shown through improved performance. Likewise, the experiments could be improved to provide a more direct comparison and evaluation of these techniques (see above).\n\n**Questions**:\nThe TD loss for Q-network training in Algorithm 1 appears to be missing the action KL / entropy at the next step. Was this a purposeful decision?\n\nHow much does pre-training help or hurt? This is presented, but performance is not compared in the experiments section.\n\n**Additional Feedback**:\n\nIntroduction:  \nI wouldn’t necessarily say that BRAC was “proposed” in Wu et al., 2019. While they did coin the term BRAC, this is built around previous works, e.g. BCQ, BEAR, etc. This is part of a larger class of relative-entropy policy search algorithms.  \n\nBackground:  \nMDP acronym is given twice.  \n\nImproving Behavior Regularized Offline Reinforcement Learning:  \nThere’s a connection between the optimal Lagrange multiplier (or “temperature”) and uncertainty in the Q-value estimate (see Fox, 2019: toward provable unbiased temporal-difference value estimation). This should probably be discussed.  \n“requires large samples to reduce” —> “requires a large number of samples to reduce”  \n“variational inference model” —> “latent variable model”  \n“can be estimated analytically”: but this really only applies in special cases, e.g. Gaussian.  \nShould explain why the behavior policy is not just one Gaussian.  \nI found the entropy regularization paragraph difficult to follow.  \n\nRelated Work:  \nThere’s another related paper by Agarwal et al., 2019 (arXiv:1907.04543).\n\nExperiments:  \n“regularizor” —> “regularizer”  \nUnclear what is interesting/new in the performance vs. KL threshold section.  ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents an interesting perspective on improving offline RL within BRAC framework. 
Given the improvements over BRAC, the paper is well organized and easy to understand. 

The overall results pique interest in comparison with more recent Offline/Batch RL papers: BRAC, BEAR, CQL. 
The results in this paper bring BRAC-family of methods closer to CQL with a number of practical improvements, and could have impact in practice. 

However, the reviewers have slight split over the marginal value of additional machinery. There do remain some concerns:
- KL divergence is not the best metric to capture OOD issues between policies. 
- The additional machinery in comparison to CQL may be unnecessary, at least in terms of results. 
- The method requires many task-specific key hyper-parameters, which limits the generality of the approach.

I would recommend rejection as it stands. The paper needs more careful empirical analysis that explains what methodical improvements are actually required and which ones only provide marginal bumps.  With multiple task-specific hyper-params, it may be tricky for these ideas to realize their potential if not clearly understood. 
Further release of sufficiently documented and easy to use implementation, will probably be required for acceptance since the main argument in the paper are number of technical improvements in BRAC.
Decision: Reject