Supported Policy Optimization for Offline Reinforcement Learning
Authors: Jialong Wu
Haixu Wu
Zihan Qiu
Jianmin Wang
Mingsheng Long
Venue: NeurIPS 2022 Accept (2022)
Keywords: Offline reinforcement learning
Deep reinforcement learning

Abstract: Policy constraint methods to offline reinforcement learning (RL) typically utilize parameterization or regularization that constrains the policy to perform actions within the support set of the behavior policy. The elaborative designs of parameterization methods usually intrude into the policy networks, which may bring extra inference cost and cannot take full advantage of well-established online methods. Regularization methods reduce the divergence between the learned policy and the behavior policy, which may mismatch the inherent density-based definition of support set thereby failing to avoid the out-of-distribution actions effectively. This paper presents Supported Policy OpTimization (SPOT), which is directly derived from the theoretical formalization of the density-based support constraint. SPOT adopts a VAE-based density estimator to explicitly model the support set of behavior policy and presents a simple but effective density-based regularization term, which can be plugged non-intrusively into off-the-shelf off-policy RL algorithms. SPOT achieves the state-of-the-art performance on standard benchmarks for offline RL. Benefiting from the pluggable design, offline pretrained models from SPOT can also be applied to perform online fine-tuning seamlessly.

Supported Policy Optimization for Offline Reinforcement Learning
Jialong Wu', Haixu Wu', Zihan Qiu², Jianmin Wang', Mingsheng Long School of Software, BNRist, Tsinghua University, China
Institute for Interdisciplinary Information Sciences, Tsinghua University, China (xujialong0229, qzhl 1628))1m11mmmll.1 com, whx20@mails. tsinghua. edu. en ijimwang, (tsigahengg) edu. cn
Abstract
Policy constraint methods to offline reinforcement learning (RL) typically utilize parameterization or regularization that constrains the policy perform actions within the support set of the behavior policy. The elaborative designs of parameter- ization methods usually intrude into the policy networks, which may bring extra inference cost and cannot take full advantage of well-established online methods. Regularization methods reduce the divergence between the learned policy and the behavior policy, which may mismatch the inherent density-based definition of support set thereby failing to avoid the out-of-distribution actions effectively. This paper presents Supported Policy Op OpTimization (SPOT), which is directly derived from the theoretical formalization of the density- based support constraint. SPOT adopts VAE-based density estimator explicitly model the support set of behav- ior policy and presents a simple but effective density-based regularization term, which can be plugged non-ine non-intrusively into off-the-shed shelf off- fol-policy RL algorithms. SPOT achieves the state-of-thh-al performance on standard benchmarks for offline RL. Benefiting from the pluggable design, offline pretrained models from SPOT can also be applied to perform online fine-tuning seamlessly.
Introduction
[9), where the agent learns from a fixed dataset, collected by arbitrary process, not only provides a bridge between RL and the data-driven paradigm but also eliminates the need to interact with the live environment, which is always expensive or risky in practical scenarios [1 Unfortunately, the absence of environment interaction also raises number of challenges. Previous work has shown that the extrapolation error of the Q-function queried by -uittf--isttibution actions significantly degrades the performance of off off-policy algorithms [9].
Offline RL 228
Avoiding -istriiiiitiinooonn actions, namely constrain the learned policy to perform actions within the support set of the behavior policy, is essential to mitigate extrapolation error. To meet this support constraint, policy constraint methods (29 to offline RL utilize either parameterization 9 50 or
regularization [5] 24 techniques. However, there are still several drawbacks in previous methods of policy constraint, limiting their performance and applications. Firstly, parameterization methods involve elaborate designs of the policy parameterization, typically coupled to generative models of the behavior policy, to directly constrain actions taken by the learned policy. But these designs intrude into the architecture of policy networks, which may bring extra inference cost and supplementary difficulties to implement and tune offline RL algorithms. Furthermore, as criticized by Fujimoto and Gu (7). these intrusive designs complicate causal attribution of performance gains and transfer of techniques between offline RL algorithms or from well-established online RL algorithms. In contrast, for the second category, regularization methods designed with non-intrusive pluggable
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

manner, which is done by simply augmenting penalty to the actor loss that measures the divergence of the learned policy from the behavior policy [T3] regularization methods may mismatch the inherent density- based definition of support set, which applies support constraint indirectly. As we show in experiments (Section 5.1). the divergence-based design prevents regularization methods from effectively avoiding out-of out-of-distribution actions and thus limits their performance.
22] However, these divergence-
In this work, we aim to design a pluggable offline RL method that also directly meets the stan- dard formalization of the support constraint based on the density of the behavior policy. We in- troduce Supported Policy OpT'imization (SPOT), regularization method which can be plugged non-intrusively into off-the-shelf off- offlpollcy RL algorithms. SPOT involves regularization term from the new perspective of explicit estimation of the behavior density. Concretely, SPOT adopts a VAF-based density estimator [9 37 to explicitly model the support set of behavior policy and presents simple yet effective regularization term directly applied to the estimated density.
Our method benefits from a closer connection between theory and algorithm, thereby achieving superior performance compared to previous methods. It also profits from the pluggable algorithmic design, which leads efficient inference and minor algorithmic modifications on top of standard off-policy algorithms. Moreover, a minimal gap between offline learning objectives and standard online learning objectives also enables SPOT to take full advantage of existing online RL algorithms and attain strong online fine-tuming performance after offline RL, exceeding state-of-the methods.
The main contributions of this work are three-fold:
We derive a regularization term for constrained policy optimization in offline RL, based on the theoretical formalization of the density-based support constraint, which directly regularizes the behavior density of actions taken by the learned policy.
We propose Supported Policy OpTimization (SPOT), practical algorithm with neural VAE-based density estimator to implement the proposed regularization term.
Compared to several strong baselines, SPOT achieves state-of tfaeeof-the-aa th- results standard offline RL benchmarks [6] and also outperforms previous methods when online fine-tuned after offline RL initialization.
2
Related Work
Our work belongs the family of policy constraint methods in offline RL, where pa- rameterization or regularization is typically utilized. Comparison between previous pol- icy constraint methods and ours has been summarized in Table In addition to the policy constraint methods, we also review behavior policy modeling in offline RL, which is commonly necessary to both pa- rameterization and regularization methods. Lastly, we briefly discuss a broader range of competitive offline RL approaches.
Table 1: Comparison among policy constraint methods.
Method
Support Constraint
Implementation
BCQ 9
Explicit Density Constrained
Intrusive
PLAS
550
Parameterization
EMaQ 11
BEAR
2
Implicit Density Constrained
Pluggable
BRAC-p 47
Regularization
TD3+BC
Explicit
Pluggable
SPOT (Ours)
Density Constrained
Regularization
Policy constraint via parameterization. Careful parameterization of the learned policy can naturally satisfy the support constraint. For example, BCQ (9) learns a generative model of the behavior policy and trains the actor to perturb randomly generated actions. The policy parameterized by BCQ is to greedily select the one maximizing Q function among large number of perturbed sampled actions. EMaQ IT] simplifies BCQ by discarding the perturbation models. PLAS [50) learns a policy in the latent space of the generative model and parameterizes the policy using the decoder of the generative model on top of the latent policy.
Policy constraint via regularization. An alternative approach is to use divergence penalty in order to compel the learned policy to stay close to the behavior policy, such as KL-divergence [5
maximum mean discrepancy (MMD) [24], Fisher divergence (22) and Wasserstein distance 47
Using divergence penalties alleviates the hard constraint of explicit parameterization but may bear
the risk of distri--ittiitttinn actions. Although BEAR (24] attempts to constrain the policy to the

---

support of the behavior policy, it heavily relies on the ampirically-foure approximate property of low-sampled MMD. TD3+BC (7] simply adds behavior cloning (BC) term the policy update and presents competitive performance on simple locomotion tasks. Note that recent SBAC [48] proposes a new policy learning objective based on performance difference lemma [7. along with a density-based regularization term similar to ours, but our work simply plugs the term into the standard policy training objective to enjoy minimal algorithmic modifications.
Behavior policy modeling in offfine RL. Most policy constraint methods need to fit an accurate generative model of the behavior policy, to sample n-diistribution actions or estimate behavior density, Conditional variational auto-encoders (CVAE) [19] 40) are typically used by past works 19, 50) to sample actions, while EMaQ [T]] opts for using an autoregressive model [10] which enables more accurate sampling. On the other hand, policy class of Gaussian 124 47 or Gaussian mixture models [2 are commonly used fit and estimate the density of behavior policy. Instead of explicitly fitting the behavior policy, recent approaches have utilized implicit constraint without ever querying the values of any -ut-ofssample actions marginal state distribution, to the best of our knowledge, we are unique in estimating the density of behavior policy based on VAE, for its flexibility to capture almost arbitrary class of distributions 20)
44 23 Although MBS- QL 30 uses BLBO to estimate
Broader range of offline RL approaches. Besides policy constraint methods based on parameteriza- tion or regularization, there exist more types of competitive offline RL approaches. IQL 23 designs a multi-step dynamic programming procedure based on expectile regression, which completely avoids any queries to values of out-of-sample actions. Pessimistic value methods, such as CQL [26 produce a lower bound on the value of the current policy to effectively alleviate overestimation, but their performance may suffer from excessive pessimism. Advantage- -eighhed regression 35 33 45)
improves upon behavior policy, while simultaneously enforcing an implicit KL-divergence constraint. Recent sequence modeling methods based on Transformers [43 also show competitive performance in both model-free (5] or model-based [14] paradigm.
3 Background
The reinforcement learning problem 41 is formulated as decision making in the environment represented by a Markov Decision Process (MDP) = (S, po, P.,, 7), where is the state space, A is the action space, (so) is the initial state distribution, p(&'|s, a) is the transition distribution, r(s, a) is the reward function, and y is the discount factor. The goal in RL is to find a policy т [[a maximizing the expected return: E, 1E ("r(st, d.)].
The optimal state-action value function or Q function Q*(s,a) a) measures the expected return starting in state taking action a and then acting optimally thereafter. A corresponding opti- mal policy can be obtained through greedy action choices π (s) arg max, Q" (s, as. The Q- learning algorithm [46] learns Q* via iterating the Bellman optimality operator T, defined as: TQ (8, a) = Eg (r(s,a) a) + 7 max.' Q (s', a')]. For large or continuous state space, the value can be represented by function approximators Qo a) with parameters 0. In practice, the parameters 0 are updated by minimizing the mean squared Bellman error with an experience replay dataset D and a
target function Q7 32 (f) = E(s.a.na'>>>) (Qo(s,a) a) - r - y max.' QB (s', a')
In continuous action space, the analytic maximum is intractable. Actor- methods [1] 8 12
perform action selection with a separate policy function T6 maximizing the value function:
(o) E(s.a...")>> [Qo(s, a) - - 20-200 (s', no (s')))
(1)
The policy can be updated following the deterministic policy gradient (DPG) theorem [39
Jn (Ø) EAND [-Qo (s, (+)))
3.1
Offlline Reinforcement Learning
In contrast to online RL methods, which interact with enviromment to collect experience data, offline
RL 28 29 methods learn from a finite and fixed dataset D = {(x,, a, r, 8')} which has been collected
by some unknown behavior policy TT. Direct application of off-policy methods on offline setting
suffers from extrapolation error 9 24 which means that an out-of-distribbutio action a in state s
can produce erroncously overestimated values Qs(s,a).
3

---

3.2 Support Constraint in Offline RL
To avoid commonly used, which means in state s to only allow action that has e-support under behavior policy: {a : TT{(a|s) > e). Kumar al. (24) first introduce the distributtor-coosttainnee operator, which instantiated to the supported operator in this work:
actions from function approximation, support constraint 9 24
Definition 3.1. Given behavior policy ug and threshold e, the supported backup operator is defined
as
T&Q (s, a) = Ex r(s,a)+ +Y
Max d'vrg(a'l=")
Q(5',a))
(3)
with its fixed point Q7 (s, a) named as the supported optimal Q function.
Following the theoretical analysis of Kumar et al. [24] we can obtain a bound of how suboptimal Q may be with respect to the optimal Q function Q":
Corollary 3.2. Let ar(e) = ITQ" -T.Q" 11x = max (TQ"(.. T.Q" (s, a)}. The suboptimal- ity of Q7 (8, a) can be upper- hoper--ouuded as ||Q" - Q:1l S a().
Note that the complete theoretical results from Kumar et al. (24] also includes a term w.r.t. the bootstrapping error. We refer readers to Kumar et al. [24] for more details. Similar results also presented by Ghasemipour et al.
4 Supported Policy Optimization
As aforementioned, performing support constraint is the typical method to mitigate extrapolation error in offline RI. Noticing that the support constraint can be formalized based on the density of behavior policy, we propose the Supported Policy OpTimization (SPOT) as a regularization method from the new perspective of explicit density estimation. Concretely, SPOT involves new regularization term, which is directly derived from the theoretical formalization of the support constraint. Besides, conditional VAE is adopted to explicitly estimate the behavior density in the regularization term. Plugged into off-po RL algorithms, we will finally arrive at the practical algorithm of SPOT.
4.1 Support Constraint via Behavior Density
Similar to how the optimal policy can be extracted from the optimal Q function, the supported optimal policy can also be recovered via greedy selection: ~ (s) arg (al=)))(()))> Q2 (s, a). For the
case of function approximation, it corresponds to a constrained policy optimization problem.
While prior works use specific parameterization of (9 or divergence penalty 24 47 to perform
support constraint, we propose to directly use behavior density (-]())) as constrain
max EA-D [Qa(s,
min log (3( >E,
where log for notational simplicity. Constraint via behavior density is simple and straightforward in the context of support constraint. We adopt log- lieeliidl instead of raw likelihood because of its mathematical convenience.
This problem imposes a constraint that the density of behavior policy is dower-bounder at every point in the state space, which is impractical solve due to the large even infinite number of constraints. Following previous works from both online RL (38) and offline RL [24 35) w.r.t. constrained policy
optimization, we instead use heuristic approximation that considers the average behavior density:
max E.ED [Q0(s, (s())
(5)
s.t. E,ED [log (T)( > ē.
With slight abuse of notation, here (=(ss stands for action distribution of the stochastic policy, while (s)s) stands for deterministic action taken by the policy.

---

Converting the constrained optimization problem into an unconstrained one by treating the constraint term penalty, we finally get the policy learning objective
(o) = E.-D Q-Qe (T)s)) - Alog To (TT(s)
(6)
where À is Lagrangian multiplier.
4.2 Explicit Estimation Behavior Density
The straightforward regularization term in Eq. (6) requires access to zg- While we only have offline data generated by тa. we can explicitly estimate the probability density at an arbitrary point with the density estimation methods []].
The variational autoencoder (VAE) 19 is among the best performing neural density-estimattion
models and we opt to use a conditional variational autoencoder [40] as our density estimator. Typically, Pe s)p(2(s)d
(a]s)) can be approximated by Deep Latent Variable Model pv (p(als)
with a fixed prior p(z[s) = (), While the marginal likelihood (als) (s) is intractable, VAE additionally uses approximate posterior gp (z/a,, 8) (= 8) and parameters y and yp can be optimized jointly with evidence lower bound (ELBO):
log Po (als) z E (eloss) log (v,a,
( a,
(=la,s) [log Pv (o
KL la (zla (=)la,))))p(( (>][((
def LELBO 40,
After training VAE, we can simply use - BELBOO to lower-bound log Pr (als) and thus approximately lower-bound log næ in Eq. (6). However, there theoretically exists bias between them, as we know log (als) LELBO (ala,, Ip (2)a, s)). To obtain estimation with lower bias,
can use the importance sampling technique [37 20
Pe 2|8)
lo log (sla,))
4p (a,
L
(a 2 20)
(8)
23 Eg (x00a
log
a (z s)
def log (a) 3/, L).
Burda et al. 3 show that log a s, v), L) gives a lower bound log (als))) and the bound becomes tighter L increases. Note that here we adopt sampling of VAE to directly estimate the density of the behavior policy instead of to estimate the divergence 224
In summary, the loss function in Eq. (6) can be implemented with the explicit density estimator as follows:
(x) EgND [-Qo (p())) Alog (5)
4.3 Practical Algorithm
The general framework derived above can be built on top of off-policy algorithms with minimal modifications. We choose TD3 [ as our base algorithm, which recently shows strong resistance to overestimation in offline RL 7 (see Section 5.2 for detailed discussion).
normalization. Following TD3+BC
7 add normalization term policy loss as
a default option for better balance between Q value objective and regularization: (= =
E.-D (o(e. Alog (
I.)] where
E., 1Q (st, (x()))) is the nor-
malization term based on the minibatch [ with size N.
Simpler density estimator. While log (aa(as; ip, L) with large is much tighter, we empirically find there is no further improvement with larger compared T. (see Figure Appendix
for results of ablation study). To make algorithm simpler, we choose to only L 1 for
5

---

Algorithm/ Supported Policy Optimization (SPOT)
Input:
Dataset D = {(s,{ r,)}
VAE Training
Initialize VAE with parameters và and yp
fort Tj do
Sample minibatch of transitions (s, ~ D Update 4 minimizing (ELRO(S, a; i, v)) in Eq.
end for
Policy Training
Initialize the policy network Th. critic network Qa and target network Qo with < 0
for t 1 to T2 do
Sample minibatch of transitions (s, a, T')) D
Update minimizing (!) in Eq.
Update á minimizing (o) in Eq. 9
Update target network: f r0 (1-T)0
end for
practical estimator, which is just the ELBO estimator of the VAE: log (als; so v/, I. 1) CELBO (s, a; yp, 1)). Note that with L 1, we can analytically separate out the KL divergence as Eq. 7 to enjoy simpler and lower-variance update.
Overall algorithm. Putting everything together, the full algorithm is summarized in Algorithm Our algorithm first trains VAE using (ELBO a; y,, vb) to obtain density estimator with sufficient accuracy. Then turns policy training analogous to common Actor-Criti methods except that we plug the regularization term computed by the density estimator into the policy loss Ja (Ø)).
Experimental Evaluation
Our experiments aim to evaluate our method comparatively, in contrast to prior offline RL methods, focusing on both offline training and online fine-tuning. We first demonstrate the effect of À on applying support constraint and show that our method is able to learn a policy with the strongest performance the same level of constraint strength, compared to previous policy constraint methods. We then evaluate SPOT on D4RL benchmark [6]. studying how effective our method is in contrast broader range of state-of-the-an offline RI. methods. Finally, we study how SPOT compares to prior methods when fine-tuning with online RL from an offline RL initialization, and investigate the computational efficiency of different methods. Code is available attttss: htaps:://githuli com/ com/thuml/ thuml/
SPOT
5.1 Analysis of Support Constraint in SPOT
Effect of on constraint strength. The coefficient SPOT is essential and corresponds specific constraint strength in the constrained policy optimization problem formalized in Eq. 4 To illustrate how effects the learned policy, we evaluate behavior density of actions taken by the policy learned with varying values of X E [0.05, 0.1, 0.5) on standard D4RL (6 Gym-MuJoCo domains. Concretely, we plot the distribution of behavior density log T (s)(s))) ~ D in Figure La where log TT estimated by our learned density estimator (Eq. (8) with L set sufficient large number 500 for more accurate estimation. As we show, with smaller X, the learned policy is much more possible to perform actions with low behavior density log ग ( On the other hand, policies learned by higher À are restricted to take only high-density actions.
Tradeoff between constraint strength and optimality. It has been shown by Kumar et al. 124 that the optimality of approximate supported optimal policy is lower-bounded by tradeoff between keeping the learned policy supported by the behavior policy (controlling extrapolation error) and keeping the supported policy set large enough to capture well-performing policies. If the constraint in Eq. 44) is strong (by a large log e), the extrapolation error is restrained to be small but the optimal

---

Telfeherah meliun- +2
balfcheeiah ueliut 11
hepper- auppe--nolian.ogn replayy-
walket meduum splay
1.35
1.50
415-
Nocestood Tictur
Hocestand Roaue
45C-
0.75
0.50
115-
08.
355-
3.0
-6
buy so-
Oreineal Sraee
Cestaits Strngls
Oupaten Strepht
30
+
BBAR
PLAS
SPor (Dum))
(a) Effect of X.
(b) Tradeoff between constraint strength and optimality.
Figure 1: Analysis on constraint strength: (a) With varying values the coefficient X, SPOT applies support constraint with different strength, which is demonstrated by the behavior density of actions taken by the learned policy: log (T))))))))
D. (b) When evaluating the performance at varying levels of constraint strength, SPOT takes shape of the "upper envelope" of all methods, showing that SPOT can always achieve the strongest performance among extensive policy constraint methods. Constraint strength is captured approximately by the 5th-percentile of the distribution log To (T)(s)))), ~ D. Extended results can be found Appendix C.4.
policy under constraint may have poor performance. Otherwise, if the constraint is weak, well- performing policies can be learned though at the risk of the extrapolation error.
We aim answer the question that at the same level of constraint being satisfied, is SPOT able to learn a policy with the strongest performance compared to previous policy constraint methods? We compare SPOT with BC (36,, BCQ 99. BEAR [24), PLAS (50) and TD3+BC 77. Hyperparameters to control constraint strength of various methods are adjusted to several values to form spectrum of constraint strength (see Appendix C.4 for details). We approximate the satisfied constraint strength log by the 5th-percentile of the distribution log ( (s) D. As shown in Figure lb our method SPOT takes shape of the "upper envelope" of all methods, demonstrating that taking advantage of exact standard formulation of support constraint, SPOT is flexible to learn the supported optimal policy and resist extrapolation error at the same time. Note that the divergence- based regularization method BEAR [44] yields poor performance in our experiments and only satisfies a loose constraint in contrast to other baselines, showing that it cannot prevent nut-of-distribution actions effectively and suffers from extrapolation error with indirect divergence regularization.
5.2 Comparisons on Offline RI. Benchmarks
Next, we evaluate our approach on the beRchmaccharrk 66 in comparison to state-of-the-art methods. We focus on Sym-MuJoCo locomotion domains and much more challenging AntMaze domains, which consists of sparse-reward tasks and requires "stitching" fragments of suboptimal trajectories traveling undirectedly in order to find a path from the start to the goal of the maze.
Baselines. We select the classic BC (36] and state-of of ff-the-art arr offline RL methods as baselines. For
methods based on dynamic programming, we compare to AWAC B33, Onestep RL (2), TD3+BC [7].
CQL 226,, and IQL [23]. We also include the sequence-modeling method Decision Transformer (5]
Hyperparameter tuning. The weight À in Eq. is essential for policy constraint. Following prior works 24 47], we allow access to the online environment to tune small set of the hyperparameter A (< 10 choices). which is reasonable setup for practical applications. See Appendix C.1 for additional discussion and details.
2 As we show, SPOT substantially outperforms state-of-the-as methods, especially in suboptimal "medium" and "medium-replay datasets with large margin, which further demonstrates the advantages of direct constraint on behavior density proposed by SPOT.
Gym-MuJoCo domains. Results for the Gym-MuJoCo domains shown in Table
AntMaze domains. Results for the AntMaze domains are shown in Table Note that D4RL recently releases a bug-fixed "-v2" version of AntMaze datasets, and thus we select competitive baselines and rerun their nuthor-provide implementations for comparison. We also include results for BCQ and
7

---

Table 2: Performance of SPOT and prior methods Gym-MuloCo tasks. m medium, medium-replay. m-e medium-expert. For baselines, we report numbers directly from the IQL paper (23,, which provides unified comparison for "-v2" datasets. For SPOT, we report the mean and standard deviation for 10 seeds.
BC
AWAC
DT
Onestep
TD3+BC
CQL
IQL
SPOT (Ours)
HalfChectah-m--ovV
55.2
42.8
86.8
93.4
90.7
91.6
86.7
86,9+4.3
Hopper-m-e-v2
52.5
55.8
107.6
103.3
98.0
105.4
91.5
99.317.1
Walker-m-e-v2
107.5
74.5
108.1
113.0
110.1
108.8
109.6
112.0.0..5
HaliCheetah-m-
42.6
43.5
42.6
48.4
48.3
44.0
47.4
58,4+100
Hopper-m-v2
52.9
57.0
67.6
59.6
59.3
58.5
66.2
86.0=8.7
Walker-m-v2
75.3
72.4
74.0
81.8
83.7
72.5
78.3
86.4
HalfChectah-m-rrv2
36.6
40.5
36.6
38.1
44.6
45.5
44.2
52.2 +.2
Hopper-m-r-e m-pper-m-r-v2
18.1
37.2
82.7
97.5
60,9
95.0
94.7
100.2
Walker-m-r-v2
26.0
27.0
66.6
49.5
81.8
77.2
73.8
91.6+22.
Gym-MuJoCo total
466.7
450.7
672.6
684.6
677.4
698.5
692.4
773.0+33.22
Table Performance of SPOT and prior methods on AntMaze tasks. For baselines, we obtain the results using author-provin provided implementations on "-v2" datasets. For BCQ and BEAR, we report numbers from D4RL paper [6]. For SPOT, we report the mean and standard deviation for 10 seeds.
BCQ
BEAR
BC
DT
TD3+BC
PLAS
CQL
IQL
SPOT (Ours)
umazc-v2
78.9
73.0
49.2
54.214.1
73.0=34.0
62.0a167
82.6157
89.6142
93.5=24
mivze-diverse-v2
55.0
61.0
41.8
41.21114
47.0=73
45.4279
10.2167
65.6283
40.7a5.1
medium play-v2
0.0
0.0
0.4
0.0g00
0.0
31.4121.
59.0:16
76,4
74.7146
medium--iveere-e v2
0.0
8.0
0.2
0.0100
0.2:004
20.6127.7
46.62210
72.827.0
79.1
large-play-v2
6.7
0.0
0.0
0.0+00
0.0=0.
2.2+38
16.4#17.
42.0+3.8
35.3+83
large-diverse-22
2.2
0.0
0.0
0.0+00
0.0+0.0
3.0+67
3.2+4.1
46.0+4.5
36.3+137
AntMaze total
142.8
142.0
91.6 95.4115.
120.2141.7
164.6_81.3
218.0159. 392.4
359.6139.7
It is not suitable to compare them with reproduced baselines but we include them in order to highlight that previous policy constraint methods struggle succeed in training challenging AntMaze domains.
BEAR trained on "-v0" datasets, directly from 6..
As shown Table 3 SPOT performs slightly worse than IQL but outperforms remaining modern offline RL baselines, including the pessimistic value method CQL and the sequence modeling method Decision Transformer. Note that IQL ingeniously designs multi-step dynamic programming and policy extraction steps to apply implicit constraint for offline RL, but when online fine-tuned after offline RL initialization, IQL is inferior to SPOT. SPOT's pluggable design can take full advantage of existing online RL algorithms (see Section 5.3 and Table 4). To the best of knowledge, our algorithm is the first to train successfully in challenging AntMaze domains with pluggable modification top of off- off-policy RL methods for offline RL.
A Allaioon. Figure 2 we perform an ablation study over the components in our method. First, we replace the VAF-basedd density estimator with behavioral-cloned Gaussian policy model 24 47]. As expected, it degrades the performance due to the lack flcxibility to model complex distribution, especially Gym-MuJoCo medium- datasets and AntMaze datasets. Then, we evaluate SPOT without Q normalization and find that the removal provides some benefits as well as damages on different datasets, with only an insignificant impact total performance. Nevertheless, we include it as a default option following TD3+BC 7
Lastly, we investigate how the base off-policy algorithm matters. Since SAC [12] has similar off- policy performance with TD3, we attempt to adopt SAC as our base method. As Xu el al. (48] argue that the entropy term will do harm to the offline setting, we also implement variant of SAC without maximum entropy regularization. Surprisingly, both variants are vulnerable to pathological extrapolation error AntMaze domains and provide poor performance (see Figure 2). We argue that

---

Guussian Demiity Estimation
wio Normalizatio
Stochattic Policy Max
Souchastic Policy Max (SAC)
100
Peceent Diftrencs
90
-100
Figure 2: Percent difference of the performance of an ablation of SPOT, compared with the original medium-expee experrt medium, mr
algorithm. Hc HalfCheetah, H Hopper, W Walker, medium-rrplaa Am AntMaze, umazze, m = medium, = large. diverse, = play. Implementation details and quantitative results can be found in Appendix C.2.
several key native designs of TD3 are beneficial to offline RL. (1). For the case of learning stochastic policy without entropy regularization, we find that the learned policy quickly degenerates into a deterministic one (with near-zero standard deviation). A concern with deterministic policies is that they are prone to overfit overestimated actions and propagate the estimation error through Bellman backups, which is even serious in offline RL. Hopefully, TD3 introduces Target Policy Smoothing into Eq. (T). which adds random noise to target actions and can alleviate the effect of error propagation. (2). For the case with entropy regularization, Bellman backup of a stochastic policy resembles Target Policy Smoothing, then the primary distinction may come from the actor learning objective. While TD3 produces deterministic action minimizing Eq. (9). stochastic policies more likely to produce out-of-distribution actions with erroncous estimated values, so the policy gradient may become biased well as with high variance. On much casier MymoMuJoCo domains, SAC-style SPOT variants are comparable to the TD3 variant on most of the datasets, but we remark that TD3 indeed extends the limit of policy constraint methods on some tasks, such hopper-medium. Our analysis suggests that TD3 may be preferable for offline RL as the base off off-policy method with native designs (such as "stochastic" critic training and deterministic actor training) addressing function approximation error not only in the online setting but also in the offline setting
8 42
5.3 Online Fine-tuning after Offline RL
Pluggable SPOT can be online fine-tuned seamlessly, which means that we only need to gradually decay the constraint strength À in the online phase in order to avoid excessive conservatism. As AWAC (33 shows that behavior models are hard to update online, we fix the VAE during online fine-tuning. Note that when À is zero, our algorithm is exactly the standard off-policy RL algorithm that SPOT builds upon.
beneficial since we enjoy a minimal gap between offline RL and well-established online RL methods and can take full advantage of them for online fine-tuning.
Since IQL [23 is the strongest base- line in our offline experiments, which also has shown superior online per- formance than prior methods [33] 26) in its paper, and most of the other baselines fail to learn meaningful re- sults, we follow the experiments of IQL. and compare to IQL. in online fine-tuning. We also compare to our base RL method TD3 [8] trained on- line from scratch. We use the chal- lenging AntMaze domains (6). Dur- ing online fine-tuning of SPOT, the regularization weight À is linearly de- cayed to one-fifth of its initial value. See Appendix C.3 for details.
Table 4: Online fine-tuning results on AntMaze tasks, show- ing initial performance after offline RL and performance after IM steps of online RL. All numbers are reported by the mean of 5 seeds.
IQL
SPOT (Ours)
umaze-v2
85.4 > 96.2
93.2 99.2 (+3.0)
umaze-diverse-v2
70.8 - 62.2
41.6 + 96.0 (433.8)
medium-play
68.6 - 89.8
75.2 + 97..4
2iviiu--iverse--2
73.4 + 90.2
73.0 + 96.2 (+6.0)
large-play-v2 large-diverse-v2
40.0 78.6
40.8 +8944 (+10.8)
40.4 > 73.4
44.0 90.8 (+17.4)
AntMaze total
378.6 - 490.4
367.8 569.0 (+78.6)
Results are shown in Table 4 While online training from scratch fails the challenging sparse reward tasks on AntMaze domains, SPOT initialized with offline RL succeeds to learn nearly optimal

---

policies and performs significantly better than the strongest baseline IQL, especially in the most difficult large maze.
5.4 Computation Cost
Regularization methods. including our SPOT, benefiting from the pluggable design, only need one forward pass of the policy network to do in- ference, while parameterization methods always need inference through secondary components, such as the generative model the critic net- work, which may bring extra time or memory cost. As demonstrated empirically in Figure 3] parameterization methods are usually slower than regularization methods. TD3+BC and our SPOT run more than two times faster compared to the most time-consuming BCQ. while SPOT also has the advantage of parameterization meth- ods, which explicitly constrain the behavior den- sity of learned actions.
418 s2S
Panmmtertiotion Repularizaion
751 ma
666
Informne Time
640
530
400
365 me
366 ms
200
BCQ
EMaQ
PLAS
BEAR
TD3-BC SPOT (Oun)
Figure 3: Runtime of various offline RL algorithms interacting with the HalfCheetah environment to produce a 1000-steps trajectory. See Appendix C.5 for the details.
SPOT indeed adds some training overhead due to VAE-based density estimator, but is still much simpler than most methods. A comparison of training time is provided in Appendix C.5. SPOT lies the second tier, only slightly worse than PLAS and beaten by the simplest TD3+BC.
6
Conclusion
We present Supported Policy OpTimization (SPOT), policy constraint method offline RL built upon off-the-shelf off- pol-cy RL algorithms. SPOT introduces a pluggable regularization term applied directly the estimated behavior density, which brings a number of important benefits. First, our algorithm is computationally efficient at inference, which only needs one forward process of the policy network for action selection. Second, capturing the standard formulation of the support constraint based on behavior density, it obtains excellent performance across different tasks in the D4RL benchmarks, including standard Gym-MuloCo tasks and much more challenging AntMaze tasks. Finally, the pluggable design of our algorithm makes it seamless to apply online fine-tuning after offline RL pre-training. Taking full advantage of well-established online methods, SPOT exceeds the state-of-the-art online fine-tuning performance the challenging AntMaze domains.
Limitations. One limitation our current method, shared by most policy constraint methods, is that the performance may be limited by the accuracy of estimation of the behavior policy. Advances in generative models, such as diffusion models [13 4 may improve the real-world performance of offline RI,, especially in scenarios with highly multimodal behaviors. An exciting direction for future work would be develop an effective pluggable constraint mechanism excluding explicit estimation of behavior policy. Another limitation of our work is that we rely online evaluation to select the best set hyperparameters. Although this evaluation protocol commonly adopted by the literature of offline RL, extensive online evaluation is not practical in real-world applications and online evaluation budgets may have a significant impact on final performance [21 (27) Specifically for SPOT, the selection of regularization weight without requirements for extensive online evaluation critical and needs to be developed, either by offline policy evaluation (34, by manual tuning based on offline training metrics and conditions (25), or by auto-tuning with dual optimization (12).
Acknowledgments
We would like to thank many colleagues, in particular, Jincheng Zhong, Haoyu Ma, Yiwen Qiu, and Yuchen Zhang for their valuable discussion and support for this work. This work was supported by the National Key Research and Development Plan (2000AA National Natural Science Foundation of China (62022050 and 62021002), Beijing Nova Program (7.201 100006820041), and BNRist Innovation Fund (BNR2021RC01102),
10

---

References
[1] Christopher M. Bishop. Pattern Recognition and Machine Learning. Springer, 2006.
[2] David Brandfonbrener, William F Whitney, Rajesh Ranganath, and Joan Bruna. Offlie rl without off-policy evaluation. In NewrlPS, 2021.
[3] Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. arXiv preprint
2015.
[4] Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu. Offline reinforcement learning via high-fidelity generative behavior modeling. arXiv preprint arXiv:2209.4454. 2022.
[5] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In NeurlPS, 2021.
[6] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.072/99 2020.
[7] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. In NeurIPS. 2021.
[8] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In /CML, 2018.
[9] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In JCML, 2019.
[10] Mathieu Germain, Karol Gregor, lain Murray, and Hugo Larochelle. MADE: Masked autoen- coder for distribution estimation. In ICML., 2015.
[11] Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, and Shixiang Shane Gu. EMaQ: Expected-max q-learning operator for simple yet effective offline and online rl. In ICML, 2021.
[12] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint 1rXiv:18/2.05900 2018.
[13] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020.
[14] Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. In NeurIPS, 2021.
[15] Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen. Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of implicit human preferences dialog. arXiv preprint arXiv:/907.000 00456, 2019.
[16] Alistair EW Johnson, Tom Pollard, Shen, Lehman Li-Wei, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. MIMIC-III, a freely accessible critical care database. Scientiffic data, 3(1):1-9, 2016.
[17] Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning.
In ICML, 2002.
[18] Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. QT-Opt: Scalable deep reinforcement learning for vision-based robotic manipulation. In CoRL, 2018.
[19] Diederik Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.
[20] Diederik P Kingma and Max Welling. introduction to variational autoencoders. arXiv preprint 1956..9900029900 2019.

---

[21] Ksenia Konyushova, Yutian Chen, Thomas Paine, Caglar Gulcehre, Cosmin Paduraru, Daniel J Mankowitz, Misha Denil, and Nando de Freitas. Active offline policy selection. NeurIPS, 2021.
[22] Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. ICML, 2021.
[23] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learnig. In ICLR, 2022.
[24] Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In NeurlPS, 2019.
[25] Aviral Kumar, Anikait Singh, Stephen Tian, Chelsea Finn, and Sergey Levine, A workflow for offline model-free robotic reinforcement learning. In CoRL, 2021.
[26] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In NeurlPS, 2020.
[27] Vladislav Kurenkov and Sergey Kolesnikov. Showing your offline reinforcement learning work:
Online evaluation budget matters. In ICML, 2022.
[28] Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforcement learning, pages 45-73. Springer, 2012.
[29] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 2020.
[30] Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Provably good batch reinforcement learning without great exploration. In NeurIPS, 2020.
[31] Will Maddern, Geoffrey Pascoe, Chris Linegar, and Paul Newman. 1 year, 1000 km: The oxford robotcar dataset. The International Journal of Robotics Research, 36(1):3-15, 2017.
[32] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland. Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 5 518(7540):529-533 2015.
[33] Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. AWAC: Accelerating online reinforcement learning with offline datasets. arXiv preprint 20Xi::2006.09359 2020.
[34] Tom Le Paine, Cosmin Paduraru, Andrea Michi, Caglar Gulcehre, Konrad Zolna, Alexander Novikov, Ziyu Wang, and Nando de Freitas. Hyperparameter selection for offline reinforcement learning. arXiv preprint arXiv:2007.09055 2020.
[35] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. advantage-weighhee regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv: 191000791 2019.
[36] Dean A Pomerleau. Alvinn: An autonomous land vehicle in a neural network. Technical report, Carnegie Melon Univ. Pittsburgh, PA. Artificial Intelligence and Psychology.. 1989.
[37] Danilo Jimenez Rezende. Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In ICML, 2014.
[38] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In ICML,, 2015.
[39] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In ICML, 2014.
[40] Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. In NeuriPS, 2015.
[41] Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
12

---

[42] Harm van Seijen, Hado van Hasselt, Shimon Whiteson, and Marco Wiering. A theoretical and empirical analysis of expected sarsa. Proceedings of the IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning: ADPRI,, 2009.
[43] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
[44] Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint arXivv2208.06199, 2022.
[45] Ziyu Wang, Alexander Novikov, Konrad Zolna, Jost Tobias Springenberg, Scott Reed, Bobak Shahriari, Noah Siegel, Josh Merel, Caglar Gulcehre, Nicolas Heess, et al. Critic regularized regression. In NeurlPS, 2020.
[46] Christopher JCH Watkins and Peter Dayan, Q-learning. Machine learning, 8(3-4):279-292, 1992.
[47] Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint 1rXiv:1911.1131,, 2019.
[48] Haoran Xu, Xianyuan Zhan, Jianxiong Li, and Honglei Yin. Offline reinforcement learning with soft behavior regularization. arXiv preprint arXiv:21 2021.
[49] Yigin Yang, Xiaoteng Ma, Chenghao Li, Zewu Zheng, Qiyuan Zhang, Gao Huang, Jun Yang, and Qianchuan Zhao. Believe what you see: Implicit constraint approach for offline multi-agent reinforcement learning. In NeurIPS, 2021.
[50] Wenxuan Zhou, Sujay Bajracharya, and David Held. PLAS: Latent action space for offline reinforcement learning. In CoRL, 2020.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 6
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Appendix D.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix A.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material or as URL)? [Yes] Code to reproduce the main results of our methods (Table is available at https //github. com/ thum1/SPOT
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] See Appendix C.
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes] See Section 3 and Appendix C.
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, cloud provider)? [Yes] See Section and Appendix C.5.
4. If you using existing assets (e.g., code, data, models) or curating/releasing
assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any assets either in the supplemental material URL? [Yes]
(d) Did you discuss whether and how consent obtained from people whose data you'l 're using/curating? [Yes] All the datasets we used are open- speresourced and we comply with the license.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [Yes] All the data and experiments are simulated without any personally identifiable information.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "YSlemDntD7o", "reviewer": "Reviewer_AVgj", "summary": "The paper considers the offline Reinforcement Learning setting, and introduces a loss that allows any off-policy RL algorithm to learn from offline data. The loss addresses the issue of out-of-distribution actions being sampled by the actor being trained. The loss is obtained by first training a (conditional) Variational Autoencoder on state-action tuples. Then, its ELBO loss (used for training the VAE itself) is used as part of the actor training, to keep in close to the support of actions in the dataset. The critic training loss does not seem to have to be modified (Equation 1 is untouched), and leverages the actor to sample actions (so, indirectly, it samples actions in the support of the dataset).\n\nAn empirical evaluation in challenging environments shows that the proposed method, SPOT (the loss described above + TD3) outperforms all the baselines being considered. The results are sometimes moderate, sometimes much more significant.", "questions": "I don't have any question for the authors", "limitations": "There does not seem to be any negative societal impact of this paper that should be discussed. The proposed algorithm outperforms the baselines in every case, and is overall strong. As such, the lack of a \"Limitations\" section, for instance, is not problematic for this paper. A mention of possible areas of future improvement would have been nice in the conclusion, that for the moments sounds a bit like an advertisement.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The proposed method is easy to implement, and indeed does not need to change fundamental aspects of an off-policy RL algorithm to make it offline.\n\nStrengths (quality, clarity, somewhat the significance):\n\n* The paper is well-written and easy to read. It flows naturally\n* The proposed method is sound, and its derivation from constraint-based optimization is intuitive and motivates the resulting loss well.\n* The empirical evaluation is thorough, with the impact of hyper-parameters being well-studied. The environments being used are challenging (not just toy environments), and many baselines are considered.\n* Source code is provided! This is a rare event, and greatly helps reproducibility and answering questions such as \"does the gradient flow to the actor through the VAE?\" (the answer is yes). The code is also clean, self-contained and easy to understand.\n\nThere is no big weakness in this work, even though its novelty is maybe a bit lower than the other aspects listed in Strengths. While the proposed method is novel, and quite distinct from the existing literature, it is still using a VAE in combination with some off-policy RL algorithm. BCQ uses the VAE to sample actions (while this work uses the VAE as part of the actor loss). BRAC estimates the behavior policy and uses the KL divergence between the actor and the behavior policy to regularize learning. BEAR is inspired from constraint-based optimization like this work, but uses the maximum mean discrepancy to keep the actor close to the actions in the dataset. So, overall, the originality of this work, albeit satisfactory, cannot be stated as a strength.\n\nMinor comments:\n\n* Section 2 in the appendix is quite interesting, and if room can be made in the paper, would deserve to be in the main text\n* The description of how $\\lambda$ decreases over time in the online fine-tuning setting is interesting and should also be in the main text (it is in the appendix currently)\n\n**Author response**\n\nThe authors mention that they will improve the conclusion of the paper and extend the main text, which were my two minor comments. The other reviewers also seem to be leaning towards acceptance, so I maintain my score of accept.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Kic2JGjLD98", "reviewer": "Reviewer_Eyz5", "summary": "\nThe author presents Supported Policy OpTimization (SPOT), a policy constraint method to offline RL built upon an off-the-shelf off-policy RL algorithm. SPOT introduces a pluggable regularization term applied directly to the estimated behavior density. The experiments are conducted in the D4RL benchmark and several SOTA baselines are taken for comparison.\n", "questions": "1. In figure 1(b), why should we compare the performance of SPOT relative to the baselines under the same constraint strength? And why SPOT can achieve better performance with the same constraint strength since the related evidence cannot be found in the current theorem. btw, I think a meaningful comparison is that SPOT can easily adjust the degree of constraint by controlling \\lambda, while other solutions do not do this well. Also, why are the other baselines only report a few points (e.g.,  In hopper-medium-replay-v2, PLAS only has one point)?\n2. In figure 2, why does the effect become worse when using Gaussian, especially in Walker2d-medium and Hopper-medium. The policies of these datasets are Gaussian distribution, so Gaussian should be the most accurate modeling.\n3. In the last paragraph of Section 5.2, you mentioned that the TD3 method is not as stable as SAC, especially in the Ant environment, and Figure 2 illustrates the problem. So, is it possible that the performance improvement of SPOT reported in Table 3 comes from our algorithm choice (TD3 instead of SAC) rather than our constraint mechanism? For a fair comparison, an ablation experiment may be required here.\n4. In Section 5.3, the author compares the performance improvement of SPOT and IQL under the online fine-tuning setting, and the author claims that the SPOT algorithm can be online fine-tuned seamlessly. This point of view is a bit confusing to me: if I have an online reinforcement learning algorithm, then, for any offline reinforcement learning algorithm, I only need to pass the parameters of the optimal policy solved offline to the online reinforcement learning algorithm as the initialization parameters of the policy, and then I can achieve “fine-tuned seamlessness”. Is this something only SPOT can do? Further, I think the comparison with IQL is unfair. The author directly uses the constraint mechanism of IQL for online training, while the SPOT algorithm reduces the constraint coefficient. Since conservative updates are unnecessary during online training, this obviously suppresses the ability of IQL-based online-trained policies.\n5. In Section 5.4, the authors compare the computation cost of inference of several algorithms and illustrate the inference speed advantage of SPOT. In my opinion, the advantage of SPOT's inference speed should exist in many algorithms, such as some algorithms based on Pluggable Regularization (such as CQL, BRAC, etc.) and most model-based offline reinforcement learning algorithms (such as MOPO). Is my statement correct? If so, I think the description in Section 5.4 is a bit overclaimed: we shouldn't consider it a SPOT-exclusive feature.\n\n", "limitations": "I will consider increasing the score if the authors clarify the above questions or give a better experiment design to demonstrate the proposed method.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nStrengths:\n\n1. the article is overall well-written and easy to follow.\n2. The motivation of the article, simple and pluggable offline RL, is meaningful in real applications.\n3. The solution is simple to implement and reasonable.\n\n\nWeaknesses\n\nThe weaknesses come from the experiment designs, which are unclear to me. I have listed the related questions for the authors to qualify it.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "9RFg_XsEz2F", "reviewer": "Reviewer_FcL9", "summary": "This paper proposes supported policy optimization (SPOT), which introduces a \"pluggable\" regularization term applied to an estimated behavior policy. In the implementation, following VAE, the authors use the ELBO to estimate the behavior policy, and following TD3+BC, add a normalization term to the policy loss. The authors emphasize that SPOT is computationally efficient at inference, obtains excellent performance, and enables effective online fine-tuning. ", "questions": "1. The authors state that \"our method benefits from a closer connection between theory and algorithm\". What does the author mean by \"theory\"? \n2. I think there is no essential difference between Equation 4 and Equation 3. Furthermore, the authors approximate Equation 4 and get Equation 5. Is the theoretical guarantee still satisfied? ", "limitations": "yes", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ This paper is well-written.\n+ The proposed method is simple and intuitive. \n+ The authors conduct extensive experiments to show the effectiveness of SPOT. \n\nWeaknesses:\n+ The paper lacks theoretical analysis.\n+ The method is somewhat incremental. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "3Wy8C7COq9", "reviewer": "Reviewer_qPyz", "summary": "The paper present a regularization method for offline reinforcement learning called SPOT. The main idea is a policy constraint inspired by support constraint on behavior policy. The method is evaluated extensively on standard offline RL benchmarks and achieves state of the art result.", "questions": "A minor suggestion is on the symbols in $\\log\\pi_\\beta(\\pi_\\phi(s)|s)$ (4), as $\\pi_\\beta$ is the probability $\\in [0,1]$ while $\\pi_\\phi$ actually is the action $\\in \\mathcal{A}$. It may cause some confusion at first glance.", "limitations": "The authors have addressed the societal impact of this work.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The idea of support constraint looks simple and effective as in Sec.4.1, and the empirical result as shown in Figure 1 and Table 2 demonstrate its effectiveness for practical problems. From the empirical result, this work achieves new state-of-the-art results on standard offline RL benchmarks and is beneficial to the related communities.\n\nWeakness: From my perspective, the effectiveness of the proposed algorithm is mainly demonstrated by empirical experiments. It would be more complete if the empirical improvement over previous methods can be justified with some theoretical analysis.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept