Constrained Policy Optimization with Explicit Behavior Density For Offline Reinforcement Learning
Authors: Jing Zhang
Chi Zhang
Wenjia Wang
Bingyi Jing
Venue: NeurIPS 2023 poster (2023)
Keywords: Offline Reinforcement Learning
GAN
Flow Model
Policy Control
TL;DR: This paper provides a policy control method for solving offline reinforcement learning problem

Abstract: Due to the inability to interact with the environment, offline reinforcement learning (RL) methods face the challenge of estimating the Out-of-Distribution (OOD) points. Existing methods for addressing this issue either control policy to exclude the OOD action or make the $Q$ function pessimistic. However, these methods can be overly conservative or fail to identify OOD areas accurately. To overcome this problem, we propose a Constrained Policy optimization with Explicit Behavior density (CPED) method that utilizes a flow-GAN model to explicitly estimate the density of behavior policy. By estimating the explicit density, CPED can accurately identify the safe region and enable exploration within the region, resulting in less conservative learning policies.  We further provide theoretical results for both the flow-GAN estimator and performance guarantee for CPED by showing that CPED can find the optimal $Q$-function value. Empirically, CPED outperforms existing alternatives on various standard offline reinforcement learning tasks, yielding higher expected returns.

Constrained Policy Optimization with Explicit Behavior Density for Offline Reinforcement Learning
Jing Zhang HKUST
Chi Zhang Kuaishou Technologies
Jzhanggy@connect. ust.hk
zhangchi10@kkaishhuu.. com
Wenjia Wang" HKUST (GZ) and HKUST
wenjiavangOust.hr hk
Bing-Yi Jing" SUSTech
ingby&sustech. edu. cn
Abstract
Due to the inability to interact with the environment, offline reinforcement learning (RL) methods face the challenge of estimating the Out-of- oiitt--Diitiitttionn (OOD) points. Existing methods for addressing this issue either control policy to exclude the OOD action make the Q function pessimistic. However, these methods can be overly conservative or fail to identify OOD areas accurately. To overcome this problem, we propose Constrained Policy optimization with Explicit Behavior density (CPED) method that utilizes flow-GAN model to explicitly estimate the density of behavior policy. By estimating the explicit density, CPED can accurately identify the safe region and enable optimization within the region, resulting in less conservative learning policies. We further provide theoretical results for both the flow-GAN estimator and performance guarantee for CPED by showing that CPED can find the optimal Q-function value. Empirically, CPED outperforms existing alternatives various standard offline reinforcement learning tasks, yielding higher expected returns.
Introduction
As form of active learning, reinforcement learning (RL) has achieved great empirical success in both simulation tasks and industrial applications [1] 2
6 7]. The great success of RL is largely due to sufficient information exploration and the online training paradigm that the agent has plenty of interactions with the environment. However, the merits of RL methods limited when is difficult or costly to sequentially interact with the environment. many real-world scenarios such driverless intelligent diagnosties [8]. online interaction is unacceptable. Thus, vanilla RL methods may possibly fail under such circumstances.
One recently developed approach to mitigating the constrained online-interaction is offline RL. 8 I0). Motivated by various data-driven techniques, the offline RL leverages prior experiences to train policy without interacting with the environment. In offline RL, critical challenge is distribution shift (also called "extrapolation error" literature). Specifically, vanilla RL methods
"Corresponding authors: Bing- Yi Jing and Wenjia Wang.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

using the Bellman equation and the Q-function approximation probably fail to deliver good estimates for Out-of -iitrf--iitributiinn (OOD) points. To this end, two types of offline RL solutions have been proposed. One type is the Q-function constraint. By adding constraints on the Q-function, this type of solution provides a pessimistic estimation of the Q-function and prevents the Q-value from being too large. The constraints include ensembling multiple Q Q-functions 12) penalizing Q-function with high uncertainty [13 [1], among others. Recently, implicit Q-learning [5] and mildly conservative Q-learning [16] are suggested. However, the uncertainty of the Q-function or tight lower bound of Q-value is difficult to estimate. Thus the performance is not guaranteed neither theoretically nor practically.
Another type of solution is policy control, where the learning policy is controlled over the support of the behavior policy. Thus the OOD points will not be visited and the distribution shift problem is is noted that previous policy control methods limit the safe areas within the close neighborhoods of the training dataset by distribution distance metrics such as KI. divergence, Such policy control approaches often lead to an over-conservative and suboptimal learning policy, and the optimal policy value may not be reached. For example, some state-action points are within the training distribution but unobserved in the training dataset, they may be considered OOD points in previous policy control approaches. Nevertheless,thhet
alleviated 17 18 19
maximum mean discrepancy (MMD), and Wasserstein Distance 20
7 18.
points are safe when the estimated Q- Q-function can well generalize on this area [12 [1]. and the optimal policy likely lies in the unseen area of the data space. In this sense, a global estimator of the conditional action data space is required to define the safe areas and examine whether unseen point is safe. To this end, Wu et al. 22 imposes the density- -based constraints and proposes the SPOT method. However, the density estimator in SPOT approximates the lower bound of the behavior policy density, and policy control methods with more accurate density estimators worth further discussing. As in the field of policy control methods for offline RL problem, the idea of ensuring the consistency of the support of learned policy and that of the behavior policy is considered the most desirable approach to tackle distribution shift. Therefore, a direct estimation density of the behavior policy is desired for guarantee support consistency.
Recently, generative models have shown strong capabilities in diverse fields, in which variational autoencoder ( (VAE) and generative adversarial network (GAN) (23 24, 25 (66 are popular tools utilized for sampling and other related applications. In RL areas, both VAE and GAN have been introduced in RL methods [19] 16 27 28]. However, as random sample generators, VAE and GAN cannot be directly applied to estimate behavior densities in offline RL., nor can they provide accurate density estimates. Wang et al. 229, Yang et al. [30), Chen et al. B1, Ghasemipour al. (32 have shown that VAE has shortcomings in estimating complex distributions, especially multimodal distributions or complex behavior policy distributions, and discuss that typically-used VAEs may not align well with the behavior dataset. As VAE is prone to cover mode, this phenomenon diminishes the VAE's capacity accurately estimate data distributions, particularly in scenarios involving multimodal data distributions, such as behavior policy distributions. Compared to VAE, GAN has more advantages in learning complex data distributions. Recently, various GAN variants have been suggested. Among them, the Flow-GAN [3 model combines the normalizing flow model [34] 35 and GAN, and provides an explicit density estimator. The Flow-GAN model is further renowned for its robustness and tuning flexibility and has garnered increasing attention.
In this work, we propose novel approach called Constrained Policy optimization with Explicit behavior Density (CPED) for offline RL. The CPED suggests a strong density-based constraint and employs an explicit and accurate density estimator identify the safe areas. Specifically, CPED utilizes the Flow-GAN model as the density estimator, whose accuracy is theoretically guaranteed. By leveraging this capability, CPED can accurately identify the feasible region, which includes both observed and unobserved but safe points (illustrated in Figure (c). By expanding the feasible region and enabling reasonable exploration within it, CPED exhibits reduced risk-aversion and demonstrates an enhanced ability to generate superior policies.
To summarize, the merits of the proposed CPED include the following.
* By introducing the Flow-GAN model, the CPED directly tackles the density estimation problem and proposes a novel policy control scheme for solving offline RL tasks.
"Code available https//giihubb.om//vvvaaajjgg

---

The CPED does not behave over-conservatively and allows exploration in the safe region, especially high-density areas, to find the optimal policy.
Practically, the effectiveness of CPED is fully demonstrated empirically. Extensive
periments on competitive tasks show that CPED substantially outperforms tfate-of-the-ar methods.
From the theoretical perspective, we show that GAN with hybrid loss can accurately estimate density, which itself contributes to the GAN community. Furthermore, we prove the policy learned from CPED can access the optimal value function, theoretically ensuring the efficiency of CPED.
a) Ground Truth
b) Neighbourhood-
c) Our CPED Method
Basad Mothods
Ofline Deta Pord
Oytiaal Priay (P(A
Sulo
Action
Action
Action
Fepieor on Arta hashbbuuttod
Sale Area criD
State
State
State
Figure 1: (a): The ground truth safe in offline RL. optimization, and the updates of policies and Q-functions are done within the green area. The blue points are collected behavior data D, and the red point denotes the optimal policy given the states. (b): In previous approaches, the exploration of the policy takes place in a small neighborhood of points in D (the orange circles). (c): The CPED relaxes the exploration area and constructs the feasible region (pink areas), which includes the unobserved but safe points (black point).
2 Related Works
Value Function Constraint. Taking OOD actions can lead to overestimated value function and introduce bootstrap error. Value function constraint methods aim to construct pessimistic value functions or reduce the uncertainty of the value function to avoid OOD actions. Following [2] 36),
pessimistic value function be obtained by ensembling multiple value functions. CQL [3
proposes a regularization term so that the estimated Q-function low-bounds its true value. TQC 37
follows the distributional RL method and learns the percentile of value function distribution. IQL
11 combines expectile regression with exponential weighted advantage function to avoid extreme value function. MCQ [16] tries use special bellman operator separate the estimation of the target value function for the in-distribution point and OOD point. The value function shows great uncertainty when taking OOD points. UWAC [4] estimates the uncertainty of the value function using the dropout technique. Urpí et al. (38) applies the tail risk measurement for the Q-function, in which the value function with high uncertainty penalized, and the learned value function forced to be pessimistic. However, finding the lower bound of the value function or estimating the uncertainty not easy. The error from estimating the value function and finding the lower bound hinders the performance of value function constraint methods. Recently, series of in-sample learning methods 4 have been proposed to optimize the Q-value function or the policy value function V solely utilizing in-sample data. As result, the extrapolation error, induced by -fitt--disstibution (OOD) data points the Q-value function, can mitigated.
33 40
Policy Control Method. To prevent the learning of policies that take out-of -iitr---iittiittioon (OOD)
actions, the policy control method incorporates penalty term during policy learning and restricts the distribution distance between the learned policy and the behavior policy. The BEAR method [17] uses the MMD distance to constrain the difference between the behavior policy and the learned policy,
measures the distribution shift between learning policy and behavior policy with KL divergence and Wasserstein distance. DOGE [46] learns a distance function to control the learning policy staying close to the behavior policy. Besides the explicit distance-
18
44
20
45 46
policy control methods, some implicit control methods 19 47 (22) which generate the learning policy action based on the estimated conditional probability distributions, are suggested. However, explicit distance control methods suffer from the generalization problem since the distance is estimated from the limited training samples. On the other hand, implicit control methods tend to generate over-conservative
3

---

policies, where the generated actions are too close to the behavior policy. Consequently, the learned policy by these methods discourages exploration and remains conservative.
Behavior Policy Density Estimation In offline RL setting, determining the action space af under given states is crucial. This is equivalent to learning the distribution of the behavior policy. How- ever, traditional density estimation methods [48] 49 50 [11] [52, 53) often fail in this scenario of density estimation due the curse of high dimensionality. Recently, deep neural network-based density estimation methods have proven successful in learning the distribution of the behavior policy, Conditional variational autoencoders (CVAE) have been utilized as both behavior policy samplers [19 17 47 16 54] and density estimators [22] Generative adversarial networks (GAN) are also introduced in offline RL scenario [55] [6). Singh et al. [57] applies the generative normalized flow model to learn an invertible map from the noisy prior to the behavior policy action space and do behavior policy sampling by the trained model. EMaQ
32 uses the Autoregressive Generative Model instead of CVAE as a sampler. Additionally, the diffusion model has been applied to estimate the density of the behavior policy in both model-free RI.
[1 and model-based RI. [58]. Specifically, SPOT [22] approaches the lower bound of the behavior density with ELBO derived from CVAE. However, to the best of our knowledge, most of the aforementioned methods either estimate the behavior density implicitly or lack the ability to provide accurate estimations. In our proposed method,
the CPED finds an exact density function using the Flow model [34 351 and GAN 23 24 25 59 60
3 Probability Controlled Offline RL Framework
3.1 Offline RL Problem Settings
In RL setting, the dynamic system is described in terms of a Markov decision process (MDP), which can be defined by set of tuples (S, A, Po. r. Here, S and A denote the state space and action space, respectively; T is conditional transition kernel, and T(s'|s a) is the conditional transition probability between states, describing the dynamics of the entire system; Po is the distribution of the initial states; r(s, a) is a deterministic reward function; and y E (0, 1) is a scalar discount factor.
The goal of RL is to find a distribution of actions (als) (named as policy) that yields the highest returns within a trajectory, i.e., maximizing the expected cumulative discounted returns. We denote this expected discounted return starting from (s, a) under policy π as (= a). Under the Q-learning framework, the optimal expected discounted return Q* (s, a) can be obtained by minimizing the L2 norm of Bellman residuals:
Eg M (s')s (1 (Q(s,)) (s,))
(1)
where Bi is the Bellman operator defined
(BQ) a) r(s,a) + (B [max Q(s', a')].
In the context of deep RL, various solutions are provided to estimate the Q-function 61
When the action space is continuous, the learning process is divided into actor (Q value function) training and critic (learning policy) training (64.. The critic network is learned from Eq without
"maximum" in the Bellman operator. The actor is formulated with various forms (65 66 67 I such as maximizing the Q-value (DDPG), the weighted log of action density (policy gradient), and clipped weighted action density ratio (PPO).
In the offline RL, the learning policy cannot interact with the dynamic system yet only learns from a training dataset ((s, s' r). Dataset D contains many triples (s, a, s',, r) that can be viewed as independent. Assume that the dataset D is generated by behavior policy Tg. We need to redefine the MDP corresponding to D under the offline RL settings.
Definition 3.1 (Offline MDP). Given a dataset 'D with continuous state and action space. Ler T denote the behavior policy that generated D. The MDP of dataset D defined by MD == is the measured state space related to D with probabil- ity measure PD, PT is the marginal probability of each state in state space of D, sd (A. is the measured action space related to D with conditional probability meusure Ta(als), Tr(('|s,a) is the transition probability between states, describing the dynamics of D, and pop is the distribution of the initial states of D.
{. A. .TD. Pu To,,, Y, where y = (S,PD) f

---

In offline RL, we can obtain the optimal Q-function by minimizing [q However, due to the distri- bution shift problem [7 8 the Bellman residuals are biased during the training phase. Specifically, at k-th iteration, since mj. is trained to maximize the Q-function, it is possible that we visits some OOD points with high estimated Q-function values. Nevertheless, we cannot evaluate the rewards of the OOD points under offline scenarios. Consequently the Bellman residuals at the OOD points may dominate the loss, and the learned Q-function value is mislead. Therefore, the optimization of offline RL must occur in the bounded safe state-action space y o such that the estimated Q-function is trustworthy.
3.2 Behavior Policy Estimation by GAN with Specific Density
In offline RL., the dataset 'D is generated by the behavior policy, whose density can be estimated via inverse RL. 688 One widely used approach in inverse RL,, maximum entropy inverse RL (MaxEnt
IRI. 69 ) models the density via a Boltzmann distribution
po(T)= = - exp((co(()))
(2)
where the energy is given by the cost function co(T) (Su. $1, a1, SH, ag) is a trajectory, and Z is a scale factor. The cost function cg(T) Ee co(ss, (x) is parameterized cost function that can be learned by maximizing the likelihood of the trajectories in D. And the scale factor Z is the sum of exponential cost trajectories that belong to the product space of the state and action of D. However, estimating Z is difficult because is scarcely possible to traverse the entire action space s/ An alternative way to estimate Z is via guided cost learning
E co(st, az),
of all the
70 where it has been shown that learning po (T) is equivalent 7k
to training the dataset DD with a GAN
70
Originally, the GAN is random sample generator for high- siggh-imensional data, and it consists of two models: generator G and discriminator D. The generator trained generate random samples that are close to those drawn from the underlying true distribution, and the discriminator tries to classify whether the input generated sample or an actual one from the training dataset. The GAN cannot provide an explicit density estimation, which is an indispensable ingredient when controlling the probability of actions and specifying an safe area. Thus, we cannot directly apply the MaxEnt IRL to our CPED. To overcome the difficulty, we follow the approach in [33] and consider GAN with flow model as a generator (we call Flow-GAN hereafter as 33
Flow-based generative models have shown their power in the fields of image processing and natural language processing [34] (35. The normalizing flow model allows us to start from a simple prior noise distribution and obtain an explicit estimate of the density function after a series of non-linear invertible transformations.
Specifically, let fo IRA -> RA be a nonlinear invertible transformation, where A is the dimension of action space st, and 0 is parameter. By change of variables, the probability density on D can be obtained by
(T( (f(T( det 0 Ōr
(3)
where 7 E 'D is a trajectory. Since fe is invertible, a generator can be constructed by GG føl In practice, both generator and discriminator in GAN are modeled as deep neural networks. Two specific neural network structures, NICE [34] and Real _VP [35]. are proposed to ensure that Gg (or fe) is invertible. The Jacobian matrices of the mapping functions f constructed by these two methods are both triangular matrices, guaranteeing efficient computation of |det 0((r))
Directly training a Flow-GAN by original loss function in GAN may lead to poor log- liglliiklssoods, and GAN is prone to mode collapse and producing less diverse distributions (33. Thus, following the approach in 33 we adopt hybrid loss function containing maximum likelihood loss and a GAN structure loss as the optimization objective for estimating the probability measure of behavior policy:
min max (Go, D&) XEE [log(p(()))
where Lo D&) can be any min-max loss function of GAN, D& is a discriminator indexed by parameter Φ and > 0 is a hyperparameter balancing the adversary learning process and the maximum likelihood process.
5

---

Remark 3.1. The reason for not simply using MLE is that MLE tends to cover all the modes of distribution (even though some modes have small probabilities). and MLE not robust against model misspecification [74] which often occurs in high dimensional scenarios.
In the following proposition, we show that the estimated density of the behavior policy using the hybrid loss is is equivalent that using MaxEnt IRL. The proof of Proposition 3.1 is provided in Appendix B
Proposition 3.1. For offline dataset DD generated by behavior policy иg. the learned likelihood function = using GAN with hybrid loss in Eq.Ais equivalent to that trained by MaxEnt IRL. If the generator of GAN can give a specific likelihood function (T),
(T), then
(T)(T) = =CZ -CZ-- exp(cco x Po (T)
(5)
where a constant related to D.
3.3 Constrained Policy Optimization with Explicit Behavior Density Algorithm (CPED)
In this work, we propose an algorithm for probabilistically controlling the learned policy, CPED. In CPED, we first utilize the Flow-GAN and obtain probability estimate of the trajectory po(r). The optimize objective follows the hybrid loss in E
min max CG D+) ( [Log(L5 (s, a)))]-
(6)
Since the episode 7 is a sequential collection of states and actions a, we rewrite the input of Lg (-)
with (s, a) instead of 7 in Eq and afterwards. From Proposition 3.1 the solution i equivalent
to the behavior estimator obtained from MaxEnt IRL, thus providing an effective estimation of the behavior policy.
Following the learning paradigm in classic RL methods, the policy learning process is divided into two parts: the actor training and critic training. With the estimated function Lg (-) approximating
the density of behavior policy, we can naturally propose a density- -aaeed constraint in policy learning, and obtain the safe according to the estimated density. The points with low density values excluded and the updated policy cannot are exceed the safe region. The actor learning is formulated as:
max E www.ppanngglllllsssssl100000000
where Te (1s) the learning policy, and S x A is an estimate of the bounded safe area s x of in Definition 3.1
There are many approaches for specifying the safe area S A. Kumar al. bounds the expected MMD distance between the learning policy and the behavior policy within predetermined threshold, while Wu et al. [22] constrains the lower bound of the estimated behavior density to identify the safe region. In this work, since we can have direct density estimation of L (s, a), we can utilize and specify the safe region as the region with not too small density. Specifically, we set
S A ((8, a) E - T. (s, a) < c). where e the threshol
The critic training is following the traditional Bellman equation as shown in Eq The entire algorithm of CPED is summarized in Algorithm In CPED, the Flow-GAN model plays crucial role in estimating the behavior density. The density function is obtained from the generator model and is iteratively updated along with the discriminator. We further use the TD3 framework solve the RL problem. It is noted that during Actor updates, the density of sampled actions should be higher than the threshold avoid out-of df-iistribttion areas.
4 Theoretical Analysis
In this section, we provide theoretical analysis of our proposed method. In Section 4.1
we prove the convergence of GAN with the hybrid loss, and based this result, we show the convergence of CPED in Section 42
"Eq 7 is a constrained optimization problem, and we are indeed using Lagrangian techniques to solve the constraint problem. During the practical optimization, the constraint on $ A ((s, a) E A penalty terms in the Lagrange function (with Lagrangian multiplier a)
log L Th (s,a) <) in 7 works

---

Algorithm The CPED algorithm
Input:
dataset D, target network update rate R, Lagrange multiplier a, training ratio of the generator and discrimination r, mini-batch size N, hyperparameters X, Initialize generator Go, discriminator Dæ. loss function in GAN Lg(-), behavior poliey likelihood
(-). Q networks {Qn, Qual actor Ty, target networks (Q Oigh target actor Tge, with x/ < 1/, v/6 7/. 1,
for i to do
Sample mini-batch of transitions (s, D
Initial Training Flow-GAN:
Φ & Lo(Go, D&)
< argming max Lg (Ga, Do) - ( log(L (s, (s)a))]
end for for
NN do Sample mini-batch of transitions (s, ~
Updating Flow-GAN:
@ a Lg Do)
< argming max (G D&) - (P(sa))DDoo((42 (s.
Updating Q-function:
Get action for the next state, {ā Tarr )T)((11)) + + dipp
Let y(s, a) min(Qng (s', a), (sg @))
T) <- argmin,, (Q (8, (r+ (5(s, 1= Updating Actor:
Update d according Eq 7 by using dual gradient descent with Lagrange multiplier
Update Target Networks:
e Kib + K)b'; v/ Kmy (1 x)T/, 1,2
end for
4.1 Convergence of GAN with the Hybrid Loss
In this subsection, show that training GAN with the hybrid loss function Eq can yield accurate density estimator. For ease of presentation, assume we observed X, ~ p" der j 1,.,, n, where μ² is the underlying true probability measure (the probability measure of behavior policy), and is the Lebesgue measure. Motivated by [24] 75 76), we consider the Integral Probability Metric (IPM) defined as
(11,21122) = sup Ex=m, Bymp))(()) fefa
where μi and μi are two probability measures, and Fa is discriminator class. Under IPM, the GAN framework with the hybrid loss solves the empirical problem (cf., Eq. (1.1) of (75 for the original GAN loss function)
fdw J fdii -> X Jo log log
Mn E argmiin max KEQ, fEFa
(9)
where Ann = # 8x, is the empirical measure, Pn a (regularized) density estimation, 2, is the generator class. and p == du estimator of the underlying true distribution. A detailed version of Theorem 4.1 and its proof can be found in Appendix A
de In the following (informal) theorem, we show that μ,, is accurate
Theorem 4.1 (Informal). Suppose the generator class Q, and the discriminator class Fa are induced by some Sobolev spaces. Choose as a constant. Under certain conditions,
dz,(4", (p)l))))))=)))))) O0pn KL(K")llll)) = Op((nn11))
where μ² is the true probubility measure, pm, is as in 2 with hybrid loss, and (L((""()m)) is the KI. divergence.
Theorem 4.1 states fast convergence rate Op(n-1/2 of the IPM and KL divergence can be achieved. Therefore, applying the hybrid loss can provide not only a good generator, but also provide an accurate density estimate, which can be utilized in our CPED framework to control the safe region.
7

---

Remark 4.1. Given the universal approximation theorem 77 78]. the neural networks can approxi- mate any smooth function. Thus, we consider Sobolev function classes instead of neural networks for the ease of mathematical treatment. If both the generator and discriminator classes are neural networks, one may adopt similar approach as in 75 [9] and then apply Theorem 4.1
Remark 4.2. In 775, the convergence rate of Ed (p", ,(n) is considered, while we provide sharper characterization of dr, ("",,)) pn). by showing the convergence in probability with certain convergence rate.
4.2 Convergence of CPED
With an accurate estimate of the underlying density, this subsection, we give a theoretical analysis of how CPED can find the optimal Q-function value over the product space of 9 s/. The proof of Theorem can be found in Appendix C
Theorem 4.2. Suppose (s a) the optimal Q-value on product space & x s. Let f(((s) := argmax, (s, a) be learning policy. Let a (-(s) be the probability-contrlle learning policy using CPED, which is defined on the support of æT. Suppose ft (11s) > 0 on st Then with probability tending to one,
(*(s,a) (,,,))=Q(s,5+(1) (1s)))
(10)
Furthermore, when minimizing the Bellman residual to solve the optimal Q-function and training the policy by iteration, have the following theorem, whose proof is provided in Appendix
Theorem 4.3. Suppose in step k + 1, the learned policy is defined as Th+1 argmnax, (T, a), Vs E L. Let V be the value function related to my in step defined as VT (s) (-15) (s,s a)}, and V* be the optimal value function over the product space I x st learning policy using CPED in step k which is defined on the support of TT. Then with probability tending to one,
Let ái be the
-V-lo S S k+1 I V*
(11)
Theorem provides assurance that CPED can achieve the optimal value function with linear convergence rate as the iteration number approaches infinity. Despite under offline settings, this rate is consistent with the standard rate in online RL [80]. When combined with Theorems 4.1 and 4.2
we can confidently conclude that CPED's effectiveness is theoretically guaranteed.
Experiments
In the experimental section, we analyze the performance of CPED on several standard offline RL tasks [81] We compare CPED with wide range of offline RL methods and analyze the performance of CPED. Further implementation details are provided Appendix E.2
5.1 Performance Comparison on Standard Benchmarking Datasets for Offline RL
The proposed CPED algorithm is evaluated on the D4RL 8 Gym-MuJoCo and AntMaze tasks.
Details of these tasks are shown in Appendix E.1 As a model-free method, CPED is compared with
the following model-free baselines: behavioral cloning (BC) method, BCQ
[19 I DT [2), AWAC
44. BEAR [17]. Onestep RI. [3 TDC+BC 45 CQL 13. IQL [15] SPOT [2] and PLAS [47].
In the experiments, the performance of all alternatives are measured by the average normalized scores
in episode. The experimental results reported in this paper either from the authors' original experiments or from our replications.
Tables and 2 show the experimental results of Gym-MuJoCo and AntMaze tasks, respectively. In GyymMMuJoCi tasks, CPED outperforms the competitors in most cases. In particular, our method has achieved remarkable success on the halfcheetah and walker2d tasks, especially under the medium- quality dataset. For the medium-expert task, CPED performs slightly inferior other tasks, which due to the fact that the expert task needs tight control so that the learning policy stays close to the behavior policy. In the AntMaze tasks, CPED performs slightly worse than IQI. but outperforms the other basclines, and it achieves better returns under "unmaze" and imedium-play tasks.

---

Table 1: The performance of CPED and other competing methods the various D4RL Gym-MuloCo tasks. We report the performance of Average normalized scores and standard deviations over random seeds. med medium, replay, expert, he halfcheetah, walker2d, ho=hopper
Dataset
BC
AWAC
DT
Onestep
TD3+BC
CQL
IQL
SPOT
CPED(OurS)
he-med
42.6
43.5
42.6
48.4
48.3
44.0
47.4
58.4
61.8
ho-med
52.9
57.0
67.6
59.6
59.3
58.5
66.2
86.0
100.1
wa-med
75.3
72.4
74.0
81.8
83.7
78.3
90.2
he-med-r
36.6
40.5
36.6
38.1
44.6
45.5
44.2
52.2
55.8+29
ho-med-r
18.1
37.2
82.7
97.5
60.9
95.0
94.7
100.2
98. 1821
wa-med-r
26.0
27.0
66.6
49.5
81.8
77.2
73.8
91.6
91.9a09
he-med-e
55.2
42.8
86.8
93.4
90.7
86.7
85
ho-med-e
52.5
55.8
107.6
103.3
98.0
105.4
91.5
99.3
95.3
wa-med-e
107.5
74.5
108.1
113.0
110.1
108.8
109.6
112.0
113.04214
Total
466.7
450.7
672.6
684.6
677.4
698.5
692.4
773.0
791.7
Table 2: The performance of CPED and other competing methods the various AntMaze tasks. We report the performance of Average normalized scores and standard deviations over 5 random seeds. med medium, diverse, p play.
Dataset
BCQ
BEAR
BC
DT
TD3+BC
PLAS
CQL
IQL
SPOT
CPED(Ours)
umaze
78.9
73.0
49.2
54.2
73.0
62.0
82.6
89.6
93.5
96.8#26
umaze-d
55.0
61.0
41.8
41.2
47.0
45.4
10.2
65.6
40.7
55,6:12
med-p
0.0
0.4
31.4
59.0
76.4
74.7
85.1
med-d
0.0
8.0
0.2
0.0
0.2
20.6
46.6
72.8
79.1
72.1
large-p
6.7
0.0
0.0
0.0
0.0
2.2
16,4
42.0
35.3
34.9453
large-d
2.2
0.0
0.0
0.0
0.0
3.0
3.2
46.0
36.3
32.3
Total
142.8
142.0
91.6
95.4
120.2
164.6
218.0
392.4
359.6
376.8+223.8
Veppoo
Thmm Arillaas
1000
6000
6000
30
0000
Vedor couitalel
AnM
4500
1.5
2.5
2.0
SCE
1803
2.00
1.63
Figure 2: (a) Average performance of BEAR and CPED on halfcheetah-mediung task averaged over seeds. BEAR can reach bottleneck very quickly. CPED remain increasing after reaching the bottleneck. (b) The time(cpoch) varying constrain parameter used in Gym-MuJoCo task. (c) The time(epoch) varying constrain parameter a used AntMaze task
To further discuss the learning dynamics of the proposed methods, the learning curve of BEAR, SPOT, and CPED return of BEAR increases very fast and remains unchanged after smooth decrease in subsequent training, indicating that policy exploration limited. This is consistent with the conservative nature of policy control methods that prevent the learning policy from straying too far from behavior policy. The pattern similar to SPOT's learning curve, which shows rapid increase and then enters platcau. In contrast, the average returns of CPED methods continue rise after sharp increase to
sificheetah-mediiie task is shown in Figure It noted that the average

---

certain extent, indicating that CPED keeps exploring the data and benefits from exploration. This also confirms Theorem (see Section from the experimental perspective.
5.2 Ablation Study
Time varying hyperparameter in policy control methods. When improving actor using Eq i Lagrange hyperparameter a is needed for this constrained optimization problem. The larger the a, the more restricted actions taken by the learning policy. Previous methods either use an auto-learning
17 or a constant a [18 22 BRAC [18] find that auto-learning a performs worse than the constant In our experiment, we apply time (epoch) varying a scheme with piece-wise constant functions and find this scheme performs better than the constant setting. The rationale behind this time-varying scheme is that employ strong constraints for the action taken at the early stage of policy training in case the model fails [ The constraint is further relaxed by decreasing a when the policy starts to converge. The piece-wise function of a for the Mujoco tasks and AntMaze tasks plot in Figure (c), and the performance results of the Mujoco tasks are shown in Figure EE.3 in in Appendix E.S We can see that the Mujoco task significantly benefits from this time varying scheme and delivers better returns in most scenarios.
Likelihood threshold 6. The hyperparameter determines the tightness of our estimated safety boundaries. A larger leads to more relaxed boundaries and versa vice. A commonly used setting for is the quantile of behavior policy density. However, the quantile value is difficult to set when the dimension is large. In our experiment, CPED uses the mean likelihood of the training batch as which decreases the effect of the estimation error of the behavior policy. See Appendix E.5 for additional discussion and details.
Conclusion and Future Work
In this article, we present CPED, a novel approach for policy control in offline RL. CPED leverages the Flow-GAN model to estimate the density of the behavior policy, enabling the identification of safe areas and avoiding the learning of policies that visit OOD points. We validate the effectiveness of CPED both theoretically and experimentally. Theoretical analysis demonstrates that CPED has high probability of achieving the optimal policy and generating large returns. Experimental results on the Gym-MuJoCo and AntMaze tasks demonstrate that CPED surpasses tttt--of-the-at competitors, highlighting its superior performance.
While CPED shows great capacity in experiment studies, there are several potential directions for its extension.. Firstly, as a generic method, GAN has diverse variants and is necessary try out more powerful GAN structures for density estimation. Secondly, when the feasible region is identified, how to explore the region efficiently remains another important issue, and we leave it to future work. Finally, it is interesting direction to examine the performance of CPED in more complex scenarios, including dataset generated by multiple behavior policies or multiple agents, etc.
Acknowledgements
We would like thank AC and reviewers for their valuable comments on the manuscript. Wenjia Wang was supported by the and Guangzhou Municipal Science and Technology Project (No. 2023A03J0003)
Joint Funding Program (No. 2023A0310019)
References
[1] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. volume 5, 2018.
[2] Bruno Scherrer, Mohammad Ghavamzadeh, Victor Gabillon, Boris Lesner, and Matthieu Geist. Approximate modified policy iteration and its application to the game of tetris. Journal of Machine Learning Research, 16, 2015. ISSN 15337928.
[3] Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):1334-13733 2016.
10

---

[4] Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on Robot Learning. pages 651-673. PMI.R, 2018.
[5] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[6] Yuxi Li. Reinforcement learning applications. arXiv preprint arXiv: 2019.
[7] M Mehdi Afsar, Trafford Crump, and Behrouz Far. Reinforcement learning based recommender systems: A survey. ACM Computing Surveys, 55(7):1-38, 2022.
[8] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint vrXiv:2005.01643 2020.
[9] Fisher Yu, Wenqi Xian, Yingying Chen, Fangchen Liu, Mike Liao, Vashisht Madhavan, and Trevor Darrell. Bdd100K- A diverse driving video database with scalable annotation tooling. Arxiv, 2018.
[10] Alistair E.W. Johnson, Tom J. Pollard, Lu Shen, Li Wei H. Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G. Mark. Mimic-iii, a freely accessible critical care database. Scientific Data, 3, 2016. ISSN 20524463, doi: 10. 10.108//data.2216.333
[11] Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. volume PartF168 11111118177,, 2020.
[12] Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty- offline reinforcement learning with diversified q-ensemble. Advances in neural information processing 3 34:7436-7447, 2021.
[13] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. volume 120-Decemberr 2020.
[14] Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua Susskind, Jian Zhang, Ruslan Salakhutdinov, and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforcement learning. arXiv preprint arXiv:2105.00mmm 105.08140, 40, 2021.
[15] lya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learnin. arXiv preprint arXiv:2110.06100 2021.
[16] Jiafei Lyu, Xiaoteng Ma. Xiu Li, and Zongqing Lu. Mildly conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2206.0474 047455 2022.
[17] Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off-polic q-learning via bootstrapping error reduction. volume 32, 2019.
[18] Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement 2019.
learning. arXiv preprint
[19] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. volume 2019-June, 2019.
[20] Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint 1rXiv:1907.00456 2019.
[21] Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de las Casas, Craig Donner, Leslie Fritz, Cristian Galperti, Andrea Huber, James Keeling, Maria Tsimpoukelli, Jackie Kay. Antoine Merle, Jean Marc Moret, Seb Noury, Federico Pesamosca, David Pfau, Olivier Sauter, Cristian Sommariva, Stefano Coda, Basil Duval, Ambrogio Fasoli, Pushmeet Kohli, Koray Kavukcuoglu, Demis Hassabis, and Martin Riedmiller. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 602, 2022. ISSN 14764687. doi: 00.1008//4118--01---444
11

---

[22] Jialong Wu, Haixu Wu, Zihan Qiu, Jianmin Wang, and Mingsheng Long. Supported policy optimization for offline reinforcement learning. arXiv preprint arXiv:2202.066239, 2022.
[23] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances Neural Information Processing Systems, 27, 2014.
[24] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International Conference Machine Learning, pages 214-223, PMLR, 2017.
[25] Chun-Liang Li, Wei-Cheng Chang, Yu Cheng, Yiming Yang, and Barnabás Póczos. MMD) GAN: Towards deeper understanding of moment matching network. In Advances in Neural Information Processing Systems, pages 2203-2213, 2017.
[26] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
[27] Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Improving sample efficiency in model-free reinforcement learning from images. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 10674-10681, 2021.
[28] Mazoure B Doan T. Gan q-learning. arXiv preprint arXiv: /805.08800048844 2018.
[29] Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.06192 2022.
[30] Shentao Yang, Zhendong Wang. Huangjic Zheng, Yihao Feng, and Mingyuan Zhou. A behavior regularized implicit policy for offline reinforcement learning. arXiv preprint 3rXiv:2202.06673 2022.
[31] Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu. Offline reinforcement learning via high-fidelity generative behavior modeling. arXiv preprint arXiv:2209. 14548, 2022.
[32] Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, and Shixiang Shane Gu. Emaq: Expected-max q-learning operator for simple yet effective offline and online rl. In International Conference on Machine Learning, pages 1891- PMLR, 2021.
[33] Aditya Grover, Manik Dhar, and Stefano Ermon. Flow-gan: Combining maximum likelihood and adversarial learning in generative models. 2018. doi: 10.1609/aai, v32// 17221.1111 18829.
[34] Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. 2015.
[35] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. 2017.
[36] Scott Fujimoto, Herke Van Hoof, and David Meger. Addressing function approximation error
in actor-critic methods. volume 4, 2018.
[37] Arsenii Kuznetsov, Pavel Shvechikov, Alexander Grishin, and Dmitry Vetrov. Controlling overestimation bias with truncated mixture of continuous distributional quantile critics. In International Conference on Machine Learning, pages 5556-5566. PMLR, 2020.
[38] Núria Armengol Urpí, Sebastian Curi, and Andreas Krause. Risk-averse offline reinforcement learning. arXiv preprint arXiv:2102.05371, 2021.
[39] Divyansh Garg, Joey Hejna, Matthieu Geist, and Stefano Ermon. Extreme q-learning: Maxent rl without entropy. arXiv preprint arXiv:2301.02328, 2023.
[40] Chenjun Xiao, Han Wang, Yangchen Pan, Adam White, and Martha White. The in-sa softmax for offline reinforcement learning, arXiv preprint 14372, 2023.
12

---

[41] Hongchang Zhang, Yixiu Mao, Boyuan Wang, Shuncheng He. Yi Xu, and Xiangyang Ji. In- sample actor critic for offline reinforcement learning. In The Eleventh International Conference on Learning Representations, 2022.
[42] Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, Victor Wai Kin Chan, and Xianyuan Zhan. Offline rl with no ood actions: In-sample learning via implicit value regularization. arXiv preprini arXiv:2303 15810, 2023.
[43] Philippe Hansen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit q-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304. 10573, 2023.
[44] Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint (rXiv:2006.09359 2020.
[45] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-222 1.-20132-20145, 2021.
[46] Jianxiong Li, Xianyuan Zhan, Haoran Xu, Xiangyu Zhu, Jingjing Liu, and Ya-Qin Zhang. Distance-sensitive offline reinforcement learning. arXiv preprint arXiv:2205.11022 2022.
[47] Wenxuan Zhou, Sujay Bajracharya, and David Held. Plas: Latent action space for offline reinforcement learning. In Conference on Robot Learning, pages 1719-1735. PMLR, 2021.
[48] R.J. Hathaway. A constrained formulation of maximum-likeliben estimation for normal mixture distributions. The Annals of Statistics, 13(2):795-800 800, 1985.
[49] Xuxu Wang and Yong Wang. Nonparametric multivariate density estimation using mixtures. Statistics and Computing, 25(2):349 364, 2015.
[50] Jerome Friedman, Trevor Hastie, and Robert Tibshirani. The Elements of Statistical Learning. volume 1. Springer series in statisties New York, NY, USA:, 2001.
[51] Hall. On trigonometric series estimates of densities. The Anuals of Statisties, 93)):683-685 1981.
[52] G. Wahba. Data-based optimal smoothing of orthogonal sereis density estimates. The Annals of Statistics, 9(1):146-156) 1981.
[53] David Donoho, Iain M. Johnstone, Gerard Kerkyacharian, and Dominique Picard. Density estimation by wavelet thresholding. The Annals of Statistics. 24(2):508 539,, 1996. ISSN 00905364. /R//attp://www jstorrrwwwwss...oo.oo........rr. wtw.:jstooorrro.........o. st/////ebee/222660
[54] Xi Chen, Ali Ghadirzadeh, Tianhe Yu, Yuan Gao, Jianhao Wang, Wenzhe Li, Bin Liang, Chelsea Finn, and Chongjie Zhang. Latent-variable advantage-weighle policy optimization for offline rl. arXiv preprint arXiv:2203..mmmmm 08949, 2022.
[55] Shentao Yang. Zhendong Wang, Huangjie Zheng, Yihao Feng, and Mingyuan Zhou. reg- ularized implicit policy for offline reinforcement learning. arXiv preprint arXiv:2202.0067. 2022.
[56] Shentao Yang, Yihao Feng, Shujian Zhang, and Mingyuan Zhou. Regularizing model- based policy stationary distribution to stabilize offline reinforcement learning. In International Conference on Muchine Leurning, pages 24980-25006. PMLR, 2022.
[57] Avi Singh, Huihan Liu, Gaoyue Zhou, Albert Yu, Nicholas Rhinehart, and Sergey Levine. Parrot: Data-driven behavioral priors for reinforcement learning, arXiv preprint arXiv:2011.10024 2020.
[58] Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint 2rXiv:2205.09991, 2022.
[59] Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. Advances in neural information processing systems, 30,
2017.
13

---

[60] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.064344 2015.
[61] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 13Xiv: 2013.
[62] Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
[63] Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In International conference on machine learning, pages 1995-2003, PMLR, 2016.
[64] R.S. Sutton and A.G. Barto. Reinforcement learning: An introduction. IEEE Transactions Neural Networks, 9, 1998. ISSN 1045-9227. doi: 10. 10.9/09/ 1998.712192.
[65] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International conference on machine learning. pages 387-395. PMLR, 2014.
[66] Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint
2015.
[67] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06344 2017.
[68] Andrew Y Ng. Stuart Russell, et al. Algorithms for inverse reinforcement learning. In Ieml, volume 1, page 2, 2000.
[69] Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. Aaai, volume 8, pages 1433-1438. Chicago, II, USA, 2008.
[70] Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. volume 1, 2016.
[71] Chelsea Finn, Paul Christiano, Pieter Abbeel, and Sergey Levine. A connection between generative adversarial networks, inverse reinforcement learning, and energy-based models. arXiv preprint 111:::161..3385.2 2016.
[72] Jonathan Ho, Jayesh K. Gupta, and Stefano Ermon. Model-free imitation learning with policy optimization. volume 6, 2016.
[73] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
[74] Halbert White. Maximum likelihood estimation of misspecified models. Econometrica: Journal of the econometric society, pages 1-25, 1982.
[75] Tengyuan Liang. How well generative adversarial networks learn distributions. Journal of Machine Learning Research, appear, 2021.
[76] Shuang Liu, Olivier Bousquet, and Kamalika Chaudhuri. Approximation and convergence properties of generative adversarial learning. arXiv preprint arXiv:1705.0888 2017.
[77] Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural networks, 2(5):359-366) 1989.
[78] Balázs Csanád Csáji et al. Approximation with artificial neural networks. Faculty of Sciences, Etvs Lornd University, Hungary, 24(48):7, 2001.
[79] Liang Ding, Lu Zou, Wenjia Wang, Shahin Shahrampour, and Rui Tuo. High-dimensional nar-maraeetric density estimation in mixed smooth sobolev spaces. arXiv preprint arXiv:2006.03696 2020.
14

---

[80] Alekh Agarwal, Nan Jiang, and Sham M. Kakade. Reinforcement learning: Theory and algorithms, 2019.
[81] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
[82] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling, Advances in neural information processing systems, 34:15084-15097, 2021.
[83] David Brandfonbrener, Will Whitney, Rajesh Ranganath, and Joan Bruna. Offline r1 without off-policy evaluation. Advances in neural information processing systems, 34:4933-4946, 2021.
[84] David I. Donoho, lain Johnstone, Gérard Kerkyacharian, and Dominique Picard. Wavelet shrinkage: asymptopia? Journal of the Royal Statistical Society: Series B (Methodological), 57 (2):301-337, 1995.
[85] Yves Meyer. Wavelets and Operators: Volume 1. Number 37. Cambridge university press, 1992.
[86] Sara van de Geer. Empirical Processes in M-estimation. Cambridge University Press, 2000.
[87] Robert A Adams and John JF Fournier. Sobolev Spaces, volume 140. Academic press, 2003.
15


Reviews:
[{"review_id": "C8REKCmJhR", "reviewer": "Reviewer_Y4vE", "summary": "This work proposes to use flow-GAN model to explicitly model the distribution of the behaviour policy and use that to alleviate the OOD actions problem in offline RL.\nA theoretical analysis of convergence is presented. Empirical evaluation on d4rl gym locomotion and antmaze show that the proposed CPED algorithm is very competitive.", "questions": "* What's the computational overhead of training this Flow-GAN?\n* Why do you pick Flow-GAN? Why not use simpler ones like VAE, or even just discretization?", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* The method is novel and very intuitive. Instead of using all different tricks to avoid OOD, explicitly modelling the behaviour policy is much more straightforward.\n* The writing is clear\n* Empirical evaluation shows the performance of CPED is comparable to SOTAs", "weaknesses": "* Compared to prior methods like IQL, DT, or even CQL. The proposed method has an additional generative modelling module, which makes the overall architecture more complex.\n* Again, the empirical results are good but it's technically not surpassing SOTA. For gym-locomotion, there are algorithms like ATAC [1] and SAC-N [2] that have similar level of performance. As for antmaze, the performance of CPED is inferior than IQL, which is not the SOTA for offline RL in general. For model-based methods, there're methods like trajectory transformer [3] that perform better than IQL.\n\n[1] https://proceedings.mlr.press/v162/cheng22b/cheng22b.pdf\n\n[2] https://arxiv.org/pdf/2110.01548.pdf\n\n[3] https://arxiv.org/abs/2106.02039\n\nMinor issues:\n\n> CPED can accurately identify the safe region and enable exploration within the region\n\nI would avoid using the term \"exploration\" here because it can hint to readers that the proposed method can be helpful for online learning but I believe this is not what the authors mean here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your review. We appreciate your valuable questions and suggestions. We have replied your comments in detail, and please see the following for your response. \n\n**Response to Weakness 1**:  We agree that the proposed CPED introduces an additional generative modeling module in the offline RL solution. Nevertheless, this will not lead to an over complex architecture.\n\nIn recent studies of offline RL methods, many generative modeling modules (VAE/GAN/diffusion model) are suggested for modeling behavior policies (BCQ/SPOT), and the sizes of such modules does not pose significant issues. For CPED, the training cost of flow-GAN is close to that of GAN model, and both models are much more efficient than diffusion models. Thus, the complexity of CPED is acceptable. In addition, it is noted that the size of DT will be significantly large when the sequence/input trajectory is long.  \n\n**Response to Weakness 2**:  Thank you for pointing out the related references and methods. We will certainty include the above references and related methods in the revised manuscript (we have cited [2] already). According to references [1] and [2], both ATAC and SAC-N provide competitive performances with CPED on hopper and halfcheetah tasks, while CPED performs slightly better on walker2d tasks. \n\nWe would like to clarify that the SAC-N method utilizes multiple Q-functions, which may contribute to better performances but can also lead to computational issues. In contrast, CPED employs only 2 Q-networks. In addition, the trajectory transformer methods lack strong theoretical interpretability and it requires extra parameters such as “return-to-go”. Therefore, we believe that the proposed CPED has potential for further improvement when multiple Q-functions or other modeling tricks are applied. \nAs we mentioned in the “author rebuttal” in the top, our CPED does not solely aim to be SOTA in experiments. Moreover, we want to show CPED has great potential in further improvement, and we will continue to delve into this aspect in the future.\n\n**Response to Minor issues**:  In response to your feedback, we will make further revisions to the term \"exploration\" to ensure that readers are not misled. The sentence will be revised as “CPED can accurately identify the safe region and allow policy learning within the region”\n\n**Response to Question 1**:  From our experiment, training the flow-GAN will indeed incurs additional computational overhead. Nevertheless, this computational overhead is not a significant concern for the overall training process. In our study, the training time of CPED is only approximately 10% longer compared with the traditional offline RL methods such as BEAR and BRAC. Therefore, the training cost of CPED is quite acceptable.\n\n**Response to Question 2**: Thank you very much for this important question. We have clearly explained the advantage as well the motivation for using Flow-GAN in the “author rebuttal” part. We are glad to re-illustrate the points here so that you can understand our motivation better. \n\nFor the issue why we choose flow-GAN instead of other generative methods (e.g. VAE): \n\n1.\t**Similarities**: In the field of policy control methods for offline RL problem, the idea of ensuring the consistency of the support of learned policy and that of the behavior policy is considered the most desirable approach to tackle distribution shift. Both CPED and SPOT aim to achieve this idea, but they do it differently.\n\n2.\t**Advantage of Flow-GAN**: A clear advantage of Flow-GAN is its capability to provide an explicit and direct density estimator for the behavior policy. In contrast, VAE can only approximate the lower bound behavior density through ELBO.\n\n3.\t**Deficiency of VAE model**: Another experimental evidence comes from ref[1] and ref[2]. In ref[1], the author claims typically-used VAEs do not align well with the behavior dataset, making it challenging to effectively cover the behavior policy distribution. The author further presents an experiment result that the model performance is significantly reduced when the VAE architecture is used (Compared Fig2 to Fig 6 ). In addition, this claim is also validated in ref[2] (in appendix D), where the performance of the method with VAE structure is inferior to that of the original method. \n\n4.\t**Further motivation for Flow-GAN**: We chose to use Flow-GAN as a density estimator because, first, we theoretically analyze in section 3.2 that estimating the density of the behavior policy in RL is equivalent to training a GAN. However, traditional GANs are essentially sample generators. To realize the idea of estimating a density function, we combine the Flow model with GAN to estimate the behavior density. The Flow-GAN is originally designed for image generation problems, in our specific implementation, we made numerous adjustments to the structure of Flow-GAN (ref[3]) , including converting many CNN into fully connected networks and simplified some residual networks, so that overall network structure can adapt to the tasks of offline RL. We believe the flow-GAN model is promising, and hope that future work will further optimize the design of Flow-GAN's structure, allowing it to unleash its true potential on more challenging behavior policy datasets.\n\n5.\t**Discretization**: As for discretization, in the currently prevalent continuous control tasks, this method faces difficulties in accurately estimating the behavior policy density.\n\n\nref[1]: https://arxiv.org/pdf/2007.11091.pdf\n\nref[2]: https://arxiv.org/pdf/2209.14548.pdf\n\nref[3]: https://arxiv.org/abs/1705.08868\n"}, {"review_id": "CT4K7bgW7J", "reviewer": "Reviewer_6qDS", "summary": "This paper proposes a new offline RL algorithm called Constrained Policy optimization with Explicit Behavior density (CPED). The main idea is utilizing a flow-GAN model to estimate the probability density explicitly, which limits the exploration in the safe region. To demonstrate the effectiveness, the author provides some theoretical proofs for the convergence of CPED, and evaluates CPED in various standard offline RL tasks. ", "questions": "1.\tIn Equation 7, the Lagrangian multiplier $\\alpha$ is mentioned in the algorithm but cannot be found. Could you clarify the actual objective function? Additionally, the hyperparameter $\\epsilon$ is not mentioned in your algorithm. How does $\\epsilon$ affect the performance of your algorithm?\n2.\tIn flow-GAN, specific neural networks are utilized for the generator and discriminator. Does this limitation affect their representational ability? Furthermore, does it result in a significant gap between the estimated policy and the behavioral policy?\n3.\tPrevious work often considers online fine-tuning tasks after offline RL. Can CPED be applied to such tasks? If so, how does CPED compare to other offline RL methods in these scenarios?\n4.\tFigure 2 shows the learning curve comparison between BEAR and CPED. However, BEAR is considered somewhat outdated. As SPOT is a more recent offline RL method and seems more relevant to CPED, I'm particularly interested in the comparison between SPOT and CPED.\n5.\tIn Section 5.2, you mention an additional trick using time-varying $\\alpha$. It appears that this trick could be applicable to SPOT as well. Since the empirical improvement is limited, I'm curious if the improvement primarily results from this trick rather than flow-GAN. Could you provide additional results of SPOT with the time-varying $\\alpha\"?\n6.\tThe choice of $\\epsilon$ is not clearly explained. What is the specific quantile of the behavior policy density that you refer to? Does it correspond to the median or other quantiles? It would be helpful if you could provide examples of this \"quantile\" in several datasets.\n", "limitations": "Please refer to the weakness parts.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "This paper is overall well-written, and it performs a comprehensive survey in offline RL and related fields such as inverse RL and generative model. The proposed method is well-motivated, which introduces flow-GAN to offline RL appropriately. This paper provides detailed theoretical analyses of the convergence to confirm the effectiveness of CPED. \n", "weaknesses": "1.\tThe novelty of this paper is somewhat limited as it primarily replaces the VAE model used in SPOT [1] with flow-GAN.\n2.\tWith the exception of the ho-med dataset, CPED does not demonstrate clear improvements and, performs even worse than some previous methods in the experimental datasets.\n3.\tThe implementation of CPED is not clearly stated, and this aspect will require further discussion in the questions.\n4.\tThe usage of certain notations is confusing, and there are some grammar errors.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. Your comments are very useful for refining our manuscript.\n\n**Response to Weakness 1**:Thank you very much for this important question. **We want to emphasize that the original intention of CPED is not to directly apply the Flow-GAN idea to offline RL problem or to simply replace the method for estimating behavior policy in SPOT with GAN**.  We have clearly explained the original contribution of our CPED method and as well the motivation for using Flow-GAN in the “***Author Rebuttal by Authors***” part at the top of this page. \n\nFor the contribution of our paper, please refer the bullet point **1.1 & 1.2 & 1.3 in Section 1** in  “***Author Rebuttal by Authors***” part at the top of this page\n\nFor the issue why we choose flow-GAN instead of other methods: please refer **Similarities/Advantage of Flow-GAN/Deficiency of VAE model/Further motivation for Flow-GAN in Section 2** in  “***Author Rebuttal by Authors***” part at the top of this page.\n\n**Response to Weakness 2**: In the manuscript, the selected competitors (DT/SPOT/IQL) are among the SOTA methods for D4RL tasks, resulting in high baselines. From Table 1&2 in our manuscript, our CPED achieves the best performance for hop-med task, and provides competitive performances with DT and SPOT for hop-med-exp and hop-med-rep tasks. For other tasks such as half-cheetah and walk2d,  CPED delivers the best performances in 4 out of 6 tasks. \n\nIn addition, the total score of the CPED is significantly higher than other competitors, thus it is clear that the proposed CPED performs better than the competitors, and it also contributes to SOTA methods for offline D4RL tasks.\n\n**Response to Weakness 3 & 4**: We will further refine the manuscript, especially revising notations and the implementation part (Algorithm 1, Section 5 and Appendix E),  so that the manuscript is more readable. For further discussion in Questions, please refer to Response to Questions later. \n\n**Response to Question 1**:  We did not specifically mention $\\alpha$ in Equation 7 because of the limited page. In fact, the problem of equation 7 is a constraint optimization problem, and the actual objective function is:\n$$\\begin{aligned} & \\max_{\\psi} \\mathbb{E}_{s \\sim \\mathcal{P}\\_\\mathcal{D} ,\\\\, a \\sim \\pi\\_{\\psi}(\\cdot|s), (s, a) \\in  \\tilde{\\mathcal{S}} \\times \\tilde{\\mathcal{A}} }[Q\\_\\eta(s,a)] \\\\\\\\\n& s.t.    \\tilde{\\mathcal{S}} \\times \\tilde{\\mathcal{A}}= \\\\{(s,a) \\in \\mathcal{S} \\times \\mathcal{A}:  -\\log L\\_{\\theta}^{\\pi\\_\\beta}(s,a) < \\epsilon \\\\} \n\\end{aligned}\\tag{A}$$ \n\nWe believe a natural solution for the constraint problem (A) is to apply Lagragian method. Thus, we did not specifically mention  Lagragian multiplier $\\alpha$ in Equation 7 for simplicity (only show in Algorithm 1). In case other readers mis-understand our idea, we will rewrite Equation 7 as Problem (A) in the manuscript, and emphasize $\\alpha$ is the Lagragian multiplier.\n\nWith respect to the effect of $\\epsilon$, limited by the page, the ablation study of $\\epsilon$ is in Appendix E, in which we consider several choices of $epsilon$ in various D4RL tasks. \n\n**Response to Question 2**: The specific neural network in flow-GAN comes from the flow model. In the original paper of Flow-GAN [1], the power and representation ability of flow-GAN are fully shown from Table 1&Table 2 in [1] in image application. In other references[2,3], even the generator is assumed to be invertible, it has not significantly impacted the performance of Flow-GAN [2,3]. The representation power of Flow-GAN remains highly competitive and effective despite this assumption.\n\n**Response to Question 3**: We really appreciate your useful comment. Since the focus of our manuscript is offline reinforcement learning, online fine-tuning is not a primary topic for us. Considering some offline RL methods (SPOT/IQL) treat online fine-tuning as extensions of their papers, the proposed CPED is also capable of downstream online training and we would like to leave this topic as future work.\n\nFollowing Section 5.3 in SPOT paper, the performance of each method (SPOT/IQL) is further improved after online fine-tuning, and the SPOT performs better than IQL. Since CPED is less conservative than both IQL and SPOT, and the offline performance of CPED is significantly better than SPOT, it is expected that the performance of CPED after online fine-tuning is better than both IQL and SPOT.\n\n**Response to Question 4**: The main purpose of Figure 2 is to demonstrate that our CPED offers more flexibility compared to traditional distance-based control methods, and it overcomes the issue of being too conservative. Consequently, from the learning curve, the CPED provides a broader \"safe area,\" leading to an upward trend in average return, while BEAR, due to its conservatism, does not show significant performance improvement. We believe that SPOT would likely demonstrate similar outcomes as it also tends to be less conservative.\n\n**Response to Question 5**:  We perform extra experiments for SPOT with time varying $\\alpha$ for Antmaze and two D4RL tasks, and the resulting figure is shown in the **pdf file** in ***Author Rebuttal by Authors***. We can see that the performances with time varying $\\alpha$ are close or even inferior to those with constant $\\alpha$, indicating the trick of varying $\\alpha$ does not result too much benefit. Therefore, we can conclude the improvement indeed come from the Flow-GAN model, rather than the tricks in model training.\n\n**Response to Question 6**: We believe you may misunderstand our idea in setting $\\epsilon$. We are not using the quantile of behavior policy density as $\\epsilon$ (used in other methods such as SPOT), instead we use the mean likelihood of the training batch in the experiment. \n\nThe choice of $\\epsilon$ is illustrated in Section 5.2, Line 324-327: “A commonly used setting is ...” \n\n[1]https://arxiv.org/abs/1705.08868\n\n[2]https://arxiv.org/abs/1410.8516\n\n[3]https://arxiv.org/abs/1605.08803\n"}, {"review_id": "deF1YfHKdi", "reviewer": "Reviewer_1U1v", "summary": "This paper proposes a novel approach named Constrained Policy optimization with Explicit Behavior density (CPED) to mitigate the existing policy control methods, e.g., overly conservative or failing to identify OOD areas accurately. CPED use a flow-GAN model to explicitly estimate behavior policy density. The strength of CPED lies in its ability to accurately identify safe regions for explorations, which in turn, leads to less conservative learning policies. ", "questions": "1. Could the authors provide more comparisons between CPED and SPOT and explain why CPED is more feasible? \n2. Regarding policy learning (Eq. 7), why not transfer it to a Lagrange function like SPOT? To the best of my knowledge, explicitly excluding bad transitions has a little edge against other weighting or penalty methods in terms of generalization, which is a key contribution mentioned by the authors. \n", "limitations": "Lack of related work: the in-sample learning paradigm [2-8] is now one of the trends in offline reinforcement learning, those new methods are worth being well mentioned in related work. \n\n---\nI decided to change my score from 4 to 5 after the first round discussion with the authors. \n\n---\n\n[1] Chen H, Lu C, Ying C, et al. Offline reinforcement learning via high-fidelity generative behavior modeling[J]. arXiv preprint arXiv:2209.14548, 2022.\n\n[2] Kostrikov I, Nair A, Levine S. Offline reinforcement learning with implicit q-learning[J]. arXiv preprint arXiv:2110.06169, 2021.\n\n[3] Xu H, Jiang L, Jianxiong L, et al. A policy-guided imitation approach for offline reinforcement learning[J]. Advances in Neural Information Processing Systems, 2022, 35: 4085-4098.\n\n[4] Garg D, Hejna J, Geist M, et al. Extreme q-learning: Maxent RL without entropy[J]. arXiv preprint arXiv:2301.02328, 2023.\n\n[5] Xiao C, Wang H, Pan Y, et al. The in-sample softmax for offline reinforcement learning[J]. arXiv preprint arXiv:2302.14372, 2023.\n\n[6] Zhang H, Mao Y, Wang B, et al. In-sample Actor Critic for Offline Reinforcement Learning[C]//The Eleventh International Conference on Learning Representations. 2022.\n\n[7] Xu H, Jiang L, Li J, et al. Offline rl with no ood actions: In-sample learning via implicit value regularization[J]. arXiv preprint arXiv:2303.15810, 2023.\n\n[8] Hansen-Estruch P, Kostrikov I, Janner M, et al. Idql: Implicit q-learning as an actor-critic method with diffusion policies[J]. arXiv preprint arXiv:2304.10573, 2023.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Well-motivated example in Fig. 1 and overall good writing. \n2. Sufficient theoretical evidence to show that CPED has a fast convergence rate and is able to ﬁnd the optimal Q-function value. ", "weaknesses": "1. The contribution is limited as it seems to directly borrow the idea of flow-gan into offline RL domains.\n2. Besides the motivation example in Fig. 1, there is no faithful evidence to show that \"accurately identify the feasible region, which includes both observed and unobserved but safe points\" both in theoretical analysis and experiments. \n3. Lack of survey and comparison with the most similar work [1], which also learns an expressive generative behavior model but by the diffusion technique.\n4. Though there is an obvious edge of CPED against SPOT in experiments, more explanations are lacking to suppose it. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their efforts and time reviewing our paper.\n\n**Response to Weakness 1**: Thank you for your comment. **We want to emphasize that the original intention of CPED is not to directly apply the Flow-GAN idea to offline RL problem** for the following reasons. \n\n1. **Advantage of Flow-GAN**: One key reason for selecting flow-GAN comes from its advantages. A clear advantage of Flow-GAN is its capability to provide an explicit and direct density estimator for the behavior policy. In contrast, other generative models (VAE) can only approximate the lower bound behavior density through ELBO.\n\n2. **Further motivation for Flow-GAN**: We choose to use Flow-GAN as a density estimator because, first, we theoretically analyze in section 3.2 that estimating the density of the behavior policy in RL is equivalent to training a GAN. However, traditional GANs (including VAE) are essentially sample generators. To realize the idea of estimating a density function, we combine the Flow model with GAN to estimate the behavior density. \n\n3. **Modifications to traditional Flow-GAN**: The Flow-GAN is originally designed for image generation problems, in our specific implementation, we make numerous adjustments to the structure of Flow-GAN (ref[1]) , including converting many CNN into fully connected networks and simplified some residual networks, so that overall network structure can adapt to the tasks of offline RL. \n\n4. **Other Generative Models in offline RL**: In recent offline RL study, generative models become popular for behavior modeling owing to their strong model capacity, such as BCQ (using VAE), SPOT(using VAE), ref[3](using diffusion model). However, it is inappropriate to claim these studies simply borrow the VAE/diffusion model into offline RL domains. \n\\end{itemize}\n\nWith respect to the contribution of our manuscript, limited by the length of rebuttal, please find the bullet point **1.1** & **1.2** & **1.3** in “***Author Rebuttal by Authors***” part at the top of this page. \n\nThus, we do not agree with the claim “*CPED borrow the idea of flow-gan into offline RL domains*”, and we believe the contribution of our CPED is underestimated.\n\n**Response to Weakness 2**:  In theory, we have demonstrated that the estimated density is accurate. Thus, as long as the density satisfies certain smoothness properties, we can prove that the estimated region is accurate. However, in practice, we are unable to observe all policies, making it challenging to quantify how well our estimation is. Nevertheless, the effectiveness of the experiments leads us to believe that the estimated policy is accurate, as the performance of offline RL heavily relies on the accuracy of the estimated region.\n\n**Response to Weakness 3**: Thank you for providing related references with diffusion technique. Actually, we have already cited [1] in the manuscript. We believe [1] is an important baseline for our CEPD. Compare [1] and CPED, we see that the CPED outperforms [1] for most D4RL and Antmaze tasks, which further shows the effectiveness of our CPED.\n\n**Response to Weakness 4**: We have provided further explanations about our contribution as well the comparison between flow-GAN and VAE (thus CEPD and SPOT) in both the response to Weakness 1 and “***Author Rebuttal by Authors***” part at the top of this page. Here we summarize the key points (limited by the rebuttal size, we do not list details for each point):\n\n1.\t**Similarities**: Please refer the \"Similarities\" part in Section 2 of “***Author Rebuttal by Authors***” part at the top of this webpage.\n\n2.\t**Advantage of CPED and Flow-GAN**: Please refer the part \"Advantage of Flow-GAN\" in **Response to Weakness 1** .\n\n3.\t**Deficiency of SPOT**: Please refer the \"Deficiency of VAE\" part in Section 2 of “***Author Rebuttal by Authors***” part at the top of this page.\n\n4.\t**Adaption of Flow-GAN and CPED for Estimating Behavior Policy**: Please refer to Further motivation for Flow-GAN in \"**Response to Weakness 1**\" above.\n\n5.\t**Modifications to traditional Flow-GAN** : Please refer to Modifications to traditional Flow-GAN in \"**Response to Weakness 1** \"above.\n\nTherefore, it is convincing that our proposed CPED has a clear edge against SPOT in both theory and experiments.  \n\n**Response to Question 1**: Please refer our responses for Weakness 1 and Weakness 4 for our motivation for using CPED (flow-GAN) and more comparisons between CPED and SPOT (flow-GAN vs VAE).\n\n**Response to Question 2**:  We are afraid you may mis-understand our idea for optimizing Equation 7 in the manuscript. The actual objective function of Equation 7 is: \n\n$$\\begin{aligned}\n & \\max_{\\psi} \\mathbb{E}_{s \\sim \\mathcal{P}\\_\\mathcal{D} ,\\\\, a \\sim \\pi\\_{\\psi}(\\cdot|s), (s, a) \\in  \\tilde{\\mathcal{S}} \\times \\tilde{\\mathcal{A}} }[Q\\_\\eta(s,a)] \\\\\\\\\n& s.t.    \\tilde{\\mathcal{S}} \\times \\tilde{\\mathcal{A}}= \\\\{(s,a) \\in \\mathcal{S} \\times \\mathcal{A}:  -\\log L\\_{\\theta}^{\\pi\\_\\beta}(s,a) < \\epsilon \\\\} \n\\end{aligned}\\tag{A}$$\n\nwhich is a constrained optimization problem, and we are indeed using Lagrangian techniques to solve the constraint problem (A). \nFor the constraint in problem (A), it is not hard constraints that exclude bad transitions as you mention. Instead it works as penalty terms in the Lagrange function (with Lagrangian multiplier $\\alpha$) . In case other readers mis-understand our idea, in the revised manuscript, we will rewrite Equation 7 as Equation A under the form of constraint optimization problem. \n\n**Response to Lack of related work**: Thank you for your comment. We have already cited [1] and [2] in the manuscript. We appreciate you pointing out the research direction of in-sample learning paradigm, we will further cite [3-8] in the revised manuscript. \n\nref[1]: https://arxiv.org/abs/1705.08868"}, {"review_id": "6RcBmHk5BJ", "reviewer": "Reviewer_krNs", "summary": "The paper proposes a novel algorithm for offline model-free reinforcement learning by constraining the actor to lie within a safe area defined by an explicit behavioral density. This behavioral density is obtained by thresholding a FlowGAN model. The resulting algorithm is competitive across various D4RL datasets.\n", "questions": "- Could a dynamics model-based uncertainty (e.g. MOReL) provide a simple alternative to the Flow-GAN threshold?\n- Could the alpha tuning scheme be replaced by something not hand-crafted to increase the applicability of the method?\n", "limitations": "Not discussed in the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Clearly presented algorithm and well-motivated method\n- Strong empirical evaluation and competitive results on D4RL MuJoCo\n- Solid theoretical justification of the convergence of the FlowGAN and RL training\n", "weaknesses": "- Model-based offline methods typically use a state-action-based uncertainty penalty which could also be repurposed for the definition of the safe area. E.g. thresholding the model uncertainty as in MOReL [1]. This would be a useful baseline to show the benefit of the Flow-GAN method\n- The hand-crafted per-environment piecewise linear scheme for alpha limits the generality of the method\n- Algorithm 1 seems to be incorrect, the preceding discussion has the Flow-GAN optimization happening before RL training, whereas Algorithm 1 has the Flow-GAN optimization interleaved with RL training.\n- Line 150: definition of the ‘offline MDP’ is not a real MDP. For example, the state and action spaces should both be sets and the definition is a tuple. It is unclear what this definition adds beyond describing the marginal state distribution and behavioral policy.\n- In Theorem 4.2, the optimal Q-function on the “restricted product space” needs to be defined and explained.\n\nMinor:\n- Line 145: ‘triples’ is likely a mistake, the transition is a 4-tuple\n- Tables 1 and 2 missing EDAC [2], the current SOTA model-free offline RL method\n\n[1] MOReL: Model-Based Offline Reinforcement Learning. Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, Thorsten Joachims.\n\n[2] Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble. Gaon An, Seungyong Moon, Jang-Hyun Kim, Hyun Oh Song.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. We appreciate your kind suggestions for improving our manuscript. \n\n**Response to Weakness 1**:  We agree that MORel is a useful baseline for CPED, and we will certainly include the MORel[1] in our manuscript as baseline method. From our understanding, MoReL provides us good estimates of the offline RL dynamics, enabling us to have a better understanding of the safety of the state space as well. In this sense, both CPED and MORel want to identify safe areas, while they implement in different approaches. \n\n**Response to Weakness 2**:  Among policy control methods for solving offline RL problems, some works (e.g., BEAR, UWAC) determine the value of alpha through automated learning (Lagrangian duality optimization). However, BRAC (ref[1]) has demonstrated through experimental studies that the performance of automatically optimized alpha is not as effective as manually determined alpha values. Therefore, the CPED selects the hand-craft threshold alpha. To better control the training process, CPED further proposes the piecewise linear scheme for adjusting alpha (To our best knowledge, the CPED is the first one using a piecewise linear scheme for determining alpha in offline RL solutions). In section 5.2 and Appendix E.5, we present the details of setting the piecewise linear alpha, and the experiments show that this scheme delivers better returns in most scenarios. \n\n\n**Response to Weakness 3:**  We appreciate you pointing out the issue of training order when training Flow-GAN and RL models. When implementing the CPED, the Flow-GAN is joined trained with RL model. Specifically, the Flow-GAN is first trained with several epochs until it becomes stable, then the RL model is interleaved trained with Flow-GAN. \n\nConsidering the Flow-GAN starts training first, in the manuscript, the description of Flow-GAN appears before the RL model. We will further revise Algorithm 1 and clearly state that the Flow-GAN model is firstly trained for a few epochs (M epochs), and then interleaved trained with RL model. Please find the attached **pdf file** ( in “**Author Rebuttal by Authors**” part at the top of this page) for the revised figure of Algorithm 1.\n \n\n**Response to Weakness 4**: Thank you for pointing out the unclear statement in the manuscript. The offline MDP is not real MDP as it is under online setting. The action/state under offline MDP is bounded in the probability space. To better illustrate the concept of “offline MDP”, we have revised the definition in Line 150. Please find the attached **pdf file** ( in “**Author Rebuttal by Authors**” part at the top of this page) for the revised Definition of “Offline MDP”\n\nWe want to further explain that the main purpose of proposing the definition of offline MDP is to highlight the distinction between offline RL and online RL. In offline RL, the state and action spaces are bounded, and this boundary is constrained by the probability measure defined over these spaces. Therefore, in subsequent theoretical proofs and algorithm designs, we work within this bounded state and action space.\n\n\n**Response to Weakness 5**:  Actually, we have explained that “restricted product space” in theorem 4.2 refers to ${\\mathscr{S} \\times \\mathscr{A}}$, where \\mathscr{S} and \\mathscr{A} are measured state space and measured action space defined in “Offline MDP” in Line 150 respectively. \n\n**Response to Minor Issue 1**: We really appreciate you for pointing out the grammar error. We will further revise the sentence as “Dataset $\\mathcal{D}$ contains many 4-tuples $(s,a,s',r)$ that can be viewed as independent.” \n\n**Response to Minor Issue 2**: Thank you for pointing out the related method and references. We will certainty include the above references and the EDAC methods in the revised manuscript (we have cited [2] already). From Table 1 in [2], the performance of EDAC is competitive with the proposed CPED and both methods contribute to the SOTA methods for offline RL task. \n\nIt is also noted that multiple Q-functions are utilized in EDAC method, while there are only 2 Q-networks in CPED. Thus, motivated by [2], we believe the proposed CPED has space for further improvement when multiple Q-functions are applied. \n \n**Response to Question 1**: We appreciate this interesting question. As far as we know, model-based uncertainty methods (e.g., MOReL) focus more on state space when defining safe regions, while CPED pays more attention to the safety of the action space. In fact, during the training phase of offline RL method, OOD actions are the main source of model failure, whereas OOD states are more critical during the testing phase. Though MoReL also considers the uncertainty of action space in its dynamics model-based uncertainty, our view on whether MoReL can provide an alternative to Flow-GAN in this regard is relatively conservative.\n\nFurthermore, we believe that the key to defining a safe area lies in the ability to estimate the distribution of the safe area well. MoReL uses Gaussian dynamics models, and whether it can be used as alternative for flow-GAN threshold depends on whether the Gaussian dynamics models have enough representational power to learn complex distributions.\n\n**Response to Question 2**:  As we mentioned in Question 2, we can apply the Lagrangian duality optimization for updating alpha to increase the applicability of CPED (e.g. BEAR and UWAC). Nevertheless, according to BRAC (ref[1]), this updating strategy may not lead to a good model performance. Thus, we select the hand-craft alpha in CPED when implementing the method. Especially we further propose piecewise linear scheme to determine alpha dynamically, and the effectiveness of the piecewise linear scheme is also validated from experiments. \n\nref[1]： https://arxiv.org/abs/1911.11361\n"}]

Author rebuttal:
We are grateful for the valuable questions and suggestions were given by all four reviewers that help us to revise our manuscript. After reading all the reviews, we have answered each reviewer's questions in detail, and also added relevant experiments based on each reviewer's suggestions to support our work.

As some reviewers still have concerns on the contribution of our manuscript as well as selecting the Flow-GAN instead of other methods (e.g. VAE), **we want to emphasize that the original intention of CPED is not to directly apply the Flow-GAN idea to offline RL problem or to simply replace the method for estimating behavior policy in SPOT with GAN**. Furthermore, we want to illustrate following issues:

1. For the contribution of our manuscript, through our exploration of the CPED method, we aim to provide the following conclusions from both theoretical and experimental perspectives:
    - **1.1** The combination of GAN and MLE-based density estimation (e.g. Flow-GAN) is more suitable for estimating behavior policies in offline RL, both in theory and experiments. We provide a theoretical guarantee indicating that the CPED can access the optimal Q-function, and extensive experiments show that CPED substantially outperforms state-of-the-art methods.
    - **1.2** Keeping the support of the learned policy close to that of the behavior policy is the key idea in policy control methods. In our proposed CPED, by introducing an explicit density function, we can effectively achieve this objective.
    - **1.3** CPED is not only intended to demonstrate its superiority through experimental results but also aims to advance the completeness of policy control methods. In fact, we believe that the Flow-GAN density estimator has significant potential, considering its representation of the policy density learning process in theory and its promising performance in experiments. We also hope that our work can serve as a catalyst, driving the development of the idea of estimating behavior policies to address offline RL problems.

    The above three points are our fundamental contributions to the offline RL community.

2. For the issue why we choose flow-GAN instead of other generative methods (e.g. VAE):
    - **Similarities**: In the field of policy control methods for offline RL problem, the idea of ensuring the consistency of the support of learned policy and that of the behavior policy is considered the most desirable approach to tackle distribution shift. Both CPED and SPOT aim to achieve this idea, but they do it differently. CPED utilizes flow-GAN to estimate the density of behavior policy while SPOT selects the VAE as the tools. 
    - **Advantage of Flow-GAN**: A clear advantage of Flow-GAN is its capability to provide an explicit and direct density estimator for the behavior policy. In contrast, VAE can only approximate the lower bound behavior density through ELBO. 
    - **Deficiency of VAE model**: Another experimental evidence comes from ref[1] and ref[2]. In ref[1], the author claims typically-used VAEs do not align well with the behavior dataset, making it challenging to effectively cover the behavior policy distribution. The author in ref[1] further presents an experiment result that the model performance is significantly reduced when the VAE architecture is used (Compared Fig2 to Fig 6). In addition, this claim is also validated in ref[2] (in appendix D), where the performance of the method with VAE structure is inferior to that of the original method. 
    - **Further motivation for Flow-GAN**: We choose Flow-GAN as a density estimator because, first, we theoretically analyze in section 3.2 that estimating the density of the behavior policy in RL is equivalent to training a GAN. However, traditional GANs (including VAE) are essentially sample generators. To realize the idea of estimating a density function, we combine the Flow model with GAN to estimate the density of behavior policy. The Flow-GAN is originally designed for image generation problems, in our specific implementation, we make numerous adjustments to the structure of Flow-GAN (ref[3]) so that it can adapt to the offline RL tasks. We hope that future work will further optimize the design of Flow-GAN's structure, allowing it to unleash its true potential on more challenging behavior policy datasets.

* ref[1]: https://arxiv.org/pdf/2007.11091.pdf
* ref[2]: https://arxiv.org/pdf/2209.14548.pdf
* ref[3]: https://arxiv.org/abs/1705.08868

Meta-review: ### Summary

The paper introduces an offline reinforcement learning (RL) method called Constrained Policy optimization with Explicit Behavior density (CPED) to address the challenge of estimating Out-of-Distribution (OOD) data points. Existing solutions for this problem either overly restrict policies or struggle to identify OOD regions accurately. CPED employs a flow-GAN model to explicitly estimate the density of behaviour policy, allowing it to precisely identify safe exploration regions and thereby achieve less conservative learning policies. The theoretical analysis demonstrates the convergence of CPED, and empirical evaluations on standard offline RL tasks, including d4rl gym locomotion and ant-maze, show that CPED is highly competitive, outperforming existing alternatives and yielding higher expected returns.


### Decision

The paper is well-written and easy to understand. The CPED algorithm technically reasonable and achieves significantly better results than other algorithms. The reviewers had some concerns regarding to the novelty of the algorithm concerned with other similar algorithms like SPOT. However, the authors did a good job addressing these concerns. The ideas in this paper is worth sharing with the rest of the NeurIPS community. I would recommend the authors to address some of the concerns raised by the reviewers in the camera-ready version of the paper.
Decision: Accept (poster)