Title: Constrained Policy Optimization with Explicit Behavior Density For Offline Reinforcement Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Offline Reinforcement Learning

GAN

Flow Model

Policy Control
Abstract: Due to the inability to interact with the environment, offline reinforcement learning (RL) methods face the challenge of estimating the Out-of-Distribution (OOD) points. Existing methods for addressing this issue either control policy to exclude the OOD action or make the $Q$ function pessimistic. However, these methods can be overly conservative or fail to identify OOD areas accurately. To overcome this problem, we propose a Constrained Policy optimization with Explicit Behavior density (CPED) method that utilizes a flow-GAN model to explicitly estimate the density of behavior policy. By estimating the explicit density, CPED can accurately identify the safe region and enable exploration within the region, resulting in less conservative learning policies.  We further provide theoretical results for both the flow-GAN estimator and performance guarantee for CPED by showing that CPED can find the optimal $Q$-function value. Empirically, CPED outperforms existing alternatives on various standard offline reinforcement learning tasks, yielding higher expected returns.

Constrained Policy Optimization with Explicit Behavior Density for Offline Reinforcement Learning

Abstract

Due to the inability to interact with the environment, offline reinforcement learning (RL) methods face the challenge of estimating the Out-of-Distribution (OOD) points. Existing methods for addressing this issue either control policy to exclude the OOD action or make the Q function pessimistic. However, these methods can be overly conservative or fail to identify OOD areas accurately. To overcome this problem, we propose a Constrained Policy optimization with Explicit Behavior density (CPED) method that utilizes a flow-GAN model to explicitly estimate the density of behavior policy. By estimating the explicit density, CPED can accurately identify the safe region and enable optimization within the region, resulting in less conservative learning policies. We further provide theoretical results for both the flow-GAN estimator and performance guarantee for CPED by showing that CPED can find the optimal Q -function value. Empirically, CPED outperforms existing alternatives on various standard offline reinforcement learning tasks, yielding higher expected returns.

1 Introduction

As a form of active learning, reinforcement learning (RL) has achieved great empirical success in both simulation tasks and industrial applications [1, 2, 3, 4, 5, 6, 7]. The great success of RL is largely due to sufficient information exploration and the online training paradigm that the agent has plenty of interactions with the environment. However, the merits of RL methods are limited when it is difficult or costly to sequentially interact with the environment. In many real-world scenarios such as driverless or intelligent diagnostics [8], online interaction is unacceptable. Thus, vanilla RL methods may possibly fail under such circumstances.

One recently developed approach to mitigating the constrained online-interaction is offline RL [8, 9, 10]. Motivated by various data-driven techniques, the offline RL leverages prior experiences to train a policy without interacting with the environment. In offline RL, a critical challenge is distribution shift (also called 'extrapolation error' in literature). Specifically, vanilla RL methods

using the Bellman equation and the Q -function approximation probably fail to deliver good estimates for Out-of-Distribution (OOD) points. To this end, two types of offline RL solutions have been proposed. One type is the Q -function constraint. By adding constraints on the Q -function, this type of solution provides a pessimistic estimation of the Q -function and prevents the Q -value from being too large. The constraints include ensembling multiple Q -functions [11, 12], penalizing Q -function with high uncertainty [13, 14], among others. Recently, implicit Q-learning [15] and mildly conservative Q-learning [16] are suggested. However, the uncertainty of the Q -function or a tight lower bound of Q -value is difficult to estimate. Thus the performance is not guaranteed neither theoretically nor practically.

Another type of solution is policy control, where the learning policy is controlled over the support of the behavior policy. Thus the OOD points will not be visited and the distribution shift problem is alleviated [17, 18, 19]. It is noted that previous policy control methods limit the safe areas within the close neighborhoods of the training dataset by distribution distance metrics such as KL divergence, maximum mean discrepancy (MMD), and Wasserstein Distance [20, 17, 18]. Such policy control approaches often lead to an over-conservative and suboptimal learning policy, and the optimal policy value may not be reached. For example, if some state-action points are within the training distribution but unobserved in the training dataset, they may be considered OOD points in previous policy control approaches. Nevertheless,these points are safe when the estimated Q -function can well generalize on this area [12, 21], and the optimal policy likely lies in the unseen area of the data space. In this sense, a global estimator of the conditional action data space is required to define the safe areas and examine whether an unseen point is safe. To this end, Wu et al. [22] imposes the density-based constraints and proposes the SPOT method. However, the density estimator in SPOT approximates the lower bound of the behavior policy density, and policy control methods with more accurate density estimators are worth further discussing. As in the field of policy control methods for offline RL problem, the idea of ensuring the consistency of the support of learned policy and that of the behavior policy is considered the most desirable approach to tackle distribution shift. Therefore, a direct estimation density of the behavior policy is desired for guarantee support consistency.

Recently, generative models have shown strong capabilities in diverse fields, in which variational autoencoder (VAE) and generative adversarial network (GAN) [23, 24, 25, 26] are popular tools utilized for sampling and other related applications. In RL areas, both VAE and GAN have been introduced in RL methods [19, 16, 27, 28]. However, as random sample generators, VAE and GAN cannot be directly applied to estimate behavior densities in offline RL, nor can they provide accurate density estimates. Wang et al. [29], Yang et al. [30], Chen et al. [31], Ghasemipour et al. [32] have shown that VAE has shortcomings in estimating complex distributions, especially multimodal distributions or complex behavior policy distributions, and discuss that typically-used V AEs may not align well with the behavior dataset. As V AE is prone to cover mode, this phenomenon diminishes the VAE's capacity to accurately estimate data distributions, particularly in scenarios involving multimodal data distributions, such as behavior policy distributions. Compared to VAE, GAN has more advantages in learning complex data distributions. Recently, various GAN variants have been suggested. Among them, the Flow-GAN [33] model combines the normalizing flow model [34, 35] and GAN, and provides an explicit density estimator. The Flow-GAN model is further renowned for its robustness and tuning flexibility and has garnered increasing attention.

In this work, we propose a novel approach called Constrained Policy optimization with Explicit behavior Density (CPED 2 ) for offline RL. The CPED suggests a strong density-based constraint and employs an explicit and accurate density estimator to identify the safe areas. Specifically, CPED utilizes the Flow-GAN model as the density estimator, whose accuracy is theoretically guaranteed. By leveraging this capability, CPED can accurately identify the feasible region, which includes both observed and unobserved but safe points (illustrated in Figure 1 c). By expanding the feasible region and enabling reasonable exploration within it, CPED exhibits reduced risk-aversion and demonstrates an enhanced ability to generate superior policies.

To summarize, the merits of the proposed CPED include the following.

By introducing the Flow-GAN model, the CPED directly tackles the density estimation problem and proposes a novel policy control scheme for solving offline RL tasks.

The CPED does not behave over-conservatively and allows exploration in the safe region, especially high-density areas, to find the optimal policy.

Practically, the effectiveness of CPED is fully demonstrated empirically. Extensive experiments on competitive tasks show that CPED substantially outperforms state-of-the-art methods.

From the theoretical perspective, we show that GAN with a hybrid loss can accurately estimate density, which itself contributes to the GAN community. Furthermore, we prove the policy learned from CPED can access the optimal value function, theoretically ensuring the efficiency of CPED.

State

Action

State

Action

Based Methods

State

Action

c) Our CPED Method

Offline Data Point

Ground Truth Safe Area

Exploration Area of Neighbourbood-

Based Mehtods

Optimal Policy

Safe Area of CPED

Unobserved but Safe Point

2 Related Works

Value Function Constraint . Taking OOD actions can lead to overestimated value function and introduce bootstrap error. Value function constraint methods aim to construct pessimistic value functions or reduce the uncertainty of the value function to avoid OOD actions. Following [11, 12, 36], a pessimistic value function can be obtained by ensembling multiple value functions. CQL [13] proposes a regularization term so that the estimated Q -function low-bounds its true value. TQC [37] follows the distributional RL method and learns the percentile of value function distribution. IQL [15] combines expectile regression with exponential weighted advantage function to avoid extreme value function. MCQ [16] tries to use a special bellman operator to separate the estimation of the target value function for the in-distribution point and OOD point. The value function shows great uncertainty when taking OOD points. UWAC [14] estimates the uncertainty of the value function using the dropout technique. Urpí et al. [38] applies the tail risk measurement for the Q -function, in which the value function with high uncertainty is penalized, and the learned value function is forced to be pessimistic. However, finding the lower bound of the value function or estimating the uncertainty is not easy. The error from estimating the value function and finding the lower bound hinders the performance of value function constraint methods. Recently, a series of in-sample learning methods [39, 40, 41, 42, 43] have been proposed to optimize the Q -value function or the policy value function V solely utilizing in-sample data. As a result, the extrapolation error, induced by out-of-distribution (OOD) data points in the Q -value function, can be mitigated.

Policy Control Method . To prevent the learning of policies that take out-of-distribution (OOD) actions, the policy control method incorporates a penalty term during policy learning and restricts the distribution distance between the learned policy and the behavior policy. The BEAR method [17] uses the MMD distance to constrain the difference between the behavior policy and the learned policy. [18, 44, 20, 45, 46] measures the distribution shift between learning policy and behavior policy with KL divergence and Wasserstein distance. DOGE [46] learns a distance function to control the learning policy staying close to the behavior policy. Besides the explicit distance-based policy control methods, some implicit control methods [19, 47, 22], which generate the learning policy action based on the estimated conditional probability distributions, are suggested. However, explicit distance control methods suffer from the generalization problem since the distance is estimated from the limited training samples. On the other hand, implicit control methods tend to generate over-conservative

policies, where the generated actions are too close to the behavior policy. Consequently, the learned policy by these methods discourages exploration and remains conservative.

Behavior Policy Density Estimation In offline RL setting, determining the action space A under given states is crucial. This is equivalent to learning the distribution of the behavior policy. However, traditional density estimation methods [48, 49, 50, 51, 52, 53] often fail in this scenario of density estimation due to the curse of high dimensionality. Recently, deep neural network-based density estimation methods have proven successful in learning the distribution of the behavior policy. Conditional variational autoencoders (CVAE) have been utilized as both behavior policy samplers [19, 17, 47, 16, 54] and density estimators [22]. Generative adversarial networks (GAN) are also introduced in offline RL scenario [55, 56]. Singh et al. [57] applies the generative normalized flow model to learn an invertible map from the noisy prior to the behavior policy action space and do behavior policy sampling by the trained model. EMaQ [32] uses the Autoregressive Generative Model instead of CVAE as a sampler. Additionally, the diffusion model has been applied to estimate the density of the behavior policy in both model-free RL [31] and model-based RL [58]. Specifically, SPOT [22] approaches the lower bound of the behavior density with ELBO derived from CVAE. However, to the best of our knowledge, most of the aforementioned methods either estimate the behavior density implicitly or lack the ability to provide accurate estimations. In our proposed method, the CPED finds an exact density function using the Flow model [34, 35] and GAN [23, 24, 25, 59, 60].

3 Probability Controlled Offline RL Framework

3.1 Offline RL Problem Settings

In RL setting, the dynamic system is described in terms of a Markov decision process (MDP), which can be defined by a set of tuples {S , A , T, p 0 , r, γ } . Here, S and A denote the state space and action space, respectively; T is a conditional transition kernel, and T ( s ′ | s, a ) is the conditional transition probability between states, describing the dynamics of the entire system; p 0 is the distribution of the initial states; r ( s, a ) is a deterministic reward function; and γ ∈ (0 , 1) is a scalar discount factor.

The goal of RL is to find a distribution of actions π ( a | s ) (named as policy) that yields the highest returns within a trajectory, i.e., maximizing the expected cumulative discounted returns. We denote this expected discounted return starting from ( s, a ) under policy π as Q π ( s, a ) . Under the Q-learning framework, the optimal expected discounted return Q ∗ ( s, a ) can be obtained by minimizing the L 2 norm of Bellman residuals:

where B is the Bellman operator defined as

In the context of deep RL, various solutions are provided to estimate the Q -function [61, 62, 63]. When the action space is continuous, the learning process is divided into actor ( Q value function) training and critic (learning policy) training [64]. The critic network is learned from Eq.1 without 'maximum' in the Bellman operator. The actor is formulated with various forms [65, 66, 67, 1] such as maximizing the Q -value (DDPG), the weighted log of action density (policy gradient), and clipped weighted action density ratio (PPO).

In the offline RL, the learning policy cannot interact with the dynamic system yet only learns from a training dataset D ( s, a, s ′ , r ) . Dataset D contains many triples ( s, a, s ′ , r ) that can be viewed as independent. Assume that the dataset D is generated by a behavior policy π β . We need to redefine the MDP corresponding to D under the offline RL settings.

Definition 3.1 (Offline MDP) . Given a dataset D with continuous state and action space. Let π β denote the behavior policy that generated D . The MDP of dataset D is defined by M D := { S , A , T D , p 0 D , r, γ } , where S = {S , P D } is the measured state space related to D with probability measure P D , P D is the marginal probability of each state in state space of D , A = {A , π β } is the measured action space related to D with conditional probability measure π β ( a | s ) , T D ( s ′ | s, a ) is the transition probability between states, describing the dynamics of D , and p 0 D is the distribution of the initial states of D .

In offline RL, we can obtain the optimal Q -function by minimizing Eq.1. However, due to the distribution shift problem [17, 8], the Bellman residuals are biased during the training phase. Specifically, at k -th iteration, since π k is trained to maximize the Q -function, it is possible that π k visits some OOD points with high estimated Q -function values. Nevertheless, we cannot evaluate the rewards of the OOD points under offline scenarios. Consequently the Bellman residuals at the OOD points may dominate the loss, and the learned Q -function value is mislead. Therefore, the optimization of offline RL must occur in the bounded safe state-action space S × A , such that the estimated Q -function is trustworthy.

3.2 Behavior Policy Estimation by GAN with Specific Density

In offline RL, the dataset D is generated by the behavior policy, whose density can be estimated via inverse RL [68]. One widely used approach in inverse RL, maximum entropy inverse RL (MaxEnt IRL, [69]), models the density via a Boltzmann distribution

where the energy is given by the cost function c θ ( τ ) = ∑ H t =0 c θ ( s t , a t ) , τ = ( s 0 , a 0 , s 1 , a 1 , ..., s H , a H ) is a trajectory, and Z is a scale factor. The cost function c θ ( τ ) = ∑ H t =0 c θ ( s t , a t ) is a parameterized cost function that can be learned by maximizing the likelihood of the trajectories in D . And the scale factor Z is the sum of exponential cost exp( -c θ ( τ )) of all the trajectories that belong to the product space of the state and action of D . However, estimating Z is difficult because it is scarcely possible to traverse the entire action space A . An alternative way to estimate Z is via guided cost learning [70], where it has been shown that learning p θ ( τ ) is equivalent to training the dataset D with a GAN [70, 71, 72, 73].

Originally, the GAN is a random sample generator for high-dimensional data, and it consists of two models: a generator G and a discriminator D . The generator is trained to generate random samples that are close to those drawn from the underlying true distribution, and the discriminator tries to classify whether the input is generated sample or an actual one from the training dataset. The GAN cannot provide an explicit density estimation, which is an indispensable ingredient when controlling the probability of actions and specifying an safe area. Thus, we cannot directly apply the MaxEnt IRL to our CPED. To overcome the difficulty, we follow the approach in [33] and consider GAN with flow model as a generator (we call if Flow-GAN hereafter as in [33]).

Flow-based generative models have shown their power in the fields of image processing and natural language processing [34, 35]. The normalizing flow model allows us to start from a simple prior noise distribution and obtain an explicit estimate of the density function after a series of non-linear invertible transformations.

Specifically, let f θ : R A ↦→ R A be a nonlinear invertible transformation, where A is the dimension of action space A , and θ is parameter. By change of variables, the probability density on D can be obtained by

where τ ∈ D is a trajectory. Since f θ is invertible, a generator can be constructed by G θ = f -1 θ . In practice, both generator and discriminator in GAN are modeled as deep neural networks. Two specific neural network structures, NICE [34] and Real_NVP [35], are proposed to ensure that G θ (or f θ ) is invertible. The Jacobian matrices of the mapping functions f constructed by these two methods are both triangular matrices, guaranteeing efficient computation of | det ∂f ( τ ) ∂τ | .

Directly training a Flow-GAN by original loss function in GAN may lead to poor log-likelihoods, and GAN is prone to mode collapse and producing less diverse distributions [33]. Thus, following the approach in [33], we adopt a hybrid loss function containing a maximum likelihood loss and a GAN structure loss as the optimization objective for estimating the probability measure of behavior policy:

where L G ( G θ , D ϕ ) can be any min-max loss function of GAN, D ϕ is a discriminator indexed by parameter ϕ , and λ > 0 is a hyperparameter balancing the adversary learning process and the maximum likelihood process.

Remark 3.1 . The reason for not simply using MLE is that MLE tends to cover all the modes of distribution (even though some modes have small probabilities), and MLE is not robust against model misspecification [74], which often occurs in high dimensional scenarios.

In the following proposition, we show that the estimated density of the behavior policy using the hybrid loss Eq.4 is equivalent to that using MaxEnt IRL. The proof of Proposition 3.1 is provided in Appendix B.

Proposition 3.1. For offline dataset D generated by behavior policy π β , the learned likelihood function L π β , using GAN with hybrid loss in Eq.4 is equivalent to that trained by MaxEnt IRL. If the generator of GAN can give a specific likelihood function p G θ ( τ ) , then

where C is a constant related to D .

3.3 Constrained Policy Optimization with Explicit Behavior Density Algorithm (CPED)

In this work, we propose an algorithm for probabilistically controlling the learned policy, CPED. In CPED, we first utilize the Flow-GAN and obtain a probability estimate of the trajectory p θ ( τ ) . The optimize objective follows the hybrid loss in Eq.4:

Since the episode τ is a sequential collection of states s and actions a , we rewrite the input of L π β θ ( · ) with ( s, a ) instead of τ in Eq.6 and afterwards. From Proposition 3.1, the solution to Eq.6 is equivalent to the behavior estimator obtained from MaxEnt IRL, thus providing an effective estimation of the behavior policy.

Following the learning paradigm in classic RL methods, the policy learning process is divided into two parts: the actor training and critic training. With the estimated function L π β θ ( · ) approximating the density of behavior policy, we can naturally propose a density-based constraint in policy learning, and obtain the safe area according to the estimated density. The points with low density values are excluded and the updated policy cannot are exceed the safe region. The actor learning is formulated as:

where π ψ ( ·| s ) is the learning policy, and ˜ S × ˜ A is an estimate of the bounded safe area S × A in Definition 3.1.

There are many approaches for specifying the safe area ˜ S × ˜ A . Kumar et al. [17] bounds the expected MMD distance between the learning policy and the behavior policy within a predetermined threshold, while Wu et al. [22] constrains the lower bound of the estimated behavior density to identify the safe region. In this work, since we can have a direct density estimation of L π β θ ( s, a ) , we can utilize it and specify the safe region as the region with not too small density. Specifically, we set ˜ S × ˜ A = { ( s, a ) ∈ S × A : -log L π β θ ( s, a ) < ϵ } , where ϵ is the threshold 3 .

The critic training is following the traditional Bellman equation as shown in Eq.1. The entire algorithm of CPED is summarized in Algorithm 1. In CPED, the Flow-GAN model plays a crucial role in estimating the behavior density. The density function is obtained from the generator model and is iteratively updated along with the discriminator. We further use the TD3 framework to solve the RL problem. It is noted that during Actor updates, the density of sampled actions should be higher than the threshold to avoid out-of-distribution areas.

4 Theoretical Analysis

In this section, we provide theoretical analysis of our proposed method. In Section 4.1, we prove the convergence of GAN with the hybrid loss, and based on this result, we show the convergence of CPED in Section 4.2.

Algorithm 1 The CPED algorithm

Input: dataset D , target network update rate κ , Lagrange multiplier α , training ratio of the generator and discrimination r , mini-batch size N , hyperparameters λ, c .

Initialize generator G θ , discriminator D ϕ , loss function in GAN L G ( · ) , behavior policy likelihood L π β θ ( · ) , Q networks { Q η 1 , Q η 2 } , actor π ψ , target networks { Q η ′ 1 , Q η ′ 2 } , target actor π ψ ′ , with ψ ′ ← ψ, η ′ i ← η i , i = 1 , 2 .

Sample mini-batch of transitions ( s, a, r, s ′ ) ∼ D

Initial Training Flow-GAN:

end for

for i = 1 to N do

Sample mini-batch of transitions ( s, a, r, s ′ ) ∼ D

θ

argmin

θ

ϕ

L

G

Updating Q-function:

π

θ

Get action for the next state, { ˜ a ∼ π ψ ′ ( ·| s ′ ) + ε, ε ∼ clip ( N (0 , σ ) , -c, c ) } Let y ( s, a ) := min( Q η ′ 1 ( s ′ , ˜ a ) , Q η ′ 2 ( s ′ , ˜ a ))

Updating Actor:

Update ψ according to Eq.7, by using dual gradient descent with Lagrange multiplier α Update Target Networks:

ψ

′

end for

4.1 Convergence of GAN with the Hybrid Loss

In this subsection, we show that training GAN with the hybrid loss function Eq.4 can yield an accurate density estimator. For ease of presentation, assume we observed X j ∼ p ∗ = d µ ∗ d ν , j = 1 , ..., n , where µ ∗ is the underlying true probability measure (the probability measure of behavior policy), and ν is the Lebesgue measure. Motivated by [24, 75, 76], we consider the Integral Probability Metric (IPM) defined as

where µ 1 and µ 2 are two probability measures, and F d is a discriminator class. Under IPM, the GAN framework with the hybrid loss Eq.4 solves the empirical problem (cf., Eq. (1.1) of [75] for the original GAN loss function)

where ˆ µ n = 1 n ∑ n j =1 δ X j is the empirical measure, ˜ µ n is a (regularized) density estimation, Q g is the generator class, and p = d µ d ν . In the following (informal) theorem, we show that µ n is an accurate estimator of the underlying true distribution. A detailed version of Theorem 4.1 and its proof can be found in Appendix A.

Theorem 4.1 (Informal) . Suppose the generator class Q g and the discriminator class F d are induced by some Sobolev spaces. Choose λ as a constant. Under certain conditions,

where µ ∗ is the true probability measure, µ n is as in Eq.9 with hybrid loss, and KL( µ ∗ || µ n ) is the KL divergence.

Theorem 4.1 states a fast convergence rate O P ( n -1 / 2 ) of the IPM and KL divergence can be achieved. Therefore, applying the hybrid loss can provide not only a good generator, but also provide an accurate density estimate, which can be utilized in our CPED framework to control the safe region.

κψ

+(1

-

κ

)

ψ

′

;

η

′

i

κη

i

+(1

-

κ

)

η

′

i

, i

= 1

,

2

(

G

θ

, D

ϕ

)

-

[

λ

E

(

s,a

)

log(

L

β

(

s, a

))]

Remark 4.1 . Given the universal approximation theorem [77, 78], the neural networks can approximate any smooth function. Thus, we consider Sobolev function classes instead of neural networks for the ease of mathematical treatment. If both the generator and discriminator classes are neural networks, one may adopt similar approach as in [75, 79] and then apply Theorem 4.1.

Remark 4.2 . In [75], the convergence rate of E d F d ( µ ∗ , µ n ) is considered, while we provide a sharper characterization of d F d ( µ ∗ , µ n ) , by showing the convergence is in probability with certain convergence rate.

4.2 Convergence of CPED

With an accurate estimate of the underlying density, in this subsection, we give a theoretical analysis of how CPED can find the optimal Q -function value over the product space of S × A . The proof of Theorem 4.2 can be found in Appendix C.

Theorem 4.2. Suppose Q ∗ ( s, a ) is the optimal Q -value on product space S × A . Let ˆ π ( ·| s ) := argmax a Q ∗ ( s, a ) be a learning policy. Let ˆ π ∆ ( ·| s ) be the probability-controlled learning policy using CPED, which is defined on the support of π β . Suppose ˆ π ∆ ( ·| s ) > 0 on A . Then with probability tending to one,

Furthermore, when minimizing the Bellman residual to solve the optimal Q -function and training the policy by iteration, we have the following theorem, whose proof is provided in Appendix D.

Theorem 4.3. Suppose in step k + 1 , the learned policy is defined as π k +1 := argmax a Q π k ( s, a ) , ∀ s ∈ S . Let V π k be the value function related to π k in step k, defined as V π k ( s ) = E a ∼ π k ( ·| s ) [ Q ( s, a )] , and V ∗ be the optimal value function over the product space S × A . Let ˆ π ∆ k be the probability-controlled learning policy using CPED in step k , which is defined on the support of π β . Then with probability tending to one,

Theorem 4.3 provides assurance that CPED can achieve the optimal value function with a linear convergence rate as the iteration number approaches infinity. Despite under offline settings, this rate is consistent with the standard rate in online RL [80]. When combined with Theorems 4.1 and 4.2, we can confidently conclude that CPED's effectiveness is theoretically guaranteed.

5 Experiments

In the experimental section, we analyze the performance of CPED on several standard offline RL tasks [81]. We compare CPED with a wide range of offline RL methods and analyze the performance of CPED. Further implementation details are provided in Appendix E.2.

5.1 Performance Comparison on Standard Benchmarking Datasets for Offline RL

The proposed CPED algorithm is evaluated on the D4RL [81] Gym-MuJoCo and AntMaze tasks. Details of these tasks are shown in Appendix E.1. As a model-free method, CPED is compared with the following model-free baselines: behavioral cloning (BC) method, BCQ [19], DT [82], AWAC [44], BEAR [17], Onestep RL [83],TD3+BC [45], CQL [13], IQL [15], SPOT [22], and PLAS [47]. In the experiments, the performance of all alternatives are measured by the average normalized scores in an episode. The experimental results reported in this paper are either from the authors' original experiments or from our replications.

Tables 1 and 2 show the experimental results of Gym-MuJoCo and AntMaze tasks, respectively. In Gym-MuJoCo tasks, CPED outperforms the competitors in most cases. In particular, our method has achieved remarkable success on the halfcheetah and walker2d tasks, especially under the mediumquality dataset. For the medium-expert task, CPED performs slightly inferior to other tasks, which is due to the fact that the expert task needs tight control so that the learning policy stays close to the behavior policy. In the AntMaze tasks, CPED performs slightly worse than IQL but outperforms the other baselines, and it achieves better returns under 'unmaze' and 'medium-play' tasks.















































































































To further discuss the learning dynamics of the proposed methods, the learning curve of BEAR, SPOT, and CPED in halfcheetah-medium task is shown in Figure 2(a). It is noted that the average return of BEAR increases very fast and remains unchanged after a smooth decrease in subsequent training, indicating that policy exploration is limited. This is consistent with the conservative nature of policy control methods that prevent the learning policy from straying too far from behavior policy. The pattern is similar to SPOT's learning curve, which shows a rapid increase and then enters a plateau. In contrast, the average returns of CPED methods continue to rise after a sharp increase to a

certain extent, indicating that CPED keeps exploring the data and benefits from exploration. This also confirms Theorem 4.3 (see Section C) from the experimental perspective.

5.2 Ablation Study

Time varying hyperparameter α in policy control methods. When improving actor using Eq.7, a Lagrange hyperparameter α is needed for this constrained optimization problem. The larger the α , the more restricted actions taken by the learning policy. Previous methods either use an auto-learning α [17, 14] or a constant α [18, 22]. BRAC [18] find that auto-learning α performs worse than the constant α . In our experiment, we apply a time (epoch) varying α scheme with piece-wise constant functions and find this scheme performs better than the constant setting. The rationale behind this time-varying scheme is that we employ strong constraints for the action taken at the early stage of policy training in case the model fails [8]. The constraint is further relaxed by decreasing α when the policy starts to converge. The piece-wise function of α for the Mujoco tasks and AntMaze tasks are plot in Figure 2(b)-(c), and the performance results of the Mujoco tasks are shown in Figure E.5 in in Appendix E.5. We can see that the Mujoco task significantly benefits from this time varying scheme and delivers better returns in most scenarios.

Likelihood threshold ϵ . The hyperparameter ϵ determines the tightness of our estimated safety boundaries. A larger ϵ leads to more relaxed boundaries and versa vice. A commonly used setting for ϵ is the quantile of behavior policy density. However, the quantile value is difficult to set when the dimension is large. In our experiment, CPED uses the mean likelihood of the training batch as ϵ , which decreases the effect of the estimation error of the behavior policy. See Appendix E.5 for additional discussion and details.

6 Conclusion and Future Work

In this article, we present CPED, a novel approach for policy control in offline RL. CPED leverages the Flow-GAN model to estimate the density of the behavior policy, enabling the identification of safe areas and avoiding the learning of policies that visit OOD points. We validate the effectiveness of CPED both theoretically and experimentally. Theoretical analysis demonstrates that CPED has a high probability of achieving the optimal policy and generating large returns. Experimental results on the Gym-MuJoCo and AntMaze tasks demonstrate that CPED surpasses state-of-the-art competitors, highlighting its superior performance.

While CPED shows great capacity in experiment studies, there are several potential directions for its extension.. Firstly, as a generic method, GAN has diverse variants and it is necessary to try out more powerful GAN structures for density estimation. Secondly, when the feasible region is identified, how to explore the region efficiently remains another important issue, and we leave it to future work. Finally, it is an interesting direction to examine the performance of CPED in more complex scenarios, including dataset generated by multiple behavior policies or multiple agents, etc.

References

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. volume 5, 2018.

Bruno Scherrer, Mohammad Ghavamzadeh, Victor Gabillon, Boris Lesner, and Matthieu Geist. Approximate modified policy iteration and its application to the game of tetris. Journal of Machine Learning Research , 16, 2015. ISSN 15337928.

Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research , 17(1):1334-1373, 2016.

Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on Robot Learning , pages 651-673. PMLR, 2018.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

Yuxi Li. Reinforcement learning applications. arXiv preprint arXiv:1908.06973 , 2019.

MMehdi Afsar, Trafford Crump, and Behrouz Far. Reinforcement learning based recommender systems: A survey. ACM Computing Surveys , 55(7):1-38, 2022.

Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 , 2020.

Fisher Yu, Wenqi Xian, Yingying Chen, Fangchen Liu, Mike Liao, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving video database with scalable annotation tooling. Arxiv , 2018.

Alistair E.W. Johnson, Tom J. Pollard, Lu Shen, Li Wei H. Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G. Mark. Mimic-iii, a freely accessible critical care database. Scientific Data , 3, 2016. ISSN 20524463. doi: 10.1038/sdata.2016.35.

Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. volume PartF168147-1, 2020.

Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-based offline reinforcement learning with diversified q-ensemble. Advances in neural information processing systems , 34:7436-7447, 2021.

Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. volume 2020-December, 2020.

Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua Susskind, Jian Zhang, Ruslan Salakhutdinov, and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforcement learning. arXiv preprint arXiv:2105.08140 , 2021.

Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169 , 2021.

Jiafei Lyu, Xiaoteng Ma, Xiu Li, and Zongqing Lu. Mildly conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2206.04745 , 2022.

Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. volume 32, 2019.

Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361 , 2019.

Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. volume 2019-June, 2019.

Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint arXiv:1907.00456 , 2019.

Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de las Casas, Craig Donner, Leslie Fritz, Cristian Galperti, Andrea Huber, James Keeling, Maria Tsimpoukelli, Jackie Kay, Antoine Merle, Jean Marc Moret, Seb Noury, Federico Pesamosca, David Pfau, Olivier Sauter, Cristian Sommariva, Stefano Coda, Basil Duval, Ambrogio Fasoli, Pushmeet Kohli, Koray Kavukcuoglu, Demis Hassabis, and Martin Riedmiller. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature , 602, 2022. ISSN 14764687. doi: 10.1038/s41586-021-04301-9.

Jialong Wu, Haixu Wu, Zihan Qiu, Jianmin Wang, and Mingsheng Long. Supported policy optimization for offline reinforcement learning. arXiv preprint arXiv:2202.06239 , 2022.

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in Neural Information Processing Systems , 27, 2014.

Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International Conference on Machine Learning , pages 214-223. PMLR, 2017.

Chun-Liang Li, Wei-Cheng Chang, Yu Cheng, Yiming Yang, and Barnabás Póczos. MMD GAN: Towards deeper understanding of moment matching network. In Advances in Neural Information Processing Systems , pages 2203-2213, 2017.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems , 33:6840-6851, 2020.

Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Improving sample efficiency in model-free reinforcement learning from images. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 10674-10681, 2021.

Mazoure B Doan T. Gan q-learning. arXiv preprint arXiv:1805.04874 , 2018.

Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.06193 , 2022.

Shentao Yang, Zhendong Wang, Huangjie Zheng, Yihao Feng, and Mingyuan Zhou. A behavior regularized implicit policy for offline reinforcement learning. arXiv preprint arXiv:2202.09673 , 2022.

Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu. Offline reinforcement learning via high-fidelity generative behavior modeling. arXiv preprint arXiv:2209.14548 , 2022.

Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, and Shixiang Shane Gu. Emaq: Expected-max q-learning operator for simple yet effective offline and online rl. In International Conference on Machine Learning , pages 3682-3691. PMLR, 2021.

Aditya Grover, Manik Dhar, and Stefano Ermon. Flow-gan: Combining maximum likelihood and adversarial learning in generative models. 2018. doi: 10.1609/aaai.v32i1.11829.

Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. 2015.

Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. 2017.

Scott Fujimoto, Herke Van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. volume 4, 2018.

Arsenii Kuznetsov, Pavel Shvechikov, Alexander Grishin, and Dmitry Vetrov. Controlling overestimation bias with truncated mixture of continuous distributional quantile critics. In International Conference on Machine Learning , pages 5556-5566. PMLR, 2020.

Núria Armengol Urpí, Sebastian Curi, and Andreas Krause. Risk-averse offline reinforcement learning. arXiv preprint arXiv:2102.05371 , 2021.

Divyansh Garg, Joey Hejna, Matthieu Geist, and Stefano Ermon. Extreme q-learning: Maxent rl without entropy. arXiv preprint arXiv:2301.02328 , 2023.

Chenjun Xiao, Han Wang, Yangchen Pan, Adam White, and Martha White. The in-sample softmax for offline reinforcement learning. arXiv preprint arXiv:2302.14372 , 2023.

Hongchang Zhang, Yixiu Mao, Boyuan Wang, Shuncheng He, Yi Xu, and Xiangyang Ji. Insample actor critic for offline reinforcement learning. In The Eleventh International Conference on Learning Representations , 2022.

Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, Victor Wai Kin Chan, and Xianyuan Zhan. Offline rl with no ood actions: In-sample learning via implicit value regularization. arXiv preprint arXiv:2303.15810 , 2023.

Philippe Hansen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit q-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304.10573 , 2023.

Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359 , 2020.

Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems , 34:20132-20145, 2021.

Jianxiong Li, Xianyuan Zhan, Haoran Xu, Xiangyu Zhu, Jingjing Liu, and Ya-Qin Zhang. Distance-sensitive offline reinforcement learning. arXiv preprint arXiv:2205.11027 , 2022.

Wenxuan Zhou, Sujay Bajracharya, and David Held. Plas: Latent action space for offline reinforcement learning. In Conference on Robot Learning , pages 1719-1735. PMLR, 2021.

R.J. Hathaway. A constrained formulation of maximum-likelihood estimation for normal mixture distributions. The Annals of Statistics , 13(2):795-800, 1985.

Xuxu Wang and Yong Wang. Nonparametric multivariate density estimation using mixtures. Statistics and Computing , 25(2):349-364, 2015.

Jerome Friedman, Trevor Hastie, and Robert Tibshirani. The Elements of Statistical Learning , volume 1. Springer series in statistics New York, NY, USA:, 2001.

P. Hall. On trigonometric series estimates of densities. The Anuals of Statistics , 9(3):683-685, 1981.

G. Wahba. Data-based optimal smoothing of orthogonal sereis density estimates. The Annals of Statistics , 9(1):146-156, 1981.

David L. Donoho, Iain M. Johnstone, Gerard Kerkyacharian, and Dominique Picard. Density estimation by wavelet thresholding. The Annals of Statistics , 24(2):508-539, 1996. ISSN 00905364. URL http://www.jstor.org/stable/2242660 .

Xi Chen, Ali Ghadirzadeh, Tianhe Yu, Yuan Gao, Jianhao Wang, Wenzhe Li, Bin Liang, Chelsea Finn, and Chongjie Zhang. Latent-variable advantage-weighted policy optimization for offline rl. arXiv preprint arXiv:2203.08949 , 2022.

Shentao Yang, Zhendong Wang, Huangjie Zheng, Yihao Feng, and Mingyuan Zhou. A regularized implicit policy for offline reinforcement learning. arXiv preprint arXiv:2202.09673 , 2022.

Shentao Yang, Yihao Feng, Shujian Zhang, and Mingyuan Zhou. Regularizing a modelbased policy stationary distribution to stabilize offline reinforcement learning. In International Conference on Machine Learning , pages 24980-25006. PMLR, 2022.

Avi Singh, Huihan Liu, Gaoyue Zhou, Albert Yu, Nicholas Rhinehart, and Sergey Levine. Parrot: Data-driven behavioral priors for reinforcement learning. arXiv preprint arXiv:2011.10024 , 2020.

Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:2205.09991 , 2022.

Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. Advances in neural information processing systems , 30, 2017.

Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434 , 2015.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 , 2013.

Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference on artificial intelligence , volume 30, 2016.

Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In International conference on machine learning , pages 1995-2003. PMLR, 2016.

R.S. Sutton and A.G. Barto. Reinforcement learning: An introduction. IEEE Transactions on Neural Networks , 9, 1998. ISSN 1045-9227. doi: 10.1109/tnn.1998.712192.

David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International conference on machine learning , pages 387-395. PMLR, 2014.

Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 , 2015.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Andrew Y Ng, Stuart Russell, et al. Algorithms for inverse reinforcement learning. In Icml , volume 1, page 2, 2000.

Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In Aaai , volume 8, pages 1433-1438. Chicago, IL, USA, 2008.

Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. volume 1, 2016.

Chelsea Finn, Paul Christiano, Pieter Abbeel, and Sergey Levine. A connection between generative adversarial networks, inverse reinforcement learning, and energy-based models. arXiv preprint arXiv:1611.03852 , 2016.

Jonathan Ho, Jayesh K. Gupta, and Stefano Ermon. Model-free imitation learning with policy optimization. volume 6, 2016.

Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems , 29, 2016.

Halbert White. Maximum likelihood estimation of misspecified models. Econometrica: Journal of the econometric society , pages 1-25, 1982.

Tengyuan Liang. How well generative adversarial networks learn distributions. Journal of Machine Learning Research, to appear , 2021.

Shuang Liu, Olivier Bousquet, and Kamalika Chaudhuri. Approximation and convergence properties of generative adversarial learning. arXiv preprint arXiv:1705.08991 , 2017.

Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural networks , 2(5):359-366, 1989.

Balázs Csanád Csáji et al. Approximation with artificial neural networks. Faculty of Sciences, Etvs Lornd University, Hungary , 24(48):7, 2001.

Liang Ding, Lu Zou, Wenjia Wang, Shahin Shahrampour, and Rui Tuo. High-dimensional non-parametric density estimation in mixed smooth sobolev spaces. arXiv preprint arXiv:2006.03696 , 2020.

Alekh Agarwal, Nan Jiang, and Sham M. Kakade. Reinforcement learning: Theory and algorithms. 2019.

Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 , 2020.

Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems , 34:15084-15097, 2021.

David Brandfonbrener, Will Whitney, Rajesh Ranganath, and Joan Bruna. Offline rl without off-policy evaluation. Advances in neural information processing systems , 34:4933-4946, 2021.

David L Donoho, Iain M Johnstone, Gérard Kerkyacharian, and Dominique Picard. Wavelet shrinkage: asymptopia? Journal of the Royal Statistical Society: Series B (Methodological) , 57 (2):301-337, 1995.

Yves Meyer. Wavelets and Operators: Volume 1 . Number 37. Cambridge university press, 1992.

Sara van de Geer. Empirical Processes in M-estimation . Cambridge University Press, 2000.

Robert A Adams and John JF Fournier. Sobolev Spaces , volume 140. Academic press, 2003.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel algorithm for offline model-free reinforcement learning by constraining the actor to lie within a safe area defined by an explicit behavioral density. This behavioral density is obtained by thresholding a FlowGAN model. The resulting algorithm is competitive across various D4RL datasets.\n\nSTRENGTHS:\n- Clearly presented algorithm and well-motivated method\n- Strong empirical evaluation and competitive results on D4RL MuJoCo\n- Solid theoretical justification of the convergence of the FlowGAN and RL training\n\nWEAKNESSES:\n- Model-based offline methods typically use a state-action-based uncertainty penalty which could also be repurposed for the definition of the safe area. E.g. thresholding the model uncertainty as in MOReL [1]. This would be a useful baseline to show the benefit of the Flow-GAN method\n- The hand-crafted per-environment piecewise linear scheme for alpha limits the generality of the method\n- Algorithm 1 seems to be incorrect, the preceding discussion has the Flow-GAN optimization happening before RL training, whereas Algorithm 1 has the Flow-GAN optimization interleaved with RL training.\n- Line 150: definition of the ‘offline MDP’ is not a real MDP. For example, the state and action spaces should both be sets and the definition is a tuple. It is unclear what this definition adds beyond describing the marginal state distribution and behavioral policy.\n- In Theorem 4.2, the optimal Q-function on the “restricted product space” needs to be defined and explained.\n\nMinor:\n- Line 145: ‘triples’ is likely a mistake, the transition is a 4-tuple\n- Tables 1 and 2 missing EDAC [2], the current SOTA model-free offline RL method\n\n[1] MOReL: Model-Based Offline Reinforcement Learning. Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, Thorsten Joachims.\n\n[2] Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble. Gaon An, Seungyong Moon, Jang-Hyun Kim, Hyun Oh Song.\n\nQUESTIONS:\n- Could a dynamics model-based uncertainty (e.g. MOReL) provide a simple alternative to the Flow-GAN threshold?\n- Could the alpha tuning scheme be replaced by something not hand-crafted to increase the applicability of the method?\n\nLIMITATIONS:\nNot discussed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel approach named Constrained Policy optimization with Explicit Behavior density (CPED) to mitigate the existing policy control methods, e.g., overly conservative or failing to identify OOD areas accurately. CPED use a flow-GAN model to explicitly estimate behavior policy density. The strength of CPED lies in its ability to accurately identify safe regions for explorations, which in turn, leads to less conservative learning policies.\n\nSTRENGTHS:\n1. Well-motivated example in Fig. 1 and overall good writing. \n2. Sufficient theoretical evidence to show that CPED has a fast convergence rate and is able to ﬁnd the optimal Q-function value.\n\nWEAKNESSES:\n1. The contribution is limited as it seems to directly borrow the idea of flow-gan into offline RL domains.\n2. Besides the motivation example in Fig. 1, there is no faithful evidence to show that \"accurately identify the feasible region, which includes both observed and unobserved but safe points\" both in theoretical analysis and experiments. \n3. Lack of survey and comparison with the most similar work [1], which also learns an expressive generative behavior model but by the diffusion technique.\n4. Though there is an obvious edge of CPED against SPOT in experiments, more explanations are lacking to suppose it.\n\nQUESTIONS:\n1. Could the authors provide more comparisons between CPED and SPOT and explain why CPED is more feasible? \n2. Regarding policy learning (Eq. 7), why not transfer it to a Lagrange function like SPOT? To the best of my knowledge, explicitly excluding bad transitions has a little edge against other weighting or penalty methods in terms of generalization, which is a key contribution mentioned by the authors.\n\nLIMITATIONS:\nLack of related work: the in-sample learning paradigm [2-8] is now one of the trends in offline reinforcement learning, those new methods are worth being well mentioned in related work. \n\n---\nI decided to change my score from 4 to 5 after the first round discussion with the authors. \n\n---\n\n[1] Chen H, Lu C, Ying C, et al. Offline reinforcement learning via high-fidelity generative behavior modeling[J]. arXiv preprint arXiv:2209.14548, 2022.\n\n[2] Kostrikov I, Nair A, Levine S. Offline reinforcement learning with implicit q-learning[J]. arXiv preprint arXiv:2110.06169, 2021.\n\n[3] Xu H, Jiang L, Jianxiong L, et al. A policy-guided imitation approach for offline reinforcement learning[J]. Advances in Neural Information Processing Systems, 2022, 35: 4085-4098.\n\n[4] Garg D, Hejna J, Geist M, et al. Extreme q-learning: Maxent RL without entropy[J]. arXiv preprint arXiv:2301.02328, 2023.\n\n[5] Xiao C, Wang H, Pan Y, et al. The in-sample softmax for offline reinforcement learning[J]. arXiv preprint arXiv:2302.14372, 2023.\n\n[6] Zhang H, Mao Y, Wang B, et al. In-sample Actor Critic for Offline Reinforcement Learning[C]//The Eleventh International Conference on Learning Representations. 2022.\n\n[7] Xu H, Jiang L, Li J, et al. Offline rl with no ood actions: In-sample learning via implicit value regularization[J]. arXiv preprint arXiv:2303.15810, 2023.\n\n[8] Hansen-Estruch P, Kostrikov I, Janner M, et al. Idql: Implicit q-learning as an actor-critic method with diffusion policies[J]. arXiv preprint arXiv:2304.10573, 2023.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new offline RL algorithm called Constrained Policy optimization with Explicit Behavior density (CPED). The main idea is utilizing a flow-GAN model to estimate the probability density explicitly, which limits the exploration in the safe region. To demonstrate the effectiveness, the author provides some theoretical proofs for the convergence of CPED, and evaluates CPED in various standard offline RL tasks.\n\nSTRENGTHS:\nThis paper is overall well-written, and it performs a comprehensive survey in offline RL and related fields such as inverse RL and generative model. The proposed method is well-motivated, which introduces flow-GAN to offline RL appropriately. This paper provides detailed theoretical analyses of the convergence to confirm the effectiveness of CPED.\n\nWEAKNESSES:\n1.\tThe novelty of this paper is somewhat limited as it primarily replaces the VAE model used in SPOT [1] with flow-GAN.\n2.\tWith the exception of the ho-med dataset, CPED does not demonstrate clear improvements and, performs even worse than some previous methods in the experimental datasets.\n3.\tThe implementation of CPED is not clearly stated, and this aspect will require further discussion in the questions.\n4.\tThe usage of certain notations is confusing, and there are some grammar errors.\n\nQUESTIONS:\n1.\tIn Equation 7, the Lagrangian multiplier $\\alpha$ is mentioned in the algorithm but cannot be found. Could you clarify the actual objective function? Additionally, the hyperparameter $\\epsilon$ is not mentioned in your algorithm. How does $\\epsilon$ affect the performance of your algorithm?\n2.\tIn flow-GAN, specific neural networks are utilized for the generator and discriminator. Does this limitation affect their representational ability? Furthermore, does it result in a significant gap between the estimated policy and the behavioral policy?\n3.\tPrevious work often considers online fine-tuning tasks after offline RL. Can CPED be applied to such tasks? If so, how does CPED compare to other offline RL methods in these scenarios?\n4.\tFigure 2 shows the learning curve comparison between BEAR and CPED. However, BEAR is considered somewhat outdated. As SPOT is a more recent offline RL method and seems more relevant to CPED, I'm particularly interested in the comparison between SPOT and CPED.\n5.\tIn Section 5.2, you mention an additional trick using time-varying $\\alpha$. It appears that this trick could be applicable to SPOT as well. Since the empirical improvement is limited, I'm curious if the improvement primarily results from this trick rather than flow-GAN. Could you provide additional results of SPOT with the time-varying $\\alpha\"?\n6.\tThe choice of $\\epsilon$ is not clearly explained. What is the specific quantile of the behavior policy density that you refer to? Does it correspond to the median or other quantiles? It would be helpful if you could provide examples of this \"quantile\" in several datasets.\n\nLIMITATIONS:\nPlease refer to the weakness parts.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes to use flow-GAN model to explicitly model the distribution of the behaviour policy and use that to alleviate the OOD actions problem in offline RL.\nA theoretical analysis of convergence is presented. Empirical evaluation on d4rl gym locomotion and antmaze show that the proposed CPED algorithm is very competitive.\n\nSTRENGTHS:\n* The method is novel and very intuitive. Instead of using all different tricks to avoid OOD, explicitly modelling the behaviour policy is much more straightforward.\n* The writing is clear\n* Empirical evaluation shows the performance of CPED is comparable to SOTAs\n\nWEAKNESSES:\n* Compared to prior methods like IQL, DT, or even CQL. The proposed method has an additional generative modelling module, which makes the overall architecture more complex.\n* Again, the empirical results are good but it's technically not surpassing SOTA. For gym-locomotion, there are algorithms like ATAC [1] and SAC-N [2] that have similar level of performance. As for antmaze, the performance of CPED is inferior than IQL, which is not the SOTA for offline RL in general. For model-based methods, there're methods like trajectory transformer [3] that perform better than IQL.\n\n[1] https://proceedings.mlr.press/v162/cheng22b/cheng22b.pdf\n\n[2] https://arxiv.org/pdf/2110.01548.pdf\n\n[3] https://arxiv.org/abs/2106.02039\n\nMinor issues:\n\n> CPED can accurately identify the safe region and enable exploration within the region\n\nI would avoid using the term \"exploration\" here because it can hint to readers that the proposed method can be helpful for online learning but I believe this is not what the authors mean here.\n\nQUESTIONS:\n* What's the computational overhead of training this Flow-GAN?\n* Why do you pick Flow-GAN? Why not use simpler ones like VAE, or even just discretization?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}