Title: VOCE: Variational Optimization with Conservative Estimation for Offline Safe Reinforcement Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Offline safe reinforcement learning

Pessimistic conservative estimation

Variational optimization

Reinforcement Learning
Abstract: Offline safe reinforcement learning (RL) algorithms promise to learn policies that satisfy safety constraints directly in offline datasets without interacting with the environment. This arrangement is particularly important in scenarios with high sampling costs and potential dangers, such as autonomous driving and robotics. However, the influence of safety constraints and out-of-distribution (OOD) actions have made it challenging for previous methods to achieve high reward returns while ensuring safety. In this work, we propose a Variational Optimization with Conservative Eestimation algorithm (VOCE) to solve the problem of optimizing safety policies in the offline dataset. Concretely, we reframe the problem of offline safe RL using probabilistic inference, which introduces variational distributions to make the optimization of policies more flexible. Subsequently, we utilize pessimistic estimation methods to estimate the Q-value of cost and reward, which mitigates the extrapolation errors induced by OOD actions. Finally, extensive experiments demonstrate that the VOCE algorithm achieves competitive performance across multiple experimental tasks, particularly outperforming state-of-the-art algorithms in terms of safety.

VOCE: Variational Optimization with Conservative Estimation for Offline Safe Reinforcement Learning

Abstract

Offline safe reinforcement learning (RL) algorithms promise to learn policies that satisfy safety constraints directly in offline datasets without interacting with the environment. This arrangement is particularly important in scenarios with high sampling costs and potential dangers, such as autonomous driving and robotics. However, the influence of safety constraints and out-of-distribution (OOD) actions have made it challenging for previous methods to achieve high reward returns while ensuring safety. In this work, we propose a V ariational O ptimization with C onservative E stimation algorithm (VOCE) to solve the problem of optimizing safety policies in the offline dataset. Concretely, we reframe the problem of offline safe RL using probabilistic inference, which introduces variational distributions to make the optimization of policies more flexible. Subsequently, we utilize pessimistic estimation methods to estimate the Q-value of cost and reward, which mitigates the extrapolation errors induced by OOD actions. Finally, extensive experiments demonstrate that the VOCE algorithm achieves competitive performance across multiple experimental tasks, particularly outperforming state-of-the-art algorithms in terms of safety. Our code is available at github.VOCE.

1 Introduction

Reinforcement learning (RL) algorithms have made remarkable achievements in various fields such as robot control [1-3] and strategy games [4-6]. However, limited by the online and trial-and-error nature, standard RL is challenging to apply to dangerous and expensive scenarios [7-9]. Offline safe RL is a promising and potential approach to tackle the above problem, which learns policy satisfying safety constraints from pre-collected offline datasets without interacting with the environment [10, 11].

Since offline safe RL methods aim to learn policies that satisfy safety constraints from offline datasets, which requires the agent to not only comply with safety constraints but also consider the influence of out-of-distribution (OOD) actions [12-15]. This makes it difficult for the algorithm to learn policies that achieve high reward returns while satisfying safety constraints [16]. There are two main

approaches to deal with the aforementioned challenges: one is based on linear programming [17, 18], and the other is based on exploration and evaluation [19-21]. Linear programming provides a way to research offline safe RL problems, but it heavily depends on the F-divergence or KL-divergence to constrain the distance between the optimal distribution and the sample distribution [17]. Therefore, it is difficult for linear programming to eliminate the extrapolation errors caused by OOD actions [10, 11]. The exploration evaluation approaches are to introduce conservative Q-value estimation under the actor-critic framework. Although exploration evaluation approaches avoid the overestimation issue through lower bound Q-values, they prematurely terminate trajectories that exceed the constraints during policy updates [16]. The exploration evaluation method with strict constraints on the sample space makes it challenging to search for reward-maximizing policies. In conclusion, existing methods face challenges in learning policies from offline datasets that maximize reward while satisfying safety constraints, particularly when considering the impact of extrapolation errors.

In this work, to solve the constrained policy optimization problem and eliminate the extrapolation errors caused by OOD actions, we propose a variational optimization with conservative estimates for the offline safe RL algorithm. Concretely, we reframe the objective of offline safe RL using probabilistic inference, enhancing the flexibility of policy optimization by replacing parameterized policies with variational distributions. Furthermore, to avoid extrapolation errors caused by OOD actions, we derive upper and lower bounds for Q-values and estimate the Q-values of costs and rewards based on these bounds. Finally, we adopt a supervised learning approach to train a parameterized policy network according to the variational distribution of policies. The main contributions of this work are listed as follows:

We utilize probabilistic inference to address the problem of offline safe RL, which introduces non-parametric variational distributions to replace parameterized policies, providing increased flexibility for optimizing safe policies in offline datasets.

We derived upper and lower bounds for Q-value estimation using the pessimistic estimation approach. Subsequently, we utilized these bounds to estimate the Q-value of costs and rewards, respectively, to avoid extrapolation errors caused by OOD actions.

We carry out extensive comparative experiments, and the results indicate that the VOCE algorithm outperforms state-of-the-art algorithms, especially in terms of safety.

2 Preliminaries

Constrained Markov decision processes (CMDP) provide a theoretical framework to solve safe RL problems [22], where the agent is cost-bounded by safety constraints. A CMDP is defined as a tuple ( S , A , C, P, r, ρ 0 , γ ) , where S ∈ R n is the state space, A ∈ R m is the action space, P : S×A×S → [0 , 1] is the transition kernel, which specifies the transition probability p ( s t +1 | s t , a t ) from state s t to state s t +1 under the action a t , r : S × A → R represents the reward function, C is the set of costs { c i : S × A → R + , i = 1 , 2 , · · · , m } for violating m constraints, γ ∈ (0 , 1] is the discount factor, and ρ 0 : S → [0 , 1] is the distribution of initial states. A policy π maps a probability distribution from state s t to action a t . We utilize π θ to denote the parameterization of the policy with parameter θ . In safe RL, the goal is to maximize the cumulative reward while satisfying safe constraints:

where the τ = { s 0 , a 0 , · · · } ∼ π denotes the trajectory. ¯ c i is the i -th safe constraint limit.

In the offline RL setting, we learn a policy according to the offline dataset D collected by one or more data-collection policies, and without online interaction with the environment ( D = { ( s t , a t , r t , c t ) i } n i =1 ). Although this way of not interacting with the environment brings a lot of advantages, the offline dataset can not cover all action-state transitions, and the policy evaluation step actually utilizes the Bellman equation of a single sample [23]. This makes the estimated Q-value vulnerable to OOD actions, which in turn severely affects the performance of the algorithm [24, 25]. As shown in Fig. 1, under the offline setting, off-policy safe RL methods neither learn a high-reward policy nor guarantee safety. Additionally, although offline RL methods obtain high rewards under safe expert data, it is difficult to guarantee safety because it directly ignores the cost. We can conclude that both offline RL and safe RL face challenges in learning policies that satisfy safety constraints

from offline datasets. Therefore, it is important to design an algorithm that learns high-reward and satisfies safety constraints, which is the focus of this paper.

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

Step(*1e4)

20

10

0

10

20

30

Reward of episode

Offline RL

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

Step(*1e4)

20

40

60

80

100

Cost of episode

Offline RL

BCQ

CRR

BEAR

CQL

Cost limit

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

Step(*1e4)

30

20

10

0

Reward of episode

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

Step(*1e4)

0

10

20

30

Cost of episode

SAC-Lag

DDPG-Lag

CVPO

Cost limit

3 Method

In this section, we present the details of Variationally Optimized with Conservatively Estimated for offline-safe RL algorithm (VOCE), which is the main contribution of this paper. We first reformulate the objective of offline-safe RL with probabilistic inference. Based on this, we derived upper and lower bounds for Q-value estimation, and adopted a pessimistic and conservative estimation approach to estimate the Q-value of costs and rewards respectively to eliminate the extrapolation error of OOD actions. Finally, we employ supervised learning to transform the variational distribution into a parameterized policy.

3.1 Offline-safe RL as Inference

From the probabilistic inference perspective, the offline safe RL can be viewed as the problem of observing safe actions in states with high rewards. As the probabilistic inference model and notation in [26, 27], we introduce an optimal variable O to represent the event of maximizing rewards. Assuming the likelihood of a given trajectory being optimal is proportional to the cumulative discounted reward, the infinite discounted reward formula is expressed as P ( O = 1 | τ ) ∝ ∑ t γ t r ( s t , a t ) . Since P ( O = 1 | τ ) ≥ 0 , we further confirm that the likelihood of a given trajectory being optimal is proportional to the exponential of the accumulative discounted reward, then we rewrite the infinite discounted reward formula as P ( O = 1 | τ ) ∝ exp( ∑ t γ t r t /α ) . The α is a temperature parameter, and r t is short for r ( s t , a t ) . Let P π ( τ ) be the probability of the trajectory under the policy π , then the log-likelihood of optimality under policy π can be written as log P π ( O = 1) = log ∑ τ P ( O = 1 | τ ) P π ( τ ) . Furthermore, according to importance sampling and Jensen's inequality, we obtain a lower bound of the log-likelihood of optimality under policy π .

where the Q ( τ ) is the auxiliary trajectory distribution. Since offline safe RL not only needs to consider maximizing cumulative rewards but also guaranteeing safety, we restrict Q ( τ ) to the constrained feasible distribution space. According to the constraint threshold ¯ c i , we define the feasible distribution space as:

where each q ( a t | s t ) ∈ Ω ¯ c i indicates that the action distribution of the sate-condition satisfies the safety constraint. By factorizing the trajectory distribution [27], we express the trajectory distribution as:

where ρ ( s 0 ) is the distribution of the initial state s 0 . Substituting Eq. (4) and (5) into Eq. (2) to eliminate the transitions p ( s t +1 | s t , a t ) , and combining the feasible distribution space of Eq. (3), we obtain the object of offline safe RL problem as shown in Proposition 3.1.

Proposition 3.1. The objective of the offline safe RL problem can be defined through probabilistic inference as follows:

Proposition 3.1 provides a new objective for the problem of offline safe RL. The aforementioned probabilistic inference method has many optimizations over the previous dual methods in solving the problem of offline safe RL. This approach provides flexibility for optimizing the optimal policy by introducing a variational distribution q ( a t | s t ) to connect the strong correlation between dual variable optimization and policy improvement. Furthermore, the approach decouples the optimal policy optimization problem into independent processes of optimal variational distribution optimization and parameterized policy update. We will introduce these two update processes in detail later.

3.2 Variational Optimization with Conservative Estimates for Offline-safe RL

The previous standard RL and safe RL also employed the above-mentioned idea of probabilistic inference to solve the policy optimization problem [26, 28]. However, due to the influence of OOD actions in the offline setting, previous RL algorithms based on probabilistic inference are difficult to address the tasks of offline-safe RL. Therefore, we adopt a pessimistic conservative estimation method to improve the offline-safe RL algorithm based on probabilistic inference and obtain a V ariational O ptimization with C onservative E stimation algorithm (VOCE). We divide the VOCE into two steps: conservatively estimated variational optimization and parameterized policy update.

3.2.1 Conservatively Estimated Variational Optimization

The goal of the conservatively estimated variational optimization is to optimize the optimal variational distribution q ( a t | s t ) with a high reward while satisfying the safety constraints. In this step, we perform the maximization of the variational distribution q ( a t | s t ) by fixing the policy π θ ( a t | s t ) . According to proposition 3.1, we write the objective of solving the optimal variational distribution as:

The optimization objective of the above variational distribution can be regarded as a KL-regularized constrained optimization problem. However, since the expected reward term E q ( a t | s t ) Q r ( s t , a t ) may be on different scales depending on the environment, it is difficult to set an appropriate penalty coefficient α for different environments. Therefore, we introduce hard constraints to replace the regularization term of the KL-divergence and rewrite the optimization objective of Eq. (7) as:

where ϵ is the KL-divergence threshold between the variational distribution q ( a t | s t ) and the parameterized policy π θ ( a t | s t ) . To solve the above-constrained problem (8), we replace the parametric q ( a t | s t ) with the non-parametric to avoid performance degradation caused by approximation errors [29]. Furthermore, we apply the Lagrange multiplier method to transform the above-constrained problem into an unconstrained problem. In the non-parametric form q ( a t | s t ) , since the objective function is linear and the constraints are convex, the constrained optimization problem shown in Eq. (8) is a convex optimization problem. Furthermore, through mild assumptions, we can obtain the strong dual form of Eq. (9).

Assumption 3.2. (Slater's condition). There exists a variational distribution that satisfies the safety constraints q ( a t | s t ) ∈ Ω ¯ c i within the current policy trust region D KL ( q ( a t | s t ) || π θ ( a t | s t )) < ϵ .

Lemma 3.3. Under Assumption 3.2, there exists a strong duality between the primal problem in Eq. (8) and the unconstrained problem in Eq. (9).

where λ, η, δ are the Lagrange multipliers of the above multiple constraints. Based on the strong duality of the unconstrained problem in Eq. (9), we further derive a closed-form solution q ∗ ( a t | s t ) for its internal maximization, as shown in Proposition 3.4. Proofs and discussions are in Appendix B.1.

Proposition 3.4. The closed-form solution of the optimal variational distribution that satisfies the safety constraint in Eq. (9) is given as:

By substituting the closed-form solution (10) into Eq. (9) and eliminating similar terms, we obtain the convex minimization problem shown in Proposition 3.5. Subsequently, we solve for the dual variables η and λ using Eq. (11). Proofs and discussions are in Appendix B.2.

Proposition 3.5. The λ, η are dual variables and are solved via the convex optimization problem of the following formula.

The Proposition 3.4 provides a proposal to solve a non-parametric variational distribution q ( a t | s t ) for the given the Q-value Q r ( s t , a t ) and Q c i ( s t , a t ) . In addition, we provide the optimality and uniqueness of the above closed-form q ∗ ( a t | s t ) solution under the premise of strong convexity in the Appendix B.2. Note that it can be seen from Eq. (10) that providing accurate Q r ( s t , a t ) and Q c i ( s t , a t ) are the premise and guarantee for accurately computing the non-parametric variational distribution q ( a t | s t ) . We can find similar conclusions in the online safe RL algorithm CVPO [26], as described in Proposition 3.4 and 3.5. In the online setting, the empirical Bellman equation is directly applied to iteratively update the Q-value. However, in the offline setting, this approach would lead to clear extrapolation errors in Q c i ( s t , a t ) and Q r ( s t , a t ) due to the OOD actions. Furthermore, it is difficult to accurately compute the variational distribution q ( a t | s t ) according to Eq. (10).

To eliminate the extrapolation error caused by the OOD actions during the evaluation of the Q-value, we utilize the pessimistic conservative estimation approach to estimate Q r and Q c i , respectively. Specifically, to eliminate the impact of extrapolation error on Q r , we need to avoid overestimation of Q r [24]. Therefore, similar to CQL [30] to learn a conservative lower bound of the Q-function by additionally minimizing the Q-values alongside a standard Bellman error objective, we choose the penalty that minimizes the expected Q-value under special action-state transitions for unknown actions produced by π M ( a t | s t ) . Then, we define the Q-value of the iterative reward function as:

where the ˆ B π ˆ Q r ( s t , a t ) = r ( s t , a t ) + γ ˆ Q r ( s t +1 , a t +1 ) . In Proposition 3.6, we show that ˆ Q r ( s t , a t ) converges to a lower bound on ˆ Q r ( s t , a t ) . However, we can tighten this bound if we are interested in V r ( s t ) . We improve the bounds by introducing an additional maximization term under the sample distribution ˆ π β . Then the iterative update Eq. (12) of the reward Q-value can be rewritten as:

where κ is a tradeoff factor. Note that the Eq. (12) and (13) utilize the empirical Bellman operator ˆ B π instead of the actual Bellman Operator B π . Following the related work [23, 31], we employ the

concentration properties to compute the error. For any ∀ s t , a t ∈ D , with the probability ≥ 1 -δ , the sampling error can be written as:

where C r,p,δ is a constant depending on the concentration properties of r ( s t , a t ) , p ( s t +1 | s t , a t ) and the δ, δ ∈ (0 , 1) . The 1 √ |D ( s t ,a t ) | represents a vector of size |S||A| containing the square root inverse count of each state action transition. Considering the sampling error as shown in Eq. (14), we can derive the condition for ˆ Q r converging to the lower-bound Q r at all ( s t , a t ) through Eq. (12). The Proposition 3.6 provides the condition for ˆ Q r to converge to the lower-bound of the Q r . Proofs and discussions are in Appendix B.3.

Proposition 3.6. For any π M ( a t | s t ) with supp π M ( a t | s t ) ⊂ ˆ π β , ∀ s t , a t ∈ D , the Q-value function Q r via iterating Eq. (12) satisfies:

Thus, if κ ≥ C r,p,δ R max (1 -γ ) √ |D ( s t ,a t ) | [ π M ( a t | s t ) ˆ π β ( a t | s t ) ] -1 , the iterative update Eq. (12) guarantees ˆ Q r ≤ Q r .

Next, when π M ( a t | s t ) = π θ ( a t | s t ) , we obtain a not lower-bound for the Q-values estimates pointwise. based on Eq. (13). We abuse the notation 1 √ |D| to represent a vector of the inverse square root of state counts. Proofs and discussions are in Appendix B.4.

Proposition 3.7. When π M ( a t | s t ) = π θ ( a t | s t ) , according to Eq. (13), we obtain a lower bound for the true value of V r = E a t ∼ π θ ( a t | s t ) [ Q r ( s t , a t )] that satisfies the following inequality:

On the other hand, taking into account the need to satisfy the safety constraint E ρ ( s 0 ) [ ∑ a q ( a t | s t ) Q c i ( s t , a t ) ] ≤ ¯ c i , the estimation of the cost Q-value must fulfill ˆ Q c i ≥ Q c i . Based on the above analysis we choose the penalty that maximizes the expected Q-value under special action-state transitions for unknown actions produced by π R ( a | s ) . Therefore, we define the Q-value of the iterative reward function as:

where χ is the trade-off factor. The π R denotes the marginal distribution corresponding to the unknown action. The Proposition 3.8 provides an upper bound on the convergence of the fixed point of ˆ Q c i , and clarifies the conditions for ˆ Q c i to converge to the upper bound of Q c i . Proofs and discussions are in Appendix B.5.

Proposition 3.8. For any π R ( a t | s t ) with supp π R ( a t | s t ) ⊂ ˆ π β , ∀ s t , a t ∈ D , the Q-value function Q c i via iterating Eq. (17) satisfies:

Thus, if χ ≥ C r,p,δ C max (1 -γ ) √ |D ( s t ,a t ) | [ π R ( a t | s t ) ˆ π β ( a t | s t ) ] -1 , the iterative update Eq. (17) can guarantee ˆ Q c i ≥ Q c i .

3.2.2 Parametered Policy Update

After solving the optimal variational distribution of each state via Eq. (10), we need to obtain the policy parameters θ . According to the solution target of Eq. (6), the optimization target can be obtained by eliminating the quantities irrelevant to θ .

where α ≥ 0 is the temperature parameters. In addition, the q ( a t | s t ) is independent of π θ ( a t | s t ) . Therefore, the optimization objective of the above Eq. (19) can be rewriten as:

4 Experimental Evaluation

In this section, we compare VOCE to previous offline safe RL methods in a range of domains and dataset compositions, including different action spaces and observation dimensions. To the best of my knowledge, there is currently no standardized dataset available in the field of offline safe RL. To facilitate further research and reproducibility of this work, we have collected a series of datasets using a trained online policy. The parameter settings of the dataset are in Appendix C.1.

4.1 Task and Baseline

Task. We choose Point-Goal , Car-Goal , Point-Button and Car-Button four tasks widely adopted in the field of safe RL [26, 29, 32-34], as the experimental tasks for this work. A detailed description of each task is in Appendix C.2.

Point-Goal

Car-Goal

Point-Button

Car-Button

Cost of episode

Cost of episode

Cost of episode

Cost of episode

Reward of episode

Reward of episode

Reward of episode

Reward of episode

20

30

8'0

0.6

0.4

35

30

0.6

0.8

0.4

35

30

40

0.6

0.4

8'0

30

35

0.6

0.8

0.4

10

25

25

25

0

20

20

10

15

15

15

10

20

10

10

20

40

60

08

100

50

D

300

100

Baselines. BCQ-Lag is an offline safe RL algorithm that combines the Lagrange multiplier method with the BCQ [24] algorithm and employs adaptive penalty coefficients to implement offline constraints task. The C-CRR is an offline safe RL algorithm improved by the CRR [35] algorithm. It implements cost constraints by introducing cost evaluation Q-value function and Lagrange multipliers. Coptidice [10] is a policy optimization method based on the optimal stationary distribution space. The aforementioned three approaches are currently state-of-the-art algorithms in offline safe RL.

4.2 Performance Comparison Experiment

To evaluate the performance of VOCE on diverse task and behavior samples, we collected sample data from three distinct behaviors within four different tasks. We introduce parameter φ to represent the proportion of trajectories in the sample data that satisfy the constraints. Then we employ φ to characterize samples to different behaviors. Fig. 2 illustrates the marginal distribution of rewards and costs for sample trajectories in the dataset at different values of φ . The results from Fig. 2 reveal that as the value φ increases, the mean cost decreases, and the mean reward increases. Fig. 3 displays the reward and cost curves of VOCE and the state-of-the-art offline safe RL methods under different φ values for four tasks. The Fig. 3 results demonstrate that the VOCE achieves higher rewards across all tasks compared to the other three methods, while also satisfying or approaching the safety constraints threshold. Especially in the Goal task, VOCE consistently meets the safety constraints across different φ values, while achieving higher reward returns. In the Button task, when the parameter φ is small, VOCE struggles to ensure safety; however, the cost curve of VOCE remains lower than the other three methods. Based on the aforementioned results analysis, it can be concluded that the VOCE exhibits competitive performance across various combinations of samples from multiple tasks, particularly excelling in terms of safety compared to the state-of-the-art algorithms currently available.

4.3 Ablation Experiment

The parameter φ of the dataset. Fig. 4 displays the boxplots of rewards and costs for the VOCE under different parameter values of φ in the Point-Goal and Point-Button tasks. The results from

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

20

0

20

Reward of episode

Point-Goal (

=0.8)

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

0

20

Car-Goal (

=0.8)

0.0

0.5

1.0

1.5

2.0

2.5

3.0

20

0

20

Point-Button (

=0.8)

0.0

0.5

1.0

1.5

2.0

2.5

3.0

10

0

10

20

30

=0.8)

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

0

10

20

30

Cost of episode

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

0

10

20

30

0.0

0.5

1.0

1.5

2.0

2.5

3.0

10

20

30

40

0.0

0.5

1.0

1.5

2.0

2.5

3.0

20

40

60

80

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

0

10

20

30

Reward of episode

Point-Goal(

=0.6)

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

0

10

20

30

Car-Goal(

=0.6)

0.0

0.5

1.0

1.5

2.0

2.5

3.0

20

0

20

Point-Button(

=0.6)

0.0

0.5

1.0

1.5

2.0

2.5

3.0

10

0

10

20

=0.6)

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

10

20

30

Cost of episode

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

10

20

30

40

0.0

0.5

1.0

1.5

2.0

2.5

3.0

20

40

0.0

0.5

1.0

1.5

2.0

2.5

3.0

25

50

75

100

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

0

10

20

Reward of episode

Point-Goal(

=0.4)

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

20

0

20

Car-Goal(

=0.4)

0.0

0.5

1.0

1.5

2.0

2.5

3.0

20

0

20

Point-Button(

=0.4)

0.0

0.5

1.0

1.5

2.0

2.5

3.0

0

10

20

=0.4)

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

Step(*1e4)

0

20

40

60

80

100

Cost of episode

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

Step(*1e4)

20

40

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Step(*1e4)

0

20

40

60

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Step(*1e4)

25

50

75

100

VOCE

BCQ-Lag

COptiDICE

C-CRR

Cost limit

Fig. 4 reveals an intriguing phenomenon: VOCE does not achieve the highest reward and lowest cost at φ =1.0 but instead attains the highest reward within the range of 0.7 to 0.9 (The range will vary depending on changes in the sampling method or policy.). This indicates that appropriately increasing the number of constraint-satisfying trajectories in the dataset benefits VOCE in improving its rewards and reducing costs. However, excessive augmentation of constraint-satisfying trajectories in the dataset may lead to a decrease in rewards and even an increase in costs.

0.5

0.6

0.7

0.8

0.9

1.0

24

26

28

30

32

34

Reward of episode

Point-Goal

0.5

0.6

0.7

0.8

0.9

1.0

0

5

10

15

20

25

Cost of episode

Point-Goal

0.5

0.6

0.7

0.8

0.9

1.0

30

35

40

Reward of episode

Point-Button

0.5

0.6

0.7

0.8

0.9

1.0

0

20

40

Cost of episode

Point-Button

Conservative estimation of rewards and costs. To assess the impact of conservative estimation on VOCE, we conducted ablation experiments on the conservative estimation of both reward and cost Q-values. Fig. 5 illustrates the rewards and costs of VOCE, VOCE-Qr, and VOCE-Qc across four tasks. VOCE-Qr stands for the VOCE method with the conservative estimation of Q-values removed for rewards. VOCE-Qc represents the VOCE method with the conservative estimation of Q-values removed for costs. The results presented in Fig. 5 demonstrate that the rewards of VOCE-Qr are notably lower than those of VOCE. This indicates that employing a lower-bound conservative estimation of reward Q-values helps eliminate extrapolation errors caused by OOD actions, thereby significantly improving the reward of VOCE. Furthermore, the results from Fig. 5 reveal that the

rewards of VOCE-Qc are comparable to or even surpass those of VOCE. However, in some tasks, the costs of VOCE-Qc exceed the cost constraints. This suggests that utilizing an upper-bound conservative estimation of cost Q-values helps reduce the costs of VOCE, thereby enhancing the safety of VOCE.

0.00

0.25

0.50

0.75

1.00

1.25

1.50

20

0

20

Reward of episode

Point-Goal

0.00

0.25

0.50

0.75

1.00

1.25

1.50

20

0

20

Car-Goal

0.0

0.5

1.0

1.5

2.0

2.5

3.0

20

0

20

Point-Button

0.0

0.5

1.0

1.5

2.0

2.5

3.0

40

20

0

20

Car-Button

0.00

0.25

0.50

0.75

1.00

1.25

1.50

Step(*1e4)

0

20

40

Cost of episode

0.00

0.25

0.50

0.75

1.00

1.25

1.50

Step(*1e4)

0

10

20

30

40

VOCE

VOCE_Qr

VOCE_Qc

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Step(*1e4)

0

20

40

60

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Step(*1e4)

0

20

40

60

Cost limit

5 Related Work

In this section, we elaborate on the work pertaining to offline safe RL and scrutinize three aspects of safe RL, offline RL, and offline safe RL.

Safe RL. Currently, Safe RL typically addresses optimization objectives with constraints using the primal-dual framework. The Lagrange version of PPO and TRPO [36, 37] widely used as the underlying baselines combines the constraint information with the original objective function via Lagrange multipliers. CPO [38] is the first general-purpose policy search algorithm for safe RL that guarantees that the near constraints are satisfied at each iteration. However, the second-order method [38, 39] requires second-order information and thus brings a higher computational burden. To address the approximation errors associated with Taylor approximations and inverting a highdimensional Fisher information matrix, first-order methods such as CUP [40] and APPO [41] achieve better performance and implementation.

Offline RL. Offline RL, also known as batch RL, considers the problem of updating a policy from an offline dataset without interacting with the environment. There are mainly two approaches of policy constraints and Q-value regularization to solve the problem of OOD actions in offline RL [42-44]. The earliest proposed Batch-Constrained deep Q-learning (BCQ) algorithm [24] is a typical policyconstrained offline RL algorithm, which employs CVAE to learn behavioral policy generation models, and finds the optimal policy by maximizing the Q-value. On this basis, a series of policy-constrained algorithms [45-48] are proposed, which mainly constrain the actions explored through the behavioral policy model. In addition, due to the obvious stability of the Q-value regularization approach, it has been extensively studied. A conservative Q-learning algorithm [30] proposed by Kumar et al., learns a conservative Q-value by enhancing the standard Bellman error objective with a simple Q-value regularizer. Subsequently, a series of Q-value regularization methods [49, 6, 50] were proposed which generally learn a conservative Q-value through regularization items or clipping Q-learning.

Offline safe RL. Offline safe RL is to solve the optimization problem of safety policy under the offline dataset. An offline safe RL algorithm with constraint penalty is proposed [16], which improves the objective function of the cost Q-value via using the constraint penalty item, and ensures the safety of the policy via terminating unsafe trajectory in advance. On the other hand, a constrained offline optimization algorithm (COPO) [11] defines the RL problem based on linear programming and constrains the distance between the final policy and offline sample behavior through regulation terms to solve the problem of offline safe RL algorithms. Additionally, this algorithm sets the discount factor γ = 1 , and changes the maximizing discount reward objective to the maximizing mean reward objective. Similar to the COPO algorithm, an offline policy optimization via stationary distribution correction estimation (CoptiDICE) [18] also utilizes the linear programming method of RL to solve

the optimal stationary distribution instead of the gradient strategy, which expands the discount factor to γ ∈ (0 , 1] .

6 Conclusion

In this work, we propose a variational optimization with conservative estimation for offline safe RL algorithm to address the problem of optimizing safe policies using offline datasets. We introduce probabilistic inference to reframe the problem of offline safe RL, and we mitigate estimation errors caused by parameter approximation through employing nonparametric variational distributions instead of parameterized policies. Furthermore, we utilize the upper and lower bounds of Q-values to estimate the Q-values of cost and reward, thereby mitigating extrapolation errors caused by OOD actions. Finally, extensive comparisons and ablation experiments demonstrate that the VOCE algorithm outperforms state-of-the-art algorithms in terms of safety.

Acknowledgments and Disclosure of Funding

This work is supported by the National Natural Science Foundation of China (No. 62372329), in part by the National Key Research and Development Program of China (No. 2021YFB2501104), in part by Shanghai Rising Star Program (No.21QC1400900), in part by Tongji-Qomolo Autonomous Driving Commercial Vehicle Joint Lab Project, and in part by Xiaomi Young Talents Program. We thank Li Shen and Yiqin Yang for the insightful discussion.

References

Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning , pages 104-114. PMLR, 2020.

Ruiqi Zhang, Jing Hou, Guang Chen, Zhijun Li, Jianxiao Chen, and Alois Knoll. Residual policy learning facilitates efficient model-free autonomous racing. IEEE Robotics and Automation Letters , 7(4):11625-11632, 2022. doi: 10.1109/LRA.2022.3192770.

Jing Hou, Guang Chen, Zhijun Li, Wei He, Shangding Gu, Alois Knoll, and Changjun Jiang. Hybrid residual multiexpert reinforcement learning for spatial scheduling of high-density parking lots. IEEE Transactions on Cybernetics , pages 1-13, 2023. doi: 10.1109/TCYB.2023. 3312647.

Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. First return, then explore. Nature , 590(7847):580-586, 2021.

Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline rl? In International Conference on Machine Learning , pages 5084-5096. PMLR, 2021.

Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-based offline reinforcement learning with diversified q-ensemble. Advances in Neural Information Processing Systems , 34, 2021.

B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yogamani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions on Intelligent Transportation Systems , 2021.

Yunxiao Shan, Boli Zheng, Longsheng Chen, Long Chen, and De Chen. A reinforcement learning-based adaptive path tracking approach for autonomous driving. IEEE Transactions on Vehicular Technology , 69(10):10581-10595, 2020.

Xiaobai Ma, Jiachen Li, Mykel J Kochenderfer, David Isele, and Kikuo Fujimura. Reinforcement learning for autonomous driving with latent state inference and spatial-temporal relationships. In 2021 IEEE International Conference on Robotics and Automation (ICRA) , pages 6064-6071. IEEE, 2021.

Jongmin Lee, Cosmin Paduraru, Daniel J Mankowitz, Nicolas Heess, Doina Precup, Kee-Eung Kim, and Arthur Guez. COptiDICE: Offline constrained reinforcement learning via stationary distribution correction estimation. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=FLA55mBee6Q .

Nicholas Polosky, Bruno C Da Silva, Madalina Fiterau, and Jithin Jagannath. Constrained offline policy optimization. In International Conference on Machine Learning , pages 17801-17810. PMLR, 2022.

Dibya Ghosh, Anurag Ajay, Pulkit Agrawal, and Sergey Levine. Offline rl policies should be trained to be adaptive. In International Conference on Machine Learning , pages 7513-7530. PMLR, 2022.

Katie Kang, Paula Gradu, Jason J Choi, Michael Janner, Claire Tomlin, and Sergey Levine. Lyapunov density models: Constraining distribution shift in learning-based control. In International Conference on Machine Learning , pages 10708-10733. PMLR, 2022.

Hoang Le, Cameron Voloshin, and Yisong Yue. Batch policy learning under constraints. In International Conference on Machine Learning , pages 3703-3712. PMLR, 2019.

Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems , 34:12731286, 2021.

Haoran Xu, Xianyuan Zhan, and Xiangyu Zhu. Constraints penalized q-learning for safe offline reinforcement learning. Proceedings of the AAAI Conference on Artificial Intelligence , 36(8): 8753-8760, 2022.

Bo Dai, Ofir Nachum, Yinlam Chow, Lihong Li, Csaba Szepesvári, and Dale Schuurmans. Coindice: Off-policy confidence interval estimation. Advances in neural information processing systems , 33:9398-9411, 2020.

Jongmin Lee, Wonseok Jeon, Byungjun Lee, Joelle Pineau, and Kee-Eung Kim. Optidice: Offline policy optimization via stationary distribution correction estimation. In International Conference on Machine Learning , pages 6120-6130. PMLR, 2021.

Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning , pages 1928-1937. PMLR, 2016.

Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning , pages 1587-1596. PMLR, 2018.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning , pages 1861-1870. PMLR, 2018.

Eitan Altman. Constrained Markov decision processes: stochastic modeling . Routledge, 1999.

Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 29. Curran Associates, Inc., 2016. URL https://proceedings.neurips.cc/paper/2016/file/ 8d8818c8e140c64c743113f563cf750f-Paper.pdf .

Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning , pages 2052-2062. PMLR, 2019.

Yiqin Yang, Xiaoteng Ma, Chenghao Li, Zewu Zheng, Qiyuan Zhang, Gao Huang, Jun Yang, and Qianchuan Zhao. Believe what you see: Implicit constraint approach for offline multi-agent reinforcement learning. Advances in Neural Information Processing Systems , 34:10299-10312, 2021.

Zuxin Liu, Zhepeng Cen, Vladislav Isenbaev, Wei Liu, Steven Wu, Bo Li, and Ding Zhao. Constrained variational policy optimization for safe reinforcement learning. In International Conference on Machine Learning , pages 13644-13668. PMLR, 2022.

Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint arXiv:1805.00909 , 2018.

Mohammed Alshiekh, Roderick Bloem, Rüdiger Ehlers, Bettina Könighofer, Scott Niekum, and Ufuk Topcu. Safe reinforcement learning via shielding. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32, 2018.

Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep reinforcement learning. arXiv preprint arXiv:1910.01708 , 7(1):2, 2019.

Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems , 33: 1179-1191, 2020.

Peter Auer, Thomas Jaksch, and Ronald Ortner. Near-optimal regret bounds for reinforcement learning. In D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou, editors, Advances in Neural Information Processing Systems , volume 21. Curran Associates, Inc., 2008. URL https://proceedings.neurips.cc/paper/2008/file/ e4a6222cdb5b34375400904f03d8e6a5-Paper.pdf .

Zuxin Liu, Zijian Guo, Haohong Lin, Yihang Yao, Jiacheng Zhu, Zhepeng Cen, Hanjiang Hu, Wenhao Yu, Tingnan Zhang, Jie Tan, et al. Datasets and benchmarks for offline safe reinforcement learning. arXiv preprint arXiv:2306.09303 , 2023.

Zuxin Liu, Zijian Guo, Yihang Yao, Zhepeng Cen, Wenhao Yu, Tingnan Zhang, and Ding Zhao. Constrained decision transformer for offline safe reinforcement learning. arXiv preprint arXiv:2302.07351 , 2023.

Jiaming Ji, Jiayi Zhou, Borong Zhang, Juntao Dai, Xuehai Pan, Ruiyang Sun, Weidong Huang, Yiran Geng, Mickel Liu, and Yaodong Yang. Omnisafe: An infrastructure for accelerating safe reinforcement learning research. arXiv preprint arXiv:2305.09304 , 2023.

Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenberg, Scott E Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, et al. Critic regularized regression. Advances in Neural Information Processing Systems , 33:7768-7778, 2020.

Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep reinforcement learning. arXiv preprint arXiv:1910.01708 , 7:1, 2019.

Jiaming Ji, Borong Zhang, Jiayi Zhou, Xuehai Pan, Weidong Huang, Ruiyang Sun, Yiran Geng, Yifan Zhong, Juntao Dai, and Yaodong Yang. Safety-gymnasium: A unified safe reinforcement learning benchmark, 2023.

Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference on machine learning , pages 22-31. PMLR, 2017.

Tsung-Yen Yang, Justinian Rosca, Karthik Narasimhan, and Peter J Ramadge. Projection-based constrained policy optimization. arXiv preprint arXiv:2010.03152 , 2020.

Long Yang, Jiaming Ji, Juntao Dai, Linrui Zhang, Binbin Zhou, Pengfei Li, Yaodong Yang, and Gang Pan. Constrained update projection approach to safe policy optimization. arXiv preprint arXiv:2209.07089 , 2022.

Juntao Dai, Jiaming Ji, Long Yang, Qian Zheng, and Gang Pan. Augmented proximal policy optimization for safe reinforcement learning. Proceedings of the AAAI Conference on Artificial Intelligence , 37(6):7288-7295, Jun. 2023. doi: 10.1609/aaai.v37i6.25888. URL https: //ojs.aaai.org/index.php/AAAI/article/view/25888 .

Marin Toromanoff, Emilie Wirbel, and Fabien Moutarde. End-to-end model-free reinforcement learning for urban driving using implicit affordances. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 7153-7162, 2020.

Jacob Buckman, Carles Gelada, and Marc G Bellemare. The importance of pessimism in fixed-dataset policy optimization. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum?id=E3Ys6a1NTGT .

Shangtong Zhang, Wendelin Boehmer, and Shimon Whiteson. Generalized off-policy actorcritic. Advances in Neural Information Processing Systems , 32, 2019.

Wenxuan Zhou, Sujay Bajracharya, and David Held. Plas: Latent action space for offline reinforcement learning. In Conference on Robot Learning , 2020.

Hua Wei, Deheng Ye, Zhao Liu, Hao Wu, Bo Yuan, Qiang Fu, Wei Yang, and Zhenhui Li. Boosting offline reinforcement learning with residual generative modeling. In International Joint Conference on Artificial Intelligence , 2021.

Jiayi Guan, Shangding Gu, Zhijun Li, Jing Hou, Yiqin Yang, Guang Chen, and Changjun Jiang. Uac: Offline reinforcement learning with uncertain action constraint. IEEE Transactions on Cognitive and Developmental Systems , 2023.

Noah Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavior modelling priors for offline reinforcement learning. In International Conference on Learning Representations , 2020. URL https://openreview.net/forum? id=rke7geHtwH .

Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems , 34:20132-20145, 2021.

Hongchang Zhang, Jianzhun Shao, Yuhang Jiang, Shuncheng He, Guanwen Zhang, and Xiangyang Ji. State deviation correction for offline reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence , volume 36, 2022.

A Implementation Details

A.1 Algorithm

The VOCE algorithm is shown in Algorithm 1.

Algorithm 1: VOCE Algorithm Input: Dataset D = { ( s t , a t , r t , s t +1 ) n i =0 } Output: Policy network π θ ; Q-value network of reward Q r , Q r t , ; Q-value network of cost Q c i , Q c i t ; 1 for each batch do 2 Sample a batch of transitions ( s t , a t , r t , s t +1 ) from the buffer D ; 3 Compute the Q-value ˆ Q r ( s t , a t ) and ˆ Q c i ( s t , a t ) via Bellman backup; 4 ˆ B π ˆ Q r ( s t , a t ) = r ( s t , a t ) + γ ˆ Q r ( s t +1 , a t +1 ) ; 5 ˆ B π ˆ Q c i ( s t , a t ) = c i ( s t , a t ) + γ ˆ Q c i ( s t +1 , a t +1 ) ; 6 Update the Q r via the Eq. (13); 7 Update the Q c i via the Eq. (17); 8 Update the optimal dual variables η and λ via solving the convex optimization issue in the Eq. (11); 9 Compute the optimal variational distribution q ∗ ( a t | s t ) by the Eq. (10) via Q r , Q c i and Lagrangian multipliers η and λ ; 10 Update the Policy Parameter π θ via the Eq. (20); 11 Q r t ← τQ r +(1 -τ ) Q r t ; 12 Q c i t, ← τQ c i +(1 -τ ) Q c i t ;

B Proofs and Discussions

B.1 Proof and Discussion the Proposition 3.4

Proposition B.1. The closed-form solution of the optimal variational distribution that satisfies the safety constraint in Eq. (21) is given as:

Proof. The unconstrained problem derived through the Lagrange multiplier method is expressed as follows:

To simplify Eq. (22), we introduce a composite Q-value and define it as Q rc ( s t , a t ) ≜ Q r ( s t , a t ) -λ T Q c i ( s t , a t ) . Then we rewrite the Eq. (22) as:

For a fixed λ and η , we write the dual problem [17, 18] of the above optimization problem of maximization max q ≥ 0 L ( q, λ, η ) :

Since the strong duality holds, satisfying the KKT condition is a necessary and sufficient condition for the solution of the primal and dual problems.

Condition 1 (Primal feasibility). q ∗ ( a t | s t ) ≥ 0 , ∀ s t , a t

Condition 2 (Dual feasibility). µ ∗ ( s t , a t ) ≥ 0 , ∀ s t , a t

Condition 4 (Complementary). q ∗ ( a t | s t ) µ ∗ ( s t , a t ) = 0 , ∀ s t , a t

Form Stationarity we have:

Since the π θ ( a t | s t ) > 0 , thus the q ∗ ( a t | s t ) > 0 . Then the Primal feasibility is always satisfied. Furthermore, we only need to consider the cases where q ∗ ( a t | s t ) > 0 .

( Case 1 q ∗ ( a t | s t ) > 0 ) In this case, µ ∗ ( s t , a t ) = 0 where Dual feasibility and Complementary slackness are satisfied. In addition, Stationarity also applies to :

Therefore, KKT conditions (conditions 1-4 ) are satisfied.

B.2 Proof and Discussion the Proposition 3.5

Proposition B.2. The λ, η are dual variables and are solved via the convex optimization problem of the following formula.

Proof. By rearranging terms and taking the expectation over a t , Eq. (21) can be rewritten as:

where the ∑ a t q ∗ ( a t | s t ) = 1 , thus the Eq. (28) can be rewritten as:

By substituting Eq. (29) and (26) into Eq. (22) to eliminate q ( a t | s t ) , we can rewrite Eq. (22) as:

Since Eq. (30) eliminates δ , the dual optimization problem of λ and η can be written as:

B.3 Proof and Discussion the Proposition 3.6

Proposition B.3. For any π M ( a t | s t ) with supp π M ( a t | s t ) ⊂ ˆ π β , ∀ s t , a t ∈ D , the Q-value function Q r via iterating Eq. (12) satisfies:

Thus, if κ ≥ C r,p,δ R max (1 -γ ) √ |D ( s t ,a t ) | [ π M ( a t | s t ) π β ( a t | s t ) ] -1 , the iterative update Eq. (12) can guarantees ˆ Q r ≤ Q r .

Proof. The iterative equation for updating the reward Q-value, which involves an additional minimization of the Q-value and the standard Bellman error objective, is expressed as follows:

where the ˆ B π ˆ Q r ( s t , a t ) = r ( s t , a t ) + γ ˆ Q r ( s t +1 , a t +1 ) . With importance sampling, the Eq. (33) can be written as:

Setting the derivative of Eq. (34) to 0 , we obtain the following expression:

Note that the Eq. (34) utilizes the empirical Bellman operator ˆ B π instead of the actual Bellman Operator B π . Following the related work [23, 31], we employ the concentration properties to determine the error. For any ∀ s t , a t ∈ D , with the probability ≤ 1 -δ , the error can be written as:

where C r,p,δ is a constant depending on the concentration properties of r ( s t , a t ) , p ( s t +1 | s t , a t ) and the δ, δ ∈ (0 , 1) . The ( D ( s t , a t )) -1 2 represents a vector of size |S||A| containing the square root inverse count of each state action transition. R max = max s t ,a t ∼ D { r ( s t , a t ) } represents the maximum reward obtained in a single step. Based on the inequality of Eq. (36) and Eq. (35) we obtain:

The fixed point in the updating process of Eq. (37) can be expressed as:

To guarantee the lower bound of ˆ Q r , we eliminate the potential overestimation caused by C r,p,δ R max (1 -γ ) √ |D ( s t ,a t ) | via choosing an appropriate κ . Since ( I -γP π ) -1 is a matrix of non-negative entries, the κ that guarantees the lower bound of Q r needs to satisfy:

B.4 Proof and Discussion the Proposition 3.7

Proposition B.4. When π M ( a t | s t ) = π θ ( a t | s t ) , according to Eq. (13), we obtain a lower bound for the true value of V r = E a t ∼ π θ ( a t | s t ) [ Q r ( s t , a t )] that satisfies the following inequality:

Thus, if κ ≥ C r,p,δ R max (1 -γ ) √ |D ( s t ) | [ E a t ∼ ˆ π β [ π θ ( a t | s t ) π θ ( a t | s t ) -1 ]] -1 , the Eq. (13) can guarantee ˆ V r ≤ V r .

Proof. The following Eq. (43) represents the iterative formula for the reward Q-value with an additional maximization term introduced under the sample distribution ˆ π β .

where κ is a tradeoff factor. With importance sampling, the Eq. (43) can be rewritten as:

Setting the derivative of Eq. (44) to 0 , we obtain the following expression:

By taking the expectation of both side of Eq. (45) with respect of the action, we obtain:

When π M ( a t | s t ) = π θ ( a t | s t ) , we obtain from Eq. (46):

Since ∑ a t π θ ( a t | s t ) = ∑ a t ˆ π β ( a t | s t ) = 1 , we can conclude that the value estimate ˆ V k ( s t ) is underestimated as ˆ V r k +1 ( s t ) ≤ ˆ B π ˆ V k ( s t ) based on Eq. (47). Next, considering the sampling error, we obtain from Eq. (45) and Eq. 36:

To guarantee the lower bound of ˆ V r , we eliminate the potential overestimation caused by C r,p,δ R max (1 -γ ) √ |D ( s t ) | via choosing an appropriate κ . Since ( I -γP π ) -1 is a matrix of non-negative entries, the κ that guarantees the lower bound of V r needs to satisfy:

B.5 Proof and Discussion the Proposition 3.8

Proposition B.5. For any π R ( a t | s t ) with supp π R ( a t | s t ) ⊂ ˆ π β , ∀ s t , a t ∈ D , the Q-value function Q c i via iterating Eq. (17) satisfies:

Thus, if χ ≥ C r,p,δ C max (1 -γ ) √ |D ( s t ,a t ) | [ π R ( a t | s t ) ˆ π β ( a t | s t ) ] -1 , the iterative update Eq. (17) can guarantee ˆ Q c i ≥ Q c i .

Proof. Using the importance sampling, Eq. (17) can be written as:

Setting the derivative of Eq. (56) to 0 , we obtain the following expression:

Note that the Eq. (57) utilizes the empirical Bellman operator ˆ B π instead of the actual Bellman Operator B π . Following the related work [23, 31], we employ the concentration properties to determine the error. For any ∀ s t , a t ∈ D , with the probability ≤ 1 -δ , the error can be written as:

where C r,p,δ is a constant depending on the concentration properties of c ( s t , a t ) , p ( s t +1 | s t , a t ) and the δ, δ ∈ (0 , 1) . The ( D ( s t , a t )) -1 2 represents a vector of size |S||A| containing the square root inverse count of each state action transition. C max = max s t ,a t ∼ D { c i ( s t , a t ) } represents the maximum cost obtained in a single step. Based on the inequality of Eq. (58) and Eq. (57) we can obtain:

The fixed point in the updating process of Eq. (59) can be expressed as:

We guarantee the upper bound of ˆ Q c i via choosing an appropriate value of χ based on the Eq. (62). Beside the ( I -γP π ) -1 is a matrix of non-negative entries.

C Experimental Details

C.1 Dataset Details

As shown in Fig. 6, we utilize the online policy CVPO to collect sample data with different φ values during three distinct stages of the training process. The variable φ represents the proportion of safe trajectories in the collected samples compared to the total number of sample trajectories. Behavior 1 corresponds to φ = 0 . 4 , behavior 2 corresponds to φ = 0 . 6 , and so on. To ensure that the collected sample data satisfies the designated φ value, we incorporate an additional trajectory filtering policy into the data collection procedure. Furthermore, the total number of steps for each category of samples collected is 1 e 7 .

0

20

40

60

80

100

120

5

10

15

20

25

30

35

40

Reward of episode

0.8

0.6

0.4

Cost of episodes

0

500

1000

1500

2000

2500

3000

Epoch

5

0

5

10

15

20

25

30

35

40

Reward of episodes

1

2

3

0

500

1000

1500

2000

2500

3000

Epoch

0

20

40

60

80

100

Cost of episodes

1

2

3

C.2 Experimental Task Details

The Point-Goal and Car-Goal task. The objective of the agent is to reach a goal while avoiding surrounding obstacles. Once the agent reaches the correct goal, the environment randomly generates a new goal. The agent receives a reward when it approaches or reaches the goal. There are hazards and vases as obstacles in the environment, and the agent incurs a cost when encountering them. Hazards are immovable, while vases can be moved. The observation state is the relative pose between the agent and obstacles and goals, represented by a point-based pseudo-LIDAR. We select two tasks, namely Point-Goal and Car-Goal tasks [26, 29]. These tasks are performed by the Point and Car agents, respectively. The Point agent is a robot constrained to a 2D plane, capable of rotation as well as forward and backward movement. The Car agent is a slightly more complex robot with two independent parallel wheels and a freely rolling rear wheel.

The Point-Button and Car-Button task. The objective of the agent is to navigate around stationary and moving obstacles in the environment and press one of the multiple target buttons. The task is similar to the goal task, where the agent receives a reward for either approaching or pressing the target button. The Button task involves the presence of dynamic obstacles that move rapidly along predefined trajectories. As the agent incurs costs upon colliding with these dynamic obstacles, the Button task is more challenging compared to the Goal task. In the Button task, we select two tasks: the Point-Button task and the Car-Button task [26, 29].

C.3 Hyper-parameters

As shown in Table 1, we share the hyperparameters of the CEVO algorithm to facilitate reproducibility. Furthermore, we will share the code on GitHub at a later time.

C.4 Datasets for the Preliminaries Section

In the preliminaries section, we employed a dataset with φ = 1 . 0 to evaluate offline RL methods. Additionally, we utilized a dataset with φ = 0 . 8 to evaluate off-policy safe RL methods. As shown in Fig. 7, φ = 1 . 0 indicates that the dataset includes samples with not only high rewards but also costs within the cost limit. In addition, φ = 0 . 8 indicates that the dataset includes samples with high rewards, while also containing samples that exceed the cost limit.

C.5 Datasets for the Ablation Experiment

Fig 7 presents the datasets collected under the same policy but with different φ values. To ensure the acquisition of datasets that satisfy the φ values, we incorporate a trajectory filtering strategy into the data collection procedure and introduce random actions. Concretely, when the random value is less than the predefined threshold ε , we employ random actions instead of the actions provided by the policy.

0

20

40

60

80

100

Cost of episode

15

20

25

30

35

40

Reward of episode

1.0

0.9

0.8

0.7

0.6

0.5

0

500

1000

1500

2000

2500

3000

Epoch

5

0

5

10

15

20

25

30

35

40

Reward of episodes

0

500

1000

1500

2000

2500

3000

Epoch

0

20

40

60

80

100

Cost of episodes

C.6 The Ablation Experiment of the Parameters

We intuitively believe that the value of the parameter ϵ in the objective 8 affects the performance of the VOCE algorithm. Therefore, we conduct ablation experiments on this parameter. The results shown in Fig 8 indicate that setting the parameter ϵ too small reduces the convergence speed of the policy and even decreases the algorithm's performance. Conversely, when the parameter ϵ is set to larger, it may lead to instability in the policy training process. Furthermore, from the illustrated results, we can conclude that within this range of parameter ϵ ∈ [0 , 1] , it ensures the policy's good convergence speed and stability. Note that the parameter ϵ ∈ [0 , 1] represents a feasible range rather than an optimal interval.

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

Step(*1e4)

20

0

20

Reward of episode

Point-Goal

0.05

0.1

1

10

100

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

Step(*1e4)

10

20

30

40

Cost of episode

Point-Goal

Cost limit

C.7 The trade-off factors

Regarding the trade-off factors κ and χ , we adopt a dual-tuning approach with gradient-based adaptation instead of a manual setting. To facilitate the observation of changes in the trade-off factors κ and χ , we record the variation curves of these parameters during the training process. The results of Fig. 9 indicate that as the model gradually converges, the trade-off factors κ and χ gradually decrease, eventually approaching zero.

0.0

0.5

1.0

1.5

2.0

2.5

3.0

0.00

0.25

0.50

0.75

1.00

0.0

0.5

1.0

1.5

2.0

2.5

3.0

0.00

0.25

0.50

0.75

1.00

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Step(*1e4)

20

0

20

40

Reward of episode

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Step(*1e4)

0

25

50

75

100

Cost of episode

C.8 The time costs of training and testing

We recorded the time costs of various modules of the VOCE and the baseline algorithm during both training and testing. The results presented in Table 2 indicate that during training, the VOCE algorithm incurs additional time costs per step compared to the baseline algorithm due to the execution of additional networks and optimization parameters. However, during the testing phase, as only the policy network is executed, the time cost of VOCE is similar to the baseline algorithm. Furthermore, the execution time per step during testing is significantly lower than 1 ms , meeting the real-time requirements for robot and autonomous driving control. Additionally, the results from Table 3 reveal that the evaluation time cost of the Q-values is relatively high due to the computation of multiple target Q-values and gradients.

D Limitations and Future Work

The setup of this work involves learning safety-constrained policies from offline data without interacting with the environment. Therefore, both the size and quality of the dataset directly impact

the algorithm's performance. Moreover, in the offline setting, the available sample data is limited, making it challenging to adequately represent the state transition matrix P , in a non-stationary environment. Consequently, the VOCE algorithm struggles to learn high-reward policies that satisfy safety constraints in non-stationary environments.

Peer reviews:
comments: SUMMARY:
This paper utilizes probabilistic inference to address the problem of offline safe RL by introducing non-parametric variational distributions. 
 Pessimistic estimation of Q-values are used to avoid extrapolation errors caused by OOD actions. Extensive comparative numerical experiments are carried with respect to both reward and cost curve.

STRENGTHS:
1. The paper has a solid chain of theoretical derivation.
2. The numerical results of VOCE seems promising.

WEAKNESSES:
1. The paper is not well-organized written. For example, several notations are not defined before used or at least not easy to find. For example, Q^{r}(s,a) used in equation 7, \pi_{M}(a_t|s_t) in equation 13. And it's more clear to have a complete summary of the main algorithm  VOCE, rather than deriving literally step by step. 

2. Some claims in the paper need further theoretical explanation or numerical experiments to clarify.

3. It's not well studied how key tuning parameters affect the numerical conclusion.

QUESTIONS:
1. The algorithm involves lots of parameters to tune, like \epsilon in equation 11, \chi in equation 17. I am a little bit suspicious of how robust those parameters affect the numerical results. Maybe some ablation study is needed to further clarify. 

2. Why pessimistic estimation of Q-values can help avoid extrapolation errors caused by OOD actions. I don't get the intuition here. More theoretical analysis / explanation or numerical experiments are needed.

LIMITATIONS:
No negative societal impact of their work is seen.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces an interesting algorithm for offline safe RL, called Variational Optimization with Conservative Estimation (VOCE). The primary challenge tackled by the authors is the influence of safety constraints and out-of-distribution (OOD) actions, which often hamper the optimization of high-reward policies while maintaining safety. Traditional methods, such as linear programming and exploration-evaluation methods, have shown their limitations in coping with these challenges. In response, the authors propose a probabilistic inference-based framework that leverages Expectation and Maximization (EM) to provide more flexible policy optimization. It employs pessimistic estimation methods to calculate the Q-value of cost and reward, thus mitigating extrapolation errors due to OOD actions.\n\nSTRENGTHS:\n1. This works is an interesting extension of off-policy safe RL (particularly CVPO) to the offline setting. The application of probabilistic inference to the problem of offline safe RL introduces non-parametric variational distributions to replace parameterized policies, improving the flexibility for optimizing safe policies.\n2. The paper's methodology is clear and well-presented. The derivation of upper and lower bounds for Q-value estimation using a pessimistic estimation approach, which is then utilized to estimate Q-values of costs and rewards to mitigate extrapolation errors from OOD actions.\n3. Experiments demonstrate the superiority of the VOCE algorithm over baseline methods in terms of safety.\n\nWEAKNESSES:\n1. It seems that many claims and arguments are directly derived from CVPO, such as Propositions 3.4 and 3.5. The authors should properly cite the references accordingly. In addition, some other recent offline safe RL [methods](https://arxiv.org/abs/2302.07351) and [benchmarks](https://arxiv.org/abs/2306.09303) could be discussed in the paper.\n2. I am not sure whether the EM optimization objectives in CVPO could be directly applied to the offline setting, since the introduction of the KL constraints in the E-step and M-step aims to regularize the policy update such that the behavior policy will not deviate too much and cause catastrophic failure in the online setting; however, in the offline setting, since the policy will not interact with the environment, this constraint seems not very necessary? Would regularizing the KL constraints between the variational distribution and the behavior policy in the datasets (with some density estimations for the behavior policies) be a better choice?\n\nQUESTIONS:\n1. I think one major challenge for these constrained optimization-based approaches for offline safe learning is to accurately estimate the Q-value functions, since a subtle inaccuracy will be amplified by comparing Qc estimates and the target threshold after more and more optimization iterations. So I am curious how the authors think and tackle this problem, apart from using pessimism. In fact, pessimism alone can not solve the Q value mismatch between the current optimized policy and the behavior policy in the datasets.\n2. Following up on the above question, I wonder how VOCE would perform in more diverse datasets with a wider span of trajectories over the cost-reward-return spaces. For example, collecting datasets with varying thresholds, hyper-parameters, and algorithms, as shown in [this](https://arxiv.org/abs/2306.09303) work.\n\nLIMITATIONS:\nI did not find related discussions.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper applies CQL to offline safe RL tasks. It introduces a non-parametric policy to search for the actions that satisfy the constraints. Lagrangian multipliers are introduced for constraining the KL divergence w.r.t. the current policy and the additional constraints. CQL-style additional terms are introduced to avoid extrapolation issues during the offline training. The method is evaluated at four safe RL environments and demonstrates that the proposed approach can achieve better rewards while better approaching the cost limits.\n\nSTRENGTHS:\n- The paper proposed a solid approach that consists necessary ingredients for offline-safe RL. The conservative estimation follows CQL and should be able to mitigate the issues in offline RL.\n- An optimal variational distribution is introduced to improve the policy in a non-parameterized way. This step is novel in offline safe RL.\n- The paper proposed an offline safe RL benchmark and made a careful study.\n\nWEAKNESSES:\n- The novelty of the method could be more extensive. The method mostly combines existing methods. The variational method, non-parameterized policy estimation, constraint formulation, and conservative Q estimation are all standard methods.\n- Some mathematical notations are not clear. For example, Eq (19) and Eq (20) are quite misleading. I think it should be log \\pi_\\theta(a_t|s_t) in Eq(2).\n- If my understanding is correct, the proposed method is not able to reach the cost limitation in several point-button and car-button environments, which suggests the inadequate of the proposed method.\n\nQUESTIONS:\nI would like to know the reason that the authors chose to follow CQL instead of other offline RL algorithms. Since there is already a KL term in the optimization objective, I think Behavior cloning or advantage-weighted approach, which directly constrains the KL divergence between the learned policy and the behavior policy, is more straightforward here and may achieve higher performance in practice. Is there any reason to consider the conservative Q estimation rather than other choices?\n\nLIMITATIONS:\nnot discussed", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
The paper presents a variational approach to offline safe reinforcement learning (RL), where the class of variational distributions is the set of policies which satisfy the cost constraints. It is shown how to obtain the closed-form solution for the optimal variational distribution, and how to extract a parametric policy given the variational distribution. To guarantee that the reward Q function is not overestimated and the cost Q function is not underestimated, the tools of Conservative Q Learning (CQL) are incorporated.

STRENGTHS:
* The variational formulation is novel to my knowledge and may inspire additional work in this area.
* VOCE performance is good compared to existing algorithms, both in terms of high reward and satisfying the cost constraint.
* The paper is generally clear and not hard to read.

WEAKNESSES:
* There is no discussion of the runtime of the algorithm, which I imagine is comparatively high, given that you have to solve an optimization problem (Eqn. 11) at every step.
* While the final hyperparameters are provided, we are not told how they were tuned, nor do we have a sense of how sensitive VOCE’s performance is to their values.

QUESTIONS:
* How are the trade-off factors ($\kappa$, $\chi$) determined? Are you using a dual tuning method akin to the $\alpha$ in CQL?
* In Eqn. (20), should be $a_t$ instead of $\cdot$?

LIMITATIONS:
No, limitations are not addressed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07