Title: Safe and Efficient: A Primal-Dual Method for Offline Convex CMDPs under Partial Data Coverage
Venue: NeurIPS 2024
Accepted: yes
Keywords: safe reinforcement learning

convex MDPs

offline

sample efficient.
Abstract: Offline safe reinforcement learning (RL) aims to find an optimal policy using a pre-collected dataset when data collection is impractical or risky. We propose a novel linear programming (LP) based primal-dual algorithm for convex MDPs that incorporates ``uncertainty'' parameters to improve data efficiency while requiring only partial data coverage assumption. Our theoretical results achieve a sample complexity of $\mathcal{O}(1/(1-\gamma)\sqrt{n})$ under general function approximation, improving the current state-of-the-art by a factor of $1/(1-\gamma)$, where $n$ is the number of data samples in an offline dataset, and $\gamma$ is the discount factor. The numerical experiments validate our theoretical findings, demonstrating the practical efficacy of our approach in achieving improved safety and learning efficiency in safe offline settings.

Safe and Efficient: A Primal-Dual Method for Offline Convex CMDPs under Partial Data Coverage

Abstract

Offline safe reinforcement learning (RL) aims to find an optimal policy using a pre-collected dataset when data collection is impractical or risky. We propose a novel linear programming (LP) based primal-dual algorithm for convex MDPs that incorporates 'uncertainty' parameters to improve data efficiency while requiring only partial data coverage assumption. Our theoretical results achieve a sample complexity of O (1 / (1 -γ ) √ n ) under general function approximation, improving the current state-of-the-art by a factor of 1 / (1 -γ ) , where n is the number of data samples in an offline dataset, and γ is the discount factor. The numerical experiments validate our theoretical findings, demonstrating the practical efficacy of our approach in achieving improved safety and learning efficiency in safe offline settings.

1 Introduction

Safe reinforcement learning (RL) aims to learn a reward-maximizing policy while satisfying multiple safety constraints, demonstrating its practicality in many real-world applications, such as autonomous driving [Kiran et al., 2021], robotics [Levine et al., 2016], and healthcare [Yu et al., 2021]. In these tasks, certain behaviors may potentially harm the agent or its surroundings, which is crucial for task completion. One way to mathematically characterize safe RL is through Constrained Markov Decision Processes (CMDPs) [Altman, 2021], where safety constraints are incorporated into the problem when optimizing the objective.

Offline RL aims to learn a sequence of actions from a pre-collected dataset to address scenarios where interacting with the environment is risky, expensive, or impractical. Ensuring sample efficiency in offline RL with function approximation typically requires additional assumptions about both the function classes and the dataset due to training instability and distribution shift issues [Fujimoto et al., 2019, Kostrikov et al., 2021, Paine et al., 2020]. Earlier studies [Chen and Jiang, 2019, Liao et al., 2022, Liu et al., 2019, Wang et al., 2019, Zhang et al., 2020b] in offline RL usually require that all functions in the function space are Bellman-complete and that the dataset has full coverage, meaning it covers the state-action distributions induced by all policies. This might be a mild and accepted assumption in offline RL without considering safety. However, it is highly unacceptable and impractical in safe offline RL, as it would require the dataset to cover all hazardous state-action pairs induced by all dangerous policies. To address the full coverage issue, later works [Chen and Jiang, 2022, Rashidinejad et al., 2021, Uehara and Sun, 2021, Xie et al., 2021, Zhan et al., 2022, Zhu et al.,

2023] reduce the assumption to single-policy coverage by using pessimism in the face of uncertainty. Unfortunately, all existing studies [Chen et al., 2022, Hong et al., 2024, Le et al., 2019] in safe offline RL still require coverage for all policies.

Beyond traditional offline safe RL, many applications do not fit the standard RL problem [Abel et al., 2021]. There is a substantial body of literature [Geist et al., 2022, Mutti et al., 2023, Zahavy et al., 2021] studying a more general scenario called convex MDPs, where the objective function is modeled as a convex (or concave) utility function instead of linear, as in the standard RL problem. This framework is quite general and captures various learning scenarios, including imitation [Abbeel and Ng, 2004], exploration [Hazan et al., 2019], and more. However, studying convex MDPs introduces additional challenges. In convex MDPs, moving beyond cumulative rewards means that the Bellman equation fails to hold due to the lack of reward additivity. This leads to breakdowns in many techniques based on Dynamic Programming (DP) [Zhang et al., 2020b]. Despite a large body of practical literature [Lee et al., 2021, Xu et al., 2022, Zheng et al., 2023], robust theoretical analysis remains lacking in this setting.

To address the coverage issue and extend the general function approximation setting, we focus on convex MDPs in the safe offline setting. Our main contributions are summarized below (the detailed comparisons can be found in Table 1):

We are the first to study offline convex MDPs under safety constraints with partial data coverage assumption. We reformulate the problem using marginalized importance weights to avoid issues caused by the Bellman equation's failure in convex MDPs.

G

G

the number of iteration is larger than n . The sample complexity of O 1 (1 -γ ) √ n improves the best existing result by a factor of 1 / (1 γ ) 2 .

We analyze the convergence rate of our proposed approach under partial coverage assumption and theoretically prove that our algorithm achieves O ( 1 (1 -γ ) √ n ) in both objective and violation bounds with general function approximation, when ( )

-

Experimental results on Imitation Learning and standard CMDPs demonstrate the generality and effectiveness of our algorithm. As Figure 1 shows our algorithm performs well even with a completely random and safety-violated offline dataset with general function approximation, which verifies our theoretical findings.

1.1 Related Work

Offline Safe RL: The offline safe reinforcement learning setting entails the agent learning from a fixed dataset while adhering to safety constraints. This involves a blend of offline RL and safe RL, yet research on this approach, particularly concerning theoretical analysis, remains limited. BCQL

augmented by BCQ [Fujimoto et al., 2019] optimizes the policy in the offline phase and applies the Lagrange method to handle constraints. CPQ [Xu et al., 2022] tackles the safety constraints by overestimating the cost value function of out-of-distribution ('unsafe') actions and updating the reward value function with 'safe' actions. Another line of work [Lee et al., 2021] using DICE-style technique optimizes policy by calculating the stationary distribution of state-action pairs instead of value function and extracts the policy by importance-sample behavior cloning. CDT [Liu et al., 2023b] combines Decision Transformer with safety constraints and utilizes data augmentation based on Pareto frontiers to enhance the safety and adaptability of the Transformer. It also focuses on the model's capability to different cost thresholds. The work [Le et al., 2019] proposes a meta-algorithm, using Fitted Q Evaluation and Fitted Q Iteration as subroutines to evaluate safety constraints and learn policy respectively. [Chen et al., 2022] analyzes the information-theoretic sample complexity lower bound and proposes a near-optimal primal-dual learning algorithm under partial data coverage but without function approximation. The most related work [Hong et al., 2024] approaches the problem from the perspective of the Actor-Critic algorithm, analyzing the sample complexity to be O ( 1 (1 -γ ) 2 √ n ) under Slater's condition and full data coverage assumption.

Convex MDP: Convex MDP problem extends the scope of MDP by focusing on convex objective functions of stationary distribution, rather than an inner product between reward and stationary distribution. To address the challenge, [Zhang et al., 2020b] introduces a variational Monte Carlo gradient estimation algorithm, demonstrating convergence to the optimal policy across general utility functions. The work [Zahavy et al., 2021] utilizes Fenchel duality to cast convex MDPs as min-max "two-player" games, proposing a meta-algorithm that addresses various convex MDPs through distinct subroutines. The work [Geist et al., 2022] approaches convex MDPs from the Mean-Field Game (MFG) perspective, establishing the equivalence between the optimal condition in convex MDPs and Nash equilibrium in MFGs. [Ying et al., 2023] studies the convex CMDP problem through a policy-based primal-dual algorithm and proves an O ( T -1 / 3 ) convergence rate in both optimality gap and constraint violation. The existing literature on convex MDPs primarily focuses on the online setting, while this paper specifically targets the offline scenario.

2 Problem Setup

2.1 Convex CMDP problem

We study a discounted constrained Markov decision process (CMDP), denoted by M = ( S , A , R, C, T, γ, µ 0 ) , where S is the state space (a finite set), A is the action space (a finite set), R : S × A → [0 , 1] and C : S × A → [0 , 1] are reward and cost functions, respectively, T : S × A → ∆( S ) is the transition probability kernel, where ∆( · ) denotes the probability simplex, γ ∈ [0 , 1) denotes the discount factor, and µ 0 ∈ ∆( S ) is the initial state distribution. We define a policy π : S → ∆( A ) as a probability mapping from states to actions. At time slot t , the agent observes state s t and takes action a t according to the policy π . For a policy π, we define its discounted state-action occupancy measure d π as follows

where P π ( s t , a t ) is the probability of state-action pair ( s, a ) being visited at time slot t under the policy π . Further, we let d π ( s ) = ∑ a ∈ A d π ( s, a ) be the discounted state occupancy measure.

Given the occupancy measure above, we can formulate the following convex CMDP problem

where f and g are both convex functions, and τ is the cost threshold. We present a single safety constraint for simple exposition, and our result can be readily generalized to the setting with multiple constraints. The set K is a probability simplex that satisfies global balance equations of the underlying Markov process

This set can also be written as a compact matrix form that

where 1 |A| = [1 , 1 , · · · , 1] ⊤ ∈ R |A| and P ∈ R |S|×|S||A| is the transition probability matrix.

The convex CMDP problem in (1) is quite general to capture various learning scenarios, including apprenticeship learning [Abbeel and Ng, 2004], standard CMDPs [Altman, 2021], pure exploration [Hazan et al., 2019], and inverse reinforcement learning in contextual MDPs [Belogolovsky et al., 2021]. Take the standard CMDP as an example, the agent aims to find a policy that maximizes cumulative reward while satisfying the safety constraints [Altman, 2021], which can be written as

where f and g are linear functions w.r.t. the state-action occupancy measure.

Take safety-aware apprenticeship learning [Zhou and Li, 2018] for another example, the agent aims to mimic the expert's demonstrations while avoiding the unsafe states in H , which can be written as

where d e represents the empirical distribution of the expert's demonstration and F ( · , · ) is the convex distance function, such as KL-divergence or total variation distance [Zhang et al., 2020a].

However, solving these problems in the online setting can be time-consuming, costly, and potentially dangerous in safety-critical contexts. In contrast, by leveraging historical data, offline RL offers a promising avenue for developing safe and effective algorithms, as introduced next.

2.2 Offline Reinforcement Learning

In offline RL, we cannot interact with the environment and only have access to dataset with a finite number of samples. Let D = ( s i , a i , r i , c i ) n i =1 be a collected offline dataset where we assume all the pairs ( s i , a i ) n i =1 are generated independently and identically distributed ( i.i.d. ) from data distribution µ ( s, a ) induced by a behavior policy π µ . Let n D ( s, a ) represents the number of occurrences of the state-action pair ( s, a ) in the offline dataset D , then µ D ( s, a ) = n D ( s, a ) /n is an empirical version of µ ( s, a ) . In offline RL, a major challenge is distribution shift, which measures the mismatch between data distribution and occupancy measure induced by candidate policies. To quantify the distribution shift, we make the following π ∗ concentrability assumption that refers to partial data coverage,

Assumption 1 ( π ∗ -concentrability) . Let π ∗ be the optimal policy to problem (1) , there exists constant C π ∗ > 0 such that d π ∗ ( s, a ) /µ ( s, a ) ≤ C π ∗ , for all s ∈ S , a ∈ A .

This assumption controls the distribution shift between offline data distribution µ and the occupancy measure d π ∗ induced by optimal policy π ∗ . Specifically, the partial data coverage assumption indicates the offline dataset D should cover state-action pairs visited by the optimal policy π ∗ . Unlike the common full data coverage assumption in previous works that the dataset D should encompass data visited by all policies [Chen and Jiang, 2019, Hong et al., 2024, Le et al., 2019], our assumption is considerably more relaxed. Furthermore, in the field of safe RL, partial coverage assumption also offers significant advantages, as the full coverage implies the behavior policy needs to visit every dangerous state and action space, which is obviously impractical.

Beyond using assumptions to limit distribution shift in offline RL, we consider Marginalized Importance Weight (MIW), a method widely used in the existing literature [Hong et al., 2024, Ozdaglar et al., 2023, Zhan et al., 2022], to further address this challenge.

Definition 1 (Marginalized Importance Weight) . Given a policy π, let d be the occupancy measure induced by π. We define marginalized importance weight w : S × A → R + as w ( s, a ) = d ( s,a ) µ ( s,a ) , ∀ s ∈ S , a ∈ A .

Note w can be regarded as the density ratio between the normalized discounted occupancy measure and data distribution. Moreover, recall the definition of w and M in equation (2) , we define matrix K ∈ R |S|×|S||A| and K D ∈ R |S|×|S||A| as

where K D can be seen as an empirical version of K in dataset D and it is straightforward to verify Kw = Md . With these notations, we establish an equivalent formulation with problem (1) w.r.t.

MIW

where the operator ' · ' denotes the element-wise product of vectors. As we aim to establish the sample-efficient learning algorithms in large state-action spaces, we regard the importance weight w as a function, i.e. w : S × A → R + that belongs to the convex function class W in the following assumption.

Assumption 2 (Realizability) . We assume w ∗ ∈ W where w ∗ is the optimal solution to the problem (5) -(7) .

This assumption assumes the optimal solution w ∗ ( π ∗ ) can be realizable for a convex function class W . Note that the assumption of the convex function class W is reasonable and standard, which is also achievable with the convexification process in case the function class is non-convex. The convexification process is a common practice in the offline safe RL literature, particularly in the context of general function approximation [Le et al., 2019, Hong et al., 2024]. Now we consider W as a discrete function class for simplicity and it readily extends to continuous settings (in Remark 1). Further, we introduce a completeness assumption that is used to relax the key constraint in (6).

Assumption 3 (Completeness) . Let x w be within the function class X . Define a mapping ϕ : such that ϕ ( w ) ⊤ ( Kw -(1 -γ ) µ 0 ) = x ⊤ w ( Kw -(1 -γ ) µ 0 ) = ∥ Kw -(1 -γ ) µ 0 ∥ 1 . Then, we have ( W , X ) -completeness under the mapping ϕ , i.e. x w ∈ X for all w ∈ W .

Intuitively, the completeness assumption allows us to replace the computation of the l 1 -norm with a linear product and simplifies our subsequent analysis. Next, we introduce the standard boundness assumption in offline RL literature with general function approximation [Chen and Jiang, 2019, 2022, Le et al., 2019, Munos and Szepesvári, 2008].

Assumption 4 (Boundness of W and X ) . We assume function classes W and X are bounded, i.e. ∥ w ∥ ∞ ≤ B w , ∀ w ∈ W and ∥ x w ∥ ∞ ≤ B w , ∀ x w ∈ X .

Combining this assumption with Assumption 1 implies B w ≥ C π ∗ . Lastly, we impose a mild assumption for the reward and cost functions in the problem (1).

Assumption 5 (Lipschitz condition) . The functions f ( x ) and g ( x ) are convex and satisfy the Lipschitz condition, where there exist constants L f and L g such that for any x, y , the following inequalities hold | f ( x ) -f ( y ) | ≤ L f ∥ x -y ∥ and | g ( x ) -g ( y ) | ≤ L g ∥ x -y ∥ .

3 Algorithm Design and Main Results

Despite the practical importance of offline safe RL in real-world applications, there is still a lack of theoretical research on this topic. The earlier literature on this setting often lacks robust theoretical analysis [Lee et al., 2021, Liu et al., 2023b, Xu et al., 2022], and articles with theoretical analysis either yield unsatisfactory results or rely on strong assumptions [Chen et al., 2022, Hong et al., 2024, Le et al., 2019].

In this section, we propose a provable algorithmic framework and establish the first theoretical result in offline convex CMDP, to the best of our knowledge. Moreover, when reducing to standard offline CMDP, we achieve a sample complexity of O ( 1 (1 -γ ) √ n ) with general function approximation, which improves the current state-of-the-art result by a factor of 1 / (1 -γ ) .

3.1 Algorithm Design

Inspired by [Ozdaglar et al., 2023], we first introduce an empirical version of problem (5)-(7) in the offline setting by incorporating a suitable relaxed parameter into the safety constraint. Intuitively, the empirical version of the problem is "close" to the original problem when the dataset is large. This is the key observation for analyzing the convergence performance and safety violations. To solve

4

(5)-(7), we present its empirical and relaxed problem:

where µ D and K D are the empirical version of µ and K, respectively; ζ and κ are the relaxed hyperparameters for the validity constraint (Bellman equation) and safety constraint. Intuitively, the parameters of ζ and κ capture the 'uncertainty' induced by the offline dataset D in terms of distribution shift and safety concerns. The values of these parameters will be specified later and play an important role in our analysis. Next, we demonstrate that the mismatch of the objective and constraint violation bound depends on the 'uncertainty', which exhibits O (1 / √ n ) distance.

Note that all parameters in the optimization problem (8)-(10) can be determined from the offline dataset. When the state-action space is not large, this problem can be efficiently solved by convex optimization solvers. However, when the state-action space is large, and the function approximation is necessary (e.g., w is parameterized by a neural network), it would be quite challenging (if not impossible) to solve this problem. To address this challenge, we propose a primal-dual algorithm that is sample-efficient and computationally tractable to solve the problem iteratively. We first introduce the Lagrange function of problem (8)-(10),

where λ and ϕ are Lagrange multipliers.

Algorithm 1: P rimal-dual algorithm for O ffline C onvex C MDP (POCC)

1 Input : Dataset D = { ( s i , a i , r i , c i ) } n i =1 , the relaxed parameters κ , ζ , and the step size η = 1 √ K ;

3 for k = 1 , 2 , . . . , K do

2 Initialization : Choose any w 1 ∈ W and the Lagrangian multipliers λ 1 = 0 , ϕ 1 = 0 ;

where P W is the projection onto set W and [ · ] h l is the projection onto interval [ l, h ] .

6 Extract the policy π K with formula (12);

5 Compute the average w K = ∑ K i =1 w i ;

7 Output: Policy π K ;

Given the Lagrange function above, we introduce our algorithm called POCC (in Algorithm 1), which takes the offline dataset D as input and runs a primal-dual method on the estimated Lagrange function. Specifically, at each iteration k , POCC updates the importance weight by gradient descent and projects it back to function class W , then updates Lagrange multipliers of validity constraint (6) and safety constraint (7) respectively. After K steps, POCC returns an averaged w K , we can extract the corresponding policy π K based on the offline dataset D as follows

where the second equality means that if ∑ a ′ ∈A w K ( s, a ′ ) π µ ( a ′ | s ) = 0 we randomly choose an action for state s .

3.2 Theoretical Results

We present the theoretical results of our proposed approach in the following theorem.

Theorem 1 (Sample complexity of π K ) . Suppose Assumptions 1-7 hold. Denote π K as the corresponding policy induced by w K . Set the relaxed parameters ζ = 2 √ 2 B w √ n √ log |W||X| δ and

κ = √ 2 L g B w √ n √ log 2 |W| δ , and the step size η = 1 √ K . Let the constants υ = 1 1 -γ (4 B +4 L +2 ε and ι = 1 1 -γ ( B 2 +4 B +4 L + L 2 √ K + ε ) , we have, with at least 1 8 δ probability,

where J r ( π K ) = f ( d K ) and J c ( π K ) = g ( d K ) are objective and constraint performance of policy π respectively, B represents the distance between initial value w 1 of the iteration and optimal solution w D to problem (8) -(10) , ε ≥ 0 is a constant, L is the max Lipschitz constant of Lagrange function and K is the number of iterations.

Remark 1. We remark that Theorem 1 remains valid even when the function class W is a continuous set. In this case, the cardinality |W| can be replaced with the covering number of W , and the union bound can be applied to its ϵ -covering set. This adjustment also preserves the sample complexity order of O (1 / √ n ) , up to a constant dependent on ϵ . For further details on the extension from discrete W to a continuous set, please refer to [Le et al., 2019, Xie and Jiang, 2021].

Theorem 1 demonstrates that the convergence performance of our algorithm can be divided into two parts: O (1 / √ n ) + O (1 / √ K ) . Regarding the first term, except for the size of dataset, it mainly depends on the function class (searching space) log |W||X| and relaxed parameters ζ and κ that capture the 'uncertainty' for addressing the distribution shift and safety concern. Moreover, the second term O (1 / √ K ) connects with the error bound between w K and optimal solution w D , which decays at the rate of 1 / √ K when the number of iterations increases. It is worth noting that the convergence rate of policy π K is still O (1 / √ n ) when the iterative number is sufficient to satisfy K ≥ n . The algorithm is gradient-based and does not involve additional computations for solving the optimization problem (8)-(10). This suggests that we can improve the convergence performance w.r.t. K by employing more advanced primal-dual techniques to reduce the convergence rate of the second term, such as those discussed in [Yu and Neely, 2017] with a convergence rate of O (1 /K ) .

Unlike most of the previous work focuses on convex MDPs in online setting [Bai et al., 2023, Ying et al., 2023, Zahavy et al., 2021, Zhang et al., 2020b], Theorem 1 to the best of our knowledge, is the first provable result in offline convex MDPs. Moreover, compared to the best result O ( 1 (1 -γ ) 2 √ n ) in offline CMDP [Hong et al., 2024, Chen et al., 2022], our results achieve a sample complexity of O ( 1 (1 -γ ) √ n ) , which outperform the state-of-the-art by a factor of 1 / (1 -γ ) . Besides, our algorithm is appropriate in the scenario with large-scale state-action space due to the general function approximation for w while the work [Chen et al., 2022] targets the tabular setting; our result is more favorable in the safety applications compared to [Hong et al., 2024] because we replace the realizability of value function ν with a slightly stronger completeness assumption but reduce the data coverage from full to partial . As stated in Assumption 1, the full data coverage not only implies access to a highly exploratory dataset but also is impractical for offline safe RL, as it assumes behavior policy needs to visit every dangerous state-action pair. Finally, we want to comment that the previous work all focuses on the standard RL, whose objective function is linear, while our algorithm is general enough to tackle the convex MDPs.

Remark 2. In the theorem, we assume prior knowledge of the behavior policy π µ for the sake of exposition. However, in practice, it is often challenging to know the behavior policy in advance, as we typically only have access to the offline dataset. The most popular approach to tackle this challenge is behavior clone. It posits that ˆ π µ can be estimated as ˆ π µ ( a | s ) = n ( s,a ) n ( s ) , where n ( s, a ) denotes the number of the occurrences of the state-action pair ( s, a ) in the offline dataset. We employed this estimation method in the experiments and results demonstrate its effectiveness.

4 Theoretical Analysis

In this section, we present a sketch of the proof of Theorem 1. We focus on illustrating the analysis of the constraint violation, and the convergence of the objective follows similar steps. The detailed proof can be found in Appendix A.

We first decompose the bound into different major terms and then sutdy them individually

The first equality holds due to the definition: J c ( π K ) = g ( d π K ) , where d π K represents the occupancy measure induced by returned policy π K . Recall the definition w ( s, a ) · µ ( s, a ) = d ( s, a ) , we then have g ( d K ) = g ( µ · w K ) . It is worth noting that the decomposition is intuitive since terms I and II relates to the idea of constructing 'uncertainty' parameter for the offline dataset and term III mainly depends on the convergence of primal-dual method.

Specifically, term I is the distance between g ( d K ) and g ( d π K ) . It represents the error that we rectify the unnormalized occupancy measure d K , which violates the validity constraint (6), to a satisfying one d π K . The term II illustrates the error incurred when applying the calculated w K from offline dataset D to the real environment µ , which depends on the sample size of the dataset. The term III is related to the relaxed parameter κ in safety constraint (7) and the distance between returned w K and optimal solution w D . Next, we present the following lemmas to bound these terms.

Lemma 1. Suppose Assumptions 1-4 hold, we have, with probability at least 1 -2 δ ,

Lemma 2. Suppose Assumptions 1-4 hold. For w K ∈ W , we have, with probability at least 1 -δ ,

Lemma 3. Suppose Assumptions 1-7 hold. For w K ∈ W , we have

Combining the above lemmas, we have, with at least 1 -3 δ probability,

5 Experiments

This section aims to justify the effectiveness of our proposed framework through numerical experiments. We test a practical version of Algorithm 1, where we replace the gradient-type update with the 'Adam'-type update (the detailed algorithm can be found in Algorithm 2 in the appendix). We test our algorithm to two specific offline convex CMDPs: 1) Safe imitation learning and 2) standard offline CMDP. Our objective is to address the following questions: (i) Are the experimental results consistent with our theory? (ii) how does the data quality affect the performance of our algorithm? The additional details can be found in the Appendix B.

5.1 Safe Imitation Learning

To showcase the generality of our algorithm, we choose imitation learning as a user case of convex MDPs and conduct the experiments in a maze environment. We design the environment, as illustrated

in Figure 2, modified from [Geist et al., 2022]. The problem of (safe) imitation learning can be formulated as F ( d ) = KL ( d || d E ) , where d E represents the stationary distribution of an expert. The environment is deterministic; agent has four actions (left, down, right, up); moving towards the wall (white) and the boundary does not change the state; the goal is to learn from the expert demonstrations (yellow) under safety constraints.

We collect data by expert demonstrations in (a) but randomly remove 25% states. We intend for the algorithm to learn to fill in the gaps using its inherent properties and function approximation, as emphasized in our theory. We are presenting two sets of results: one that considers safety constraints with a cost threshold of 0, and another that does not consider safety constraints. It's important to note that we cannot simply take the stationary distribution of the dataset as our final result. This is due to several reasons, including the fact that expert demonstrations are incomplete (25% of states are removed), simple replacement leads to poor results, and it's not suitable when considering the safety of the agents.

0

2

4

6

8

10

0

2

4

6

8

10

0

2

4

6

8

10

0

2

4

6

8

10

3

4

5

6

7

0

2

4

6

8

10

0

2

4

6

8

10

0

2

4

6

8

10

0

2

4

6

8

10

2

3

4

5

6

7

T

Our results are presented in Figure 2. In Figure (2b), it is demonstrated that our policy can accurately replicate the expert distribution using function approximation and completely bridge the gap required by the expert. Additionally, when taking into account the safety constraints, which involve setting the cost in the entire top-right corners, it is evident that our policy avoids states with cost and behaves appropriately, aligning with our theoretical framework.

5.2 Offline CMDP

We consider an 8x8 grid world environment FrozenLake, with the initial state being the top-left grid. The agent has four actions: N (north), S (south), E (east), and W (west). The primary objective is to reach the goal while avoiding all holes. The game terminates if the agent achieves the goal within 25 steps. A reward of 1 is obtained when the agent achieves the goal, and the main cost function assigns a cost of 1 if the agent falls into a hole and 0 otherwise. The diagram of the environment has been presented in Figure 1 in the introduction.

Initially, we simulate a mixture of different percentages of optimal and uniform policies to collect the offline dataset. We employ various behavior policies π D , running 200 trajectories to collect the offline dataset D , with each trajectory having a maximum of 50 time steps to ensure that the optimal goal is included in the dataset.

Additionally, we increase the difficulty of the environment compared to classical FrozenLake, such that if the agent falls into a hole, it can also come out in the next step. This implies that our cost constraint influences the training. We set the cost threshold as 0 here, which means that the agent is not supposed to incur any cost. Note that a higher percentage with a uniform policy indicates that the problem becomes more difficult. We set the discount factor as γ = 0 . 99 , ζ = 0 . 1 , and κ in our algorithm. Furthermore, we encode the environment with one-hot encoding and employ a more practical algorithm. We refer to w as a single hidden-layer neural network which we describe in Algorithm 2 in Appendix B. We set the learning rate of 10 -5 for w and 10 -4 for Lagrange multipliers.

In remark (), we have stated that there are two approaches when facing with the scenario that the behavior policy is unknown, and here we choose to use the behavior clone method that we will estimate the behavior policy through the offline dataset. We compare our algorithm with COptiDICE

[Lee et al., 2021], which is a well-acknowledged baseline algorithm in the offline safe RL literature [Hong et al., 2024, Liu et al., 2023b].

0

50

100

150

200

Training step

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

×

10 3

Reward

0

50

100

150

200

Training step

0

2

4

6

×

10 3

Cost

10 2

10 3

10 4

Datasize

0

.

4

0

.

6

0

.

8

1

.

0

Reward

10 2

10 3

10 4

Datasize

0

2

4

6

8

Cost

p

= 0

.

75

p

= 0

.

5

p

= 0

.

25

p

= 0

.

0

Ours

COptiDICE

Denote p as the percentage of optimal policy within the behavior policy. We evaluate the algorithm with different behavior policies p = { 0 . 75 , 0 . 5 , 0 . 25 , 0 } and dataset sizes. We present the results in Figure 3. In the first two figures, it can be seen that our algorithm can consistently find the optimal path even with completely random data. Conversely, COptiDICE behaves well when the proportion of optimal policy is 0.75, but cannot even learn a logical and safe policy when the majority of the data in dataset is random and constraint violated. In the last two figures, we test the performance of algorithms in different sizes of dataset, where p = 0 . 5 in behavior policy. The results demonstrate that our algorithm can find a safe and optimal path as the dataset size increases. In contrast, the results from the COptiDICE algorithm show high variance, where it can only find a safe path in about 50% runs.

In summary, our algorithm performs well across various behavior policies and dataset sizes, which is consistent with our theoretical results and assumptions.

6 Conclusions

In this work, we consider convex CMDPs in the offline setting. We propose a sample efficient RL approach that addresses the challenges in offline convex CMDPs. We theoretically prove that we can suffer O (1 / √ n ) sample complexity in both performance and violation bound with general function approximation under mild data coverage assumption, which is the first result in offline convex MDPs as best of our knowledge and surpasses the state-of-the-art result in offline safe RL by a factor of 1 / (1 -γ ) . Experimental studies further demonstrate the effectiveness and generality of our framework.

References

Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning , page 1, 2004.

David Abel, Will Dabney, Anna Harutyunyan, Mark K Ho, Michael Littman, Doina Precup, and Satinder Singh. On the expressivity of markov reward. Advances in Neural Information Processing Systems , 34:7799-7812, 2021.

Eitan Altman. Constrained Markov decision processes . Routledge, 2021.

Qinbo Bai, Amrit Singh Bedi, Mridul Agarwal, Alec Koppel, and Vaneet Aggarwal. Achieving zero constraint violation for concave utility constrained reinforcement learning via primal-dual approach. Journal of Artificial Intelligence Research , 78:975-1016, 2023.

Stav Belogolovsky, Philip Korsunsky, Shie Mannor, Chen Tessler, and Tom Zahavy. Inverse reinforcement learning in contextual mdps. Machine Learning , 110(9):2295-2334, 2021.

Dimitri P Bertsekas. Nonlinear programming. Journal of the Operational Research Society , 48(3): 334-334, 1997.

Fan Chen, Junyu Zhang, and Zaiwen Wen. A near-optimal primal-dual method for off-policy learning in cmdp. Advances in Neural Information Processing Systems , 35:10521-10532, 2022.

Jinglin Chen and Nan Jiang. Information-theoretic considerations in batch reinforcement learning. In International Conference on Machine Learning , pages 1042-1051. PMLR, 2019.

Jinglin Chen and Nan Jiang. Offline reinforcement learning under value and density-ratio realizability: the power of gaps. In Uncertainty in Artificial Intelligence , pages 378-388. PMLR, 2022.

Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning , pages 2052-2062. PMLR, 2019.

Matthieu Geist, Julien Pérolat, Mathieu Laurière, Romuald Elie, Sarah Perrin, Oliver Bachem, Rémi Munos, and Olivier Pietquin. Concave utility reinforcement learning: The mean-field game viewpoint. In Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems , pages 489-497, 2022.

Elad Hazan, Sham Kakade, Karan Singh, and Abby Van Soest. Provably efficient maximum entropy exploration. In International Conference on Machine Learning , pages 2681-2691. PMLR, 2019.

Jean-Baptiste Hiriart-Urruty and Claude Lemaréchal. Convex analysis and minimization algorithms I: Fundamentals , volume 305. Springer science & business media, 1996.

Kihyuk Hong, Yuhang Li, and Ambuj Tewari. A primal-dual-critic algorithm for offline constrained reinforcement learning. In International Conference on Artificial Intelligence and Statistics , pages 280-288. PMLR, 2024.

BRavi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yogamani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions on Intelligent Transportation Systems , 23(6):4909-4926, 2021.

Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169 , 2021.

Hoang Le, Cameron Voloshin, and Yisong Yue. Batch policy learning under constraints. In International Conference on Machine Learning , pages 3703-3712. PMLR, 2019.

Jongmin Lee, Cosmin Paduraru, Daniel J Mankowitz, Nicolas Heess, Doina Precup, Kee-Eung Kim, and Arthur Guez. Coptidice: Offline constrained reinforcement learning via stationary distribution correction estimation. In International Conference on Learning Representations , 2021.

Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. Journal of Machine Learning Research , 17(39):1-40, 2016.

Peng Liao, Zhengling Qi, Runzhe Wan, Predrag Klasnja, and Susan A Murphy. Batch policy learning in average reward markov decision processes. Annals of statistics , 50(6):3364, 2022.

Boyi Liu, Qi Cai, Zhuoran Yang, and Zhaoran Wang. Neural trust region/proximal policy optimization attains globally optimal policy. Advances in neural information processing systems , 32, 2019.

Zuxin Liu, Zijian Guo, Haohong Lin, Yihang Yao, Jiacheng Zhu, Zhepeng Cen, Hanjiang Hu, Wenhao Yu, Tingnan Zhang, Jie Tan, et al. Datasets and benchmarks for offline safe reinforcement learning. arXiv preprint arXiv:2306.09303 , 2023a.

Zuxin Liu, Zijian Guo, Yihang Yao, Zhepeng Cen, Wenhao Yu, Tingnan Zhang, and Ding Zhao. Constrained decision transformer for offline safe reinforcement learning. In International Conference on Machine Learning , pages 21611-21630. PMLR, 2023b.

Rémi Munos and Csaba Szepesvári. Finite-time bounds for fitted value iteration. Journal of Machine Learning Research , 9(5), 2008.

Mirco Mutti, Riccardo De Santi, Piersilvio De Bartolomeis, and Marcello Restelli. Convex reinforcement learning in finite trials. Journal of Machine Learning Research , 24(250):1-42, 2023.

Angelia Nedi´ c and Asuman Ozdaglar. Subgradient methods for saddle-point problems. Journal of optimization theory and applications , 142:205-228, 2009.

Asuman E Ozdaglar, Sarath Pattathil, Jiawei Zhang, and Kaiqing Zhang. Revisiting the linearprogramming framework for offline rl with general function approximation. In International Conference on Machine Learning , pages 26769-26791. PMLR, 2023.

Tom Le Paine, Cosmin Paduraru, Andrea Michi, Caglar Gulcehre, Konrad Zolna, Alexander Novikov, Ziyu Wang, and Nando de Freitas. Hyperparameter selection for offline reinforcement learning. arXiv preprint arXiv:2007.09055 , 2020.

Paria Rashidinejad, Banghua Zhu, Cong Ma, Jiantao Jiao, and Stuart Russell. Bridging offline reinforcement learning and imitation learning: A tale of pessimism. In Advances Neural Information Processing Systems (NeurIPS) , volume 34, pages 11702-11716, 2021.

Masatoshi Uehara and Wen Sun. Pessimistic model-based offline reinforcement learning under partial coverage. arXiv preprint arXiv:2107.06226 , 2021.

Lingxiao Wang, Qi Cai, Zhuoran Yang, and Zhaoran Wang. Neural policy gradient methods: Global optimality and rates of convergence. arXiv preprint arXiv:1909.01150 , 2019.

Tengyang Xie and Nan Jiang. Batch value-function approximation with only realizability. In Int. Conf. Machine Learning (ICML) , pages 11404-11413. PMLR, 2021.

Tengyang Xie, Ching-An Cheng, Nan Jiang, Paul Mineiro, and Alekh Agarwal. Bellman-consistent pessimism for offline reinforcement learning. Advances in neural information processing systems , 34:6683-6694, 2021.

Haoran Xu, Xianyuan Zhan, and Xiangyu Zhu. Constraints penalized q-learning for safe offline reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 8753-8760, 2022.

Donghao Ying, Mengzi Amy Guo, Yuhao Ding, Javad Lavaei, and Zuo-Jun Shen. Policy-based primal-dual methods for convex constrained markov decision processes. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 10963-10971, 2023.

Chao Yu, Jiming Liu, Shamim Nemati, and Guosheng Yin. Reinforcement learning in healthcare: A survey. ACM Computing Surveys (CSUR) , 55(1):1-36, 2021.

Hao Yu and Michael J Neely. A simple parallel algorithm with an o(1/t) convergence rate for general convex programs. SIAM Journal on Optimization , 27(2):759-783, 2017.

Tom Zahavy, Brendan O'Donoghue, Guillaume Desjardins, and Satinder Singh. Reward is enough for convex mdps. Advances in Neural Information Processing Systems , 34:25746-25759, 2021.

Wenhao Zhan, Baihe Huang, Audrey Huang, Nan Jiang, and Jason Lee. Offline reinforcement learning with realizability and single-policy concentrability. In Conference on Learning Theory , pages 2730-2775. PMLR, 2022.

Junyu Zhang, Amrit Singh Bedi, Mengdi Wang, and Alec Koppel. Cautious reinforcement learning via distributional risk in the dual domain. arXiv preprint arXiv:2002.12475 , 2020a.

Junyu Zhang, Alec Koppel, Amrit Singh Bedi, Csaba Szepesvari, and Mengdi Wang. Variational policy gradient method for reinforcement learning with general utilities. Advances in Neural Information Processing Systems , 33:4572-4583, 2020b.

Yinan Zheng, Jianxiong Li, Dongjie Yu, Yujie Yang, Shengbo Eben Li, Xianyuan Zhan, and Jingjing Liu. Feasibility-guided safe offline reinforcement learning. In The Twelfth International Conference on Learning Representations , 2023.

Weichao Zhou and Wenchao Li. Safety-aware apprenticeship learning. In Computer Aided Verification: 30th International Conference, CAV 2018, Held as Part of the Federated Logic Conference, FloC 2018, Oxford, UK, July 14-17, 2018, Proceedings, Part I 30 , pages 662-680. Springer, 2018.

Hanlin Zhu, Paria Rashidinejad, and Jiantao Jiao. Importance weighted actor-critic for optimal conservative offline reinforcement learning. arXiv preprint arXiv:2301.12714 , 2023.

A Proof of Theorem 1

In this section, we give the complete proof of Theorem 1. Specifically, we bound the regret first and constraint violation second. Hence, we first present some auxiliary lemmas and combine them to prove the final theorem.

Proof of objective bound

We also decompose the expression by adding and subtracting corresponding terms and prove them individually.

Next, we will prove these items individually.

Lemma 4. ∀ w ∗ ∈ W , w ∗ satisfies constraint functions (6) and (7) with at least 1 -2 δ probability.

Proof. First, we focus on the safety constraint (7). We prove it with Hoeffding inequality. Recall the Hoeffding inequality:

Take µ D · w as the random variable, we have

Since µ ( s, a ) ∈ [0 , 1] , and meanwhile w lies in [ -B w , B w ] , we have ( µ D -µ ) · w lies in interval [ -B w , B w ] . To satisfy the safety constraint function:

We set t = √ 2 B w √ n √ log 2 |W| δ . Combine with the Hoeffding inequality, we have

For the constraint function, recall we suppose that it satisfies the Lipschitz condition,

Before proceeding to the following derivation, we first state a fact:

The above equation means that if a variable z is greater than t with probability less than or equal to δ , then any variable such as y that is lower than that variable must also be lower than t with probability less than or equal to δ .

We get

Note the fact that ( µ D · w -µ · w ) is obviously lower than | µ D · w -µ · w | , then

Appendix

Take the union bound for w ∈ W , then

Then for all w ∗ ∈ W ,

Because w ∗ is the optimal solution for problem (5)-(7) and we suppose the assumption of realizability is set up, then in equation (7), w ∗ satisfies

Combing with equation (19), we get

So we have, with at least 1 -δ probability,

Next, we focus on the validity constraint function (6).

Here, we take x ⊤ ( K D -K ) w as the random variable and note the fact that it lies in the interval [ -2 B w , 2 B w ] . Use the Hoeffding inequality we have that ∀ x ∈ B ,

Set t = 2 B w √ 2 log( |W||X| /δ ) √ n , the inequality becomes the form as:

Take the union bound for all w and x , we have

So for all w ∗ ∈ W ,

Note the fact ∥ Kw ∗ -(1 -γ ) µ 0 ∥ = 0 , we have the inequality for all w ∗ ∈ W

That is, with at least 1 -δ probability,

Take the union bound for equation (23) and (29) completes the proof.

Next, we show that f ( µ D · w D ) is close to f ( µ · w ∗ ) .

Then the empirical covering number n ( ϵ, W ) is defined as the size of the smallest ϵ -cover. Intuitively, the ϵ cover set can represents the original set in the sense of ϵ . This definition is very useful when the original set is continuous and we want to take the union bound of it. By this ϵ cover we can take the union bound of the continuous set and the distance of them is measured by ϵ .

Lemma 5. We have with 1 -3 δ probability.

Proof. From Lemma 4, we have with probability 1 -δ .

Proof. This is easy to prove as we have a general result for all w ∈ W in equation (31). Here we take w K ∈ W , which completes the proof.

with probability at least 1 -2 δ . Now, we use Hoeffding inequality to bound f ( µ D · w ) -f ( µ · w ) . Take ( µ D -µ ) · w as the random variable and note it lies in interval [ -B w , B w ] , then we have

Set t = B w √ 2 log(2 |W| /δ ) √ n , the inequality becomes

Use the Lipschitz condition and union bound of w , we have

Choose w ∗ ∈ W and combine with the inequality f ( µ D · w D ) ≤ f ( µ D · w ∗ ) , we get the final result

with at least 1 -3 δ probability.

Next, we prove term II that f ( µ · w K ) is close to f ( µ D · w K ) .

Lemma 6. We have

Note that w K violates the validity constraint in problem (5). This means that the calculated result may not satisfy the quality of the occupancy measure. To find the relation, we utilize the analogous lemmas in [Ozdaglar et al., 2023] to get the error bound between validity constraint violation and absolute objective difference.

Lemma 7. Let d K be a variable solved by Program (8) that violates the validity constraint (6) and π K is the policy induced by d K . We can get that

with at least 1 -δ probability.

Proof. Define the marginalized occupancy measure as ˆ d K ( s ) = ∑ a ∈A d K ( s, a ) and ˆ d π K ( s ) = ∑ a ∈A d π K ( s, a ) . Then, we can write

Let P |S||A| π ∈ R be the state transition matrix, i.e.,

Also, we define the matrix G π = Diag ( π d ( · | s 1 ) , π d ( · | s 2 ) , · · · , π d ( · | s |S| )) ∈ R |S||A|×|S| , and notice the fact that MG π = I -γP π . Now, since d π satisfies the constraints in Problem, we have Md π = (1 -γ ) µ 0 . This implies:

Here the last inequality is because γ ∥ P π ( ˆ d K -ˆ d π K ) ∥ 1 ≤ γ ∥ ˆ d K -ˆ d π K ∥ 1 , which follows from the fact that P π is a column stochastic matrix.

We have:

The first inequality holds because of the Lipschitz condition of function f . So combining the above two inequalities, we get

which follows the definition of density ratio w . Next, we give the bound of ∥ Kw K -(1 -γ ) µ 0 ∥ 1 . From inequality (26) above we have that

Then take w as w K and x as 1-norm, with at least 1 -δ we have where the first inequality is because of the triangle inequality and the last inequality follows the fact that w K is the optimal solution to problem (8) and the above inequality. Combining (40) and (38) completes the proof.

Now, we have the bound of term I and II. To further demonstrate the final bound, we also need to investigate the distance between the returned value w K by primal-dual method and optimal solution w D .

Our algorithm 1 is the classical type of primal-dual subgradient method and our analysis mostly refers to [Nedi´ c and Ozdaglar, 2009]. We put it here for the sake of completeness and add the theoretical analysis of our algorithm.

First, recall the Lagrange function and updated rules of the algorithm

For simplicity, we combine the Lagrange multipliers into one variable and call it ν , accordingly, we set the constraint g ( w ) ≤ 0 . Note that constraints are analogous and we would split them in the end. Now the updated rules become

where ν k ∈ M and note that we turn the interval of ν into a projection set which is an equivalent transformation. The Lagrange function is denoted as

Theorem 2 (Saddle-Point Theorem in [Bertsekas, 1997]) . The pair ( w D , ν D ) is a primal-dual optimal solution if and only if

where w D ∈ W and ν D ≥ 0 .

The analysis of convergence is based on this theorem and it is a standard result that characterizes the primal-dual optimal solutions as the saddle points of the Lagrange function[Nedi´ c and Ozdaglar, 2009].

Theorem 3. According to the updated rules, denote the solution to problem (8) as w K . Compared to the optimal solution w D , we have the upper bound for performance and constraint violation:

where η is the constant learning rate, w 1 is an initial state, ε ≥ 0 , K is the iteration step and L is the max subgradient of the Lagrange function. Furthermore, we denote, w ∗ = w D , f ( w K ) = f ( µ D · w K ) and f ∗ = f ( µ D · w D ) , we use the above notations for simplicity.

We first introduce some lemmas and prove the theorem step by step.

Lemma 8. Suppose that the sequence { w k } and { ν k } are generated by the updated rules. Then we have

Proof. We prove (1) and (2) is similar to (1) .

where the first equality comes from the update rule and the last inequality is because of the convexity of the Lagrange function.

Assumption 6 (Boundness of subgradient) . We assume that the subgradient of the Lagrange function is bounded, such as

where L is the max Lipschitz constant in Lagrange function.

Lemma 9. Set w k = 1 k ∑ k i =1 w k i and ν k = 1 k ∑ k i =1 ν k i , we have

Proof. We only prove (1) and equation (2) is similar to (1) . By using Lemma 8 we have

Sum it from 1 to k ,

And combine with 1 k ∑ k i =1 L ( w,ν i ) ≥ L ( w,ν k ) which is because of Jensen's inequality, we have

Assumption 7 (Slater's condition) . There exists a vector ˆ w such that

we refer to ˆ w as a Slater's vector.

Note that Slater's condition is assumed in many primal-dual method literature [Chen et al., 2022, Hong et al., 2024, Nedi´ c and Ozdaglar, 2009].

Lemma 10. Under Assumption 7, suppose q ∗ is the dual optimal solution and ¯ w is a vector that satisfies Slater's condition, we have

where γ = min 1 ≤ j ≤ m {-g j ( ˆ w ) } and m is the number of constraints.

The proof is referred to [Hiriart-Urruty and Lemaréchal, 1996]. The lemma above motivates the choice of dual set, that is

where ˜ q ≤ q ∗ and r ≥ 0 is a constant.

Proof. Now, we are ready to give the general proof of Theorem 3. Combine Lemma 8 and Assumption 6, we have

Moreover,

where the first equation is because of convexity of L ν and the last equation dues to the Slater's condition. We have, ∀ ν ∈ M and i ≥ 1

Then, sum over i = 1 , · · · , k ,

Because the above function is for all ν , then

Set

Hence, if s + = 0 , equation (52) holds. If s + = 0 , define vector:

̸

Combine Lemma 10, we have which implies that for ˆ ν ∈ M ,

Recall the definition of s , we have

Then,

Until now, we assume the dual set is appropriately chosen in every iteration and we should quantify it. For the dual set

and the result in (57), We should choose

which is

So the dual set in every iteration is

which implies

For simplicity, we take the first term as a constant which doesn't affect our result:

The result of (1) in Theorem 3 completes.

Next, we give the bound of the objective value. From the definition of the Lagrange function,

Then,

Note that

Set ν = 0 and combine Lemma 8, we have

Then

Sum over i = 0 , · · · , k -1 ,

which completes the proof of Theorem 3.

Then we have the all results to prove the regret bound in Theorem 1.

Theorem 3 indicates that after K iterations the objective and violation bound for problem (8) are

where we set the initial Lagrange values equal to 0, the constant step size η = 1 √ K and B is the distance between w 1 and optimal solution w D .

And recall Lemma 7, we have with at least 1 -2 δ probability,

Then, we have, with at least 1 -δ probability, the bound is,

We get the final result,

where we set ι = 4 B +4 L + B 2 + L 2 √ K + ε

Proof of violation bound

In the main text, we split the violation bound into three parts and list the relevant lemmas. Here we give the proofs for these lemmas and theorem.

Proof of Lemma 1

Proof. This proof is quite similar to Lemma 7 since we always want to establish connections between the required function and validity error bound. The main difference is the objective function and constraint function. So we keep the first half conclusion of the Lemma 7 which is

Then for the safety constraint function, we have

where the first inequality is because Lipschitz condition of the safety constraint function. Also by combining the above functions, we have

And in Lemma 7 we have already established the bound of ∥ Kw K -(1 -γ ) µ 0 ∥ 1 , then with at least 1 -2 δ probability

which completes the proof.

Proof of Lemma 2

Proof. In this lemma, actually, we will prove that g ( µ · w K ) is close to g ( µ D · w K ) . We take ( µ -µ D ) · w as the random variable and it lies in [ -B w , B w ] . We have the Hoeffding inequality where we set t = √ 2 B w √ n √ log 2 |W| δ . Next, take the union bound for w and use the Lipschitz condition for g , we have

So for w K ∈ W ,

which means that with at least 1 -δ probability,

The proof is complete.

Proof of Lemma 3

Proof. This lemma is easy to be verified because we choose the relax parameter κ is equal to √ 2 L g B w √ n √ log 2 |W| δ and recall the result in Theorem 3 which completes the proof.

Thus, take the Lemma 1, 2 and 3 together, we have, with at least 1 -3 δ ,

where we set υ = 1 1 -γ (4 B +4 L +2 ε ) which completes the proof.

B Experiments

Weintroduce a practical version of our primal-dual algorithm, following the key structure in Algorithm 1. We utilize this algorithm to conduct various experiments in this paper.

Algorithm 2: Practical version of POCC

1 Input : Dataset D = { ( s i , a i , r i , c i ) } n i =1 , cost threshold τ , learning rate η λ , η w , η ϕ , relaxed parameter ζ , κ .; 2 Initialize network of importance weight w ψ ; 3 for k = 1 , 2 , . . . , K do 4 Jointly optimize: w k +1 ψ = Adam ( w k ψ -η ϕ ∇ L ( w ψ , λ k , ϕ k ) ) , λ k +1 = Adam ( λ k -η λ ∇ L ( w k ψ , λ, ϕ k ) , ϕ k +1 = Adam ( ϕ k -η τ ∇ L ( w k ψ , λ k , ϕ ) ) . 5 Compute the importance weight: ∀ ( s, a ) , w ( s, a ) = w ψ ( s, a ) ; 6 Extract the policy: π ( a | s ) = w ψ ( s,a ) π µ ( a | s ) ∑ a ′ ∈ A w ψ ( s,a ′ ) π µ ( a ′ | s ) ;

7 Output: Policy π ;

Hyperparameters In the experiments of maze and FrozenLake environments, we set the discount factor as γ = 0 . 99 and the cost threshold as τ = 0 , indicating that the agent is not supposed to incur any cost. For our algorithm 2 and COptiDICE [Lee et al., 2021], we perform a grid search on the set { 0 . 00001 , 0 . 00005 , 0 . 0001 , 0 . 0005 , 0 . 001 , 0 . 005 } to determine the learning rate for parameters. We ultimately select a learning rate of 0 . 00001 for the neural network and a Lagrange scalar learning

rate of 0.0001. We set ζ = 0 . 1 and κ = 0 . 001 in our algorithm, and we set α = 0 . 5 in COptiDICE. In both environments, we run the algorithm for K = 100000 iterations to ensure model convergence. Additionally, we use fully-connected neural networks with a single hidden layer of width 64.

Continuous environments We also run a list of experiments in SafetyGym with offline datasets provided by [Liu et al., 2023a] and compare with comprehensive baselines (e.g., CPQ in [R5], PDCA in [R4], CoptiDICE in [R6], and BEAR-Lagrangian in [R7], [R8]). Note that only PDCA and our algorithm provide theoretical results. In these experiments, to deal with the continuous state-action space, we use the fully connected single hidden-layer neural network of width 128 to represent w . We summarize the evaluation results in the following table. All the rewards and costs are normalized and the cost threshold is 1. Each value is averaged over 20 evaluation episodes and 3 random seeds.

-

Compute setting We run the experiments with NVIDIA GeForce RTX 3080 Ti 8-Core Processor.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Claims in the abstract and introduction accurately reflect our paper's contributions.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We include the limitations in our assumptions.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide a proof sketch of our theorem in the main paper and complete the full proof in the supplemental material.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We disclose all the information needed to reproduce the main experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide the data and code.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We specify the experimental settings in the supplemental material.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report error bars correctly in the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide the sufficient information on the computer resources.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in this paper conform with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There is no societal impact of the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite the original paper and abide by the license.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper formulates offline convex MDPs under safety constraints and proposes a linear programming-based primal-dual algorithm for solving it. The authors make a partial data coverage assumption yet achieve a sample complexity of $\\mathcal{O}(\\frac{1}{(1-\\gamma)\\sqrt{n}})$ while the current SOTA is $\\mathcal{O}(\\frac{1}{(1-\\gamma)^2\\sqrt{n}})$. This paper also conducted the empirical evaluation and showed that the proposed algorithm achieves better performance in terms of reward and safety.\n\nSTRENGTHS:\n- Offline convex CMDPs are promising formulation that covers wide-ranging problems including safe imitation learning or standard offline CMDPs.\n- The theoretical results are strong. The sample complexity is improved by $1-\\gamma$ compared to the SOTA, which is an important contribution to the offline safe RL community. The assumptions the authors make are 1) concentrability of the optimal policy, 2) realizability, 3) completeness, 4) Boundness of $\\mathcal{W}$ and $\\mathcal{X}$, and 5) Lipschitz continuity of the $f$ and $g$. Though the number of assumptions is large and each assumption is strong, all of them are standard assumptions in previous work and we understand that it is almost impossible to prove sample complexity without them.\n- While this paper is highly theoretical, the authors also provide empirical results. I think it is nice to evaluate their algorithm in two settings: safe imitation learning and offline safe RL.\n\nWEAKNESSES:\n- While I admit that the main contribution of this paper is theory, I consider that the empirical experiments are not fully conducted. Both environments are toy problems, and there is no baseline method in 5.1. In 5.2, the benchmark task is the frozen lake, which is a much easier task than previous work tried to solve. I know COptiDICE is a well-known baseline algorithm, but there are many more algorithms that perform better than COptiDICE.\n- The authors say \"We include the limitations in our assumptions\", but I do not think that the limitations of this paper exist except for the assumptions. I would recommend that the authors discuss the limitations such as scalability, empirical performance in more complicated tasks, etc.\n\n**Minor Comments and Typos**\n- Line 45: The following sentence seems weird to me. Should this be \"We formulate ...\"?\n    - \"We for the first study the offline.\"\n- Figure 3: (a), (b), ..., (d) do not exist in Figure 3 while they are mentioned in the caption.\n\nQUESTIONS:\n- In lines 229 - 238, the authors claim as follows. However, the empirical results are provided for grid-world environments. The size of the grid world is rather small (i.e., $8 \\times 8$). Could you tell me why the authors used such small environments? I personally think that the following claim is not fully supported.\n> Besides, our algorithm is appropriate in the scenario with large-scale state-action space\n- I think I could follow the theorem and proof, but could you tell me what enables the authors to achieve the SOTA sample complexity $\\mathcal{O}(\\frac{1}{(1-\\gamma)\\sqrt{n}})$? What is the biggest reason why the author could improve the current SOTA by a factor of $1/(1-\\gamma)$?\n\nLIMITATIONS:\nThough the authors discuss the limitations regarding assumptions, I think there are other limitations such as the applicability to more complicated tasks with continuous state-action spaces, computational complexity, etc.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-13"}

comments: SUMMARY:
This paper investigates batch RL with safety constraints and function approximation, which is a question of both theoretical and practical importance.

STRENGTHS:
The assumptions considered in this paper are less restrictive compared to previous works. In particular, most previous works consider linear objective, and this paper relaxes linearity to convexity, which makes the results apply to a broader class of learning problems (e.g. batch RL with entropy regularization). It is also good to see the experimental results.

WEAKNESSES:
(1) The presentation is a bit messy. The class X is not well-explained. The Lipschitz constant L appeared in Theorem 1 is not defined in the main text. Is L bounded by L_f, L_g and other parameters?

(2) There is an unstated assumption: the convergence of the proposed algorithm actually relies on the convexity of W (Lemma 8).  

(3) The upper bounds are stated in terms of log|W|, the log-cardinality of the function class W. However, it may be problematic to assume W is finite: if W is both convex and finite, then W must be a singleton, and the results are vacuous. Therefore, the upper bounds should instead be stated in terms of the log covering number of W.

QUESTIONS:
Q1. Is there a relationship between the proposed primal-dual update rules and the actor-critic algorithms?

Q2. In assumption 4, it is required that all functions in W have a uniform upper bound B_w, which implies $B_w\geq C_\pi^\star $. It would be better to explicitly state that it requires prior knowledge on the concentrability constant.

Q3. The class X does not appear in the algorithm. Is it introduced purely for the sake of analysis?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-06-29

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel linear programming based primal-dual algorithm for convex MDPs which incorporates “uncertainty” parameters to improve data efficiency, while requiring only partial data coverage assumption. The authors provide theoretical results achieve a sample complexity of $O(1/((1-\\gamma)\\sqrt{n}))$ under general function approximation, improving the current state-of-the-art by a factor of $1/(1 − \\gamma)$, where $n$ is the number of data samples in an offline dataset, and $\\gamma$ is the discount factor. The authors also run experiments to validate their theoretical findings, which demonstrate the practical efficacy of their approach in achieving improved safety and learning efficiency in safe offline settings.\n\nSTRENGTHS:\n1.\tThe studied problem, i.e., safe offline RL in convex MDPs, is well-motivated and can be applied to autonomous driving, robotics, etc. \n2.\tThe authors design a novel linear programming based primal-dual algorithm for convex MDPs under only the partial coverage assumption, instead of the full coverage assumption. The authors also provide sample complexity for this algorithm, which improves the current result by a factor of $1/(1 − \\gamma)$.\n3.\tEmpirical evaluations are also presented to validate the practical efficacy of the proposed algorithm.\n\nWEAKNESSES:\n1.\tCan the authors give more explanations on Assumptions 2-4. Why do you introduce the set $\\mathcal{W}$ and $\\mathcal{X}$. Does the algorithm need to know these two sets in advance? If yes, what does it mean in practice?\n2.\tDoes the algorithm need to know the behavior policy $\\mu(a|s)$ in advance (in Eq. (12))? This is not a practical assumption.\n3.\tThe authors should give more comparison to the existing results for offline (linear) constrained RL, since (linear) constrained MDPs is an important example of convex MDPs.\n4.\tWhat are the technical challenges and novelty in the offline convex MDP (RL) problem, compared to the existing online convex MDP (RL) works?\n\nQUESTIONS:\nPlease see the weaknesses above.\n\nLIMITATIONS:\nPlease see the weaknesses above.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the offline safe reinforcement learning (RL) problem and proposes a primal-dual algorithm to solve this problem. The key contributions are that the paper focuses on the more general setting of convex Markov Decision Processes (convex MDPs), where the objective function is a convex utility function, rather than the standard linear rewards. The authors propose a primal-dual algorithm that can handle the offline safe RL problem under the convex MDP setting, crucially requiring only partial data coverage, a weaker assumption compared to previous work. The theoretical analysis provides the sample complexity of the proposed algorithm. Experimental results validate the theoretical findings and demonstrate the practical efficacy of the approach.\n\nSTRENGTHS:\nThis paper studies a very interesting and important topic in the RL community. The writing is clear and the results are well presented.\n\nWEAKNESSES:\nMy main concern with this paper is the limited technical novelty of the contributions. The proposed formulation (8)-(10) appears to be a direct extension of the setup in prior work [26], where the linear reward and cost functions are generalized to convex functions. The algorithm itself is a standard primal-dual method for solving convex optimization problems, rather than a new technical innovation.\n\nFurthermore, the experimental evaluation is quite limited in scope. The authors only compare their approach against a small number of baseline methods, and the experiments are run on a relatively small set of problem instances. This raises questions about the generalizability and practical significance of the empirical results.\n\nWhile the paper tackles an important and relevant problem in offline safe reinforcement learning, the technical contributions seem incremental given the prior work in this area.\n\nQUESTIONS:\n-- line 181, how large the dataset has to be in order for (8)-(10) to be close to (5)-(7)?\n\n-- When w is parametrized by a neural network, does the proposed formulation satisfy all the assumptions made?\n\nLIMITATIONS:\nsee weakness", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}