C2IQL: Constraint-Conditioned Implicit Q-learning for Safe Offline Reinforcement Learning
Authors: Zifan LIU
Xinran Li
Jun Zhang
Venue: ICML 2025 poster (2025)
Keywords: safe offline reinforcement learning
constrained implicit Q-learning
cost reconstruction
constraint-conditioned ability
Primary area: reinforcement_learning->batchoffline

Abstract: Safe offline reinforcement learning aims to develop policies that maximize cumulative rewards while satisfying safety constraints without the need for risky online interaction. However, existing methods often struggle with the out-of-distribution (OOD) problem, leading to potentially unsafe and suboptimal policies. To address this issue, we first propose Constrained Implicit Q-learning (CIQL), a novel algorithm designed to avoid the OOD problem. In particular, CIQL expands the implicit update of 
reward value functions to constrained settings and then estimates cost value functions under the same implicit policy. Despite its advantages, the further performance improvement of CIQL is still hindered by the inaccurate discounted approximations of constraints. Thus, we further propose Constraint-Conditioned Implicit Q-learning (C2IQL). Building upon CIQL, C2IQL employs a cost reconstruction model to derive non-discounted cumulative costs from discounted values and incorporates a flexible, constraint-conditioned mechanism to accommodate dynamic safety constraints. Experiment results on DSRL benchmarks demonstrate the superiority of C2IQL compared to baseline methods in achieving higher rewards while guaranteeing safety constraints under different threshold conditions.

C2IQL: Constraint- Implicit Q-learning for Safe Offline Reinforcement Learning
Zifan Liu' Xinran Li!
Jun Zhang'
Abstract
cable when such interactions costly, risky, or impracti- cal, such as in autonomous driving (Fang et al., 2022) and robotics (Weerakoon et al., 2024). To address this chal- lenge, offline RL (Fujimoto et al., 2019) has been proposed, focusing on learning policies from pre-existing datasets without further environmental interaction. challenge that arises in this offline setting is the out-of- distribution (OOD) problem, where an agent may inaccu- rately estimate the value of state-action pairs not found in the dataset. Various methods have been developed to tackle this challenge, such as regularization (Fujimoto et al., 2019) and constrained bootstrapping (Wu et al., 2020).
Safe offline reinforcement learning aims to de- velop policies that maximize cumulative rewards while satisfying safety constraints without the need for risky online interaction. However, ex- isting methods often struggle with the out-of- distribution (OOD) problem, leading to poten- tially unsafe and suboptimal policies. To address this issue, we first propose Constrained Implicit Q-learning (CIQL), a novel algorithm designed to avoid the OOD problem. In particular, CIQL. expands the implicit update of reward value func- tions constrained settings and then estimates cost value functions under the same implicit pol- icy. Despite its advantages, the further perfor- mance improvement of CIQL is still hindered by the inaccurate discounted approximations of constraints. Thus, we further propose Constraint- Conditioned Implicit Q-learning (C2IQL). Build- ing upon CIQL, C21QL employs a cost reconstruc- tion model to derive non-discounted cumulative costs from discounted values and incorporates mechanism to accommodate dynamic safety constraints. Experi- ment results on DSRL benchmarks demonstrate the superiority of C2IQL compared baseline methods in achieving higher rewards while guar- anteeing safety constraints under different thresh- old conditions.
significant
While showing impressive achievements in reward-only tasks, offline RL may lose its effectiveness in safety-critiaal scenarios. For example, unsafe operations could harm pa- tients in healthcare (Den Hengst et al., 2022), unsafe driv- ing styles may lead to accidents (Fang et al., 2022), and unsafe decisions may incur additional costs in financial in- vestments (Froot et al., 1993). In these situations, safety categorized into the safe exploration process that prevents agents from exploring risky states or actions, and safe op- timal criterion that ensures constraint satisfaction of target policies (Garcıa & Fernández, 2015). Although offline RL naturally addresses the safe exploration process by avoid- ing online interaction, it is challenging to satisfy the safe optimal criterion in offline settings due to the OOD problem (Zheng et al., 2024). Thus, safe offline RL (SORL) (Le et al., 2019), which incorporates safety constraints into the learn- ing process, has been proposed by combining safe online RL algorithms and offline RL algorithms. While existing SORL methods (Xu et al., 2022; Lee et al., 2022) have made progress in mitigating the 0OD problem and improving the constraint satisfaction ability, further improvement is im- peded by two key drawbacks: (I) the inability avoid the OOD problem in offline RL completely, and (2) the lack of accurate and flexible constraint handling in safe RL. We will elaborate on these in the following:
flexible,
1. Introduction
Reinforcement learning (RL) (Sutton & Barto, 2018) has emerged as a robust framework to solve decision-making problems through constant interactions with environments. However, this "trial and error" approach may not be appli-
'Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology, Hong Kong SAR, China. Correspondence to: Jun Zhang <eejzhang @ hs?.
From the perspective of offline RL, cxisting SORL methods that leverage OOD detection or regularization can only miti- gate the OOD problem but not avoid it. Specially, the agent remains susceptible to encountering unseen state-action pairs during model updates, which can lead unsafe poli-
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

C2IQL: Constraint- Implicit Q-lcarning for Safe Offliee Reinforcement Learning
cies. For instance, some approaches (Fujimoto et al., 2019) train a model for OOD detection but the detection models themselves also introduce errors. Other methods (Kostrikov et al., 2021; ; Xu et al., 2022), which restrict the learned pol- icy to be close to the behavior policy, can result in constraint violation or suboptimal policy since the behavior policy for collecting data is usually unknown. Implicit Q-learning (IQL) (Kostrikov al.. 2022) offers a potential solution to effectively avoiding the OOD problem. Nonetheless, it is challenging to explicitly achieve both reward maximiza- tion and constraint satisfaction since the policy is implicitly hidden in the value function.
To handle flexible threshold requirement, we draw in- spiration from the goal-conditioned RL method and incorporate constraint-condittoneed ability into CIQL, This allows C2IQL to allocate the cost budgets more effectively throughout the task horizon and cope with a wider range of threshold requirements for a better balance between reward maximization and constraint satisfaction.
(3) We evaluate the proposed C2IQL on Bullet-Safety-Gymn (Gronauer, 2022) and SaferyGynnasium (Ji al., 2023) with DSRL datasets under different threshold conditions. Our empirical results and comprehensive ablation studies demonstrate C2IQL's superiority in both constraint satisfac- tion and reward maximization.
From the perspective of safe RL, we identify two key short- comings that arise from the previous constraint formation: (I) inaccurate discounted formulation of constraints and (2) fixed constraint thresholds that fail to adapt to the dynamic requirements imposed by the environments as well as the long-horizon task nature. These problems can hinder the agent's performance, preventing it from constraint satisfac- tion or reward maximization. While some methods, such as average-constrained policy optimization (Agnihotri et al., 2024), attempt to address the first problem by solving the constrained Markov decision process on average settings rather than discounted settings, they are mainly suitable for on-policy methods, where the behavior policy and the target policy are the same, making them less applicable to offline RL. Furthermore, most RL-based methods have not adequately addressed the challenge of adapting to dynamic constraint thresholds, which crucial for horizoorzoon tasks and varying environmental requirements.
2. Background
In this section, we briefly introduce the constrained opti- mization problem in safe RL and provide essential back- ground on the constraint-penalize method of CPQ, which will be adopted to update the constrained value function, and the principle of IQL. to avoid the OOD problem in of- fline RL, which we generalize to constrained settings in our method. At last, we provide an overview of related works of safe RL, offline RI. and SORL respectively.
2.1. Problem Settings
Constrained Markov Decision Process (CMDP): CMDP (Altman, 1998) is typical formulation of safe RL defined by M = (S,A, p.r, rrr,c,,, 7),, where S is the state space, A is the action space, P(s' |s, a) is the transition function, r(s, a) and c(s, a) are the reward and cost functions, and y is the dis- count factor. In safe RI, we aim to find a policy π (als) that maximizes the performance measure while satisfying the safety constraints. In episodic tasks, where the trajectory is denoted
To address the above challenges, this paper proposes -onnriiit-CCdiiiioneed Implicit Q-learning (C2IQL), which is trained within the pre-collected dataset distribution. In particular, C2IQL contains a cost reconstruction model and is equipped with constraint-conditioned ability for better constraint satisfaction and reward maximization. The key contributions are summarized as follows:
{(So ao, ro, (o), ..., (ST, ar, mr,,Tr.Co cr))r; r(st, a.), = c( c($1, az), 8t+1 ~ P(8 at)), with T being the trajectory length, the optimization problem of safe RL can be written as:
(1) From the offline perspective, we first propose Con- strained IQL (CIQL) to address the OOD problem in con- strained settings. Specifically, we leverage the constraint- penalized update to expand the reward value function in IQI and re-derive cost value functions by generalizing IQL. so that both value functions can be updated under the same implicit policy without explicitly extracting it.
max Br-r(R(T)
(1)
8.l. E [C(T) S L,
where R(T)= = ET T' is the cumulative reward, C(r) Et= Cs is the cumulative cost, and L is the constraint threshold.
(2) From the safety perspective, we improve CIQL and propose C2IQL by addressing the following two key issues regarding the inaccurate constraint thresholds:
However, not all environments have explicit termination signals and the estimate of the cumulative reward/cost via the Bellman equation can easily lead to maximizing an infinite value. To ensure convergence and improve stability, discounting is introduced for a general notation for episodic and continuing tasks, which can be proven to have instance dependent sample complexity bound (Jiang & Ye, 2024)
To obtain accurate constraint estimations, we propose to train a cost reconstruction model to reconstruct non- discounted cumulative costs from the corresponding discounted values.
2

---

C2IQL: Constraint- Implicit Q-lcarning for Safe Offliee Reinforcement Learning
with a discount factor. In this case, the objective can be rewritten max (T);
Since the policy is implicitly hidden in the reward value and Q-value functions, IQL utilizes Advantage Weighted Regression (AWR) to extract the policy:
8.1. (T) S I,
La = B(ss)=
(+)) log [[|[s)],
where (T) E Di= rr,,] denotes the (reward)
(8)
value function, (T) E 2 >/a] denotes the cost value function and I. = L 1-?
where a E (0, 0o) is a temperature parameter to balance Q-eaannig optimization and behavior cloning.
T. 1-7 is an average approx- imation version for L. In SORL, the agent is trained with a fixed and pre-collected dataset = {(s, a, s') under unknown behavior policies.
Implicit Diffusion Q-learning (IDQL) Estruch et al., 2023). IDQL generalizes the expectile regression in IQL to any arbitrary convex function f with f0) 0 and re-derives IQI. as standard actor-critic algorithm, in which the optimal value function is given by
2.2. Constraints Penalized Q-learning (CPQ)
CPQ is a kind of SORL method updated in a shielding way. The update of CPQ (Xu et al., 2022) consists of three steps. First, CPQ trains a Conditional Variational AutoEncoder (CVAB) (Sohn et al., 2015) to detect OOD state-action pairs and manually train their cost Q-values (Q7) to be larger than the constraint threshold. Second, CPQ proposes constraints penalized update of the reward Q-values (QF), which is:
V (8) = arg min [f Vr "
(s, (TTn((s))
(s))],
(9)
where p(aas) is the behavior policy and the expectile regres- sion in IQL emerges as a special case when ʃ(-) = Lg(-).
While policy extraction through AWR in IQL is empirically effective, IDQL shows that this process be derived from first principles. Based on Equation (9), IDQL proves that the learned implicit policy follows:
O7 (st, at) =rt + BBaliiaa
1(Q dc+1) S L) (f (44)), (3)
u(als)) Tue (s)))-V((ss]
where 1(-) is the indicator function. Finally, the policy is optimized under the constraint, which is:
imp x
1Q QTaR (s, a) - Vs))
(10)
where f of av, me and AWR in IQL is another special case of f, called exponential regression.
π arg max Bas [ (s, s L (s)
(4)
However, CPQ fails to satisfy the constraint as the CVAE model may fail to detect OOD pairs some cases.
Since the policy is implicitly hidden in the value function, it is difficult to update the cost value function under the same policy for IQL-based methods without extracting it when extending to constrained settings.
2.3. Implicit Q-learning (IQL.
Implicit Q-learning (IQL) (Kostrikov et al., 2022). IQL addresses the fundamental OOD challenge in offline Q- learning by restricting the maximization operation within the dataset support. This OOD problem arises during value bootstrapping when standard Q-learning attempts to maxi- mize over actions that may not exist in the offline dataset:
2.4. Related Work
Offline RL: Offline RL aims to optimize a policy on a pre-collected dataset to avoid costly interaction with the environment. A fundamental challenge in offline RL is the OOD problem. Many methods have been proposed mit- igate it by regularizing the optimized policy close to the behavior policy. For example, BCQ (Fujimoto et al., 2019) achieves this by training a generative model as the behavior policy; BEAR (Wu et al., 2020) proposes a constrained boot- strapping operation to reduce the accumulated OOD error; CQL (Kumar et al., 2020) forces the values of OOD state- action pairs to be conservative. Instead of mitigating the OOD problem, IQL (Kostrikov al., 2022) tries to avoid it by employing expectile regression to learn Q-values within the dataset. More recently, some works formulate offline RL as a return-conditioned generation problem and address it using generative models, such as Decision Transformer (DT) (Chen et al., 2021: Zheng et al., 2022; Wu et al., 2024), and Decision Convformer (DC) (Kim et al., 2024).
O7 (8t, ae) = r. + y max O7 (s).
(5)
Directly taking the maximum within the dataset is sensi- tive to outliers (lucky samples) in the dataset and can lead to overestimation. Thus, IQL introduces value function based on expectile regression that estimates conservative expectile of the Q-value distribution:
I = [(...)) (77))s)))))))))))))) (s)),
(6)
where (u) I 1uu 0)|u² and x e 1).
Then the reward Q-value function is updated as:
(st, at) = rt + +V7 (8141).
3

---

C2IQL: Constraint Implicit Q-lcarning for Safe Offline Reinforcement Learning
Safe RL: Safe RL aims maximize future rewards while satisfying safety constraints (Garcıa& & Fernández, 2015). Primal-Dual Optimization (PDO) and Constrained Policy Optimization (CPO) are two common approaches in safe RI. PDO (Chow et al., 2018; Ding et al., 2020) leverages Lagrangian multiplier methods to update the primal and dual objectives respectively. Stooke et al. (2020) propose to incopporaee the PID control to address the instability is- sue of PDO. CPO (Achiam et al., 2017; Yang et al., 2020) takes a different route by transforming the constraint into surrogate objective and updating with it through second- order approximation. To reduce the computation cost of CPO, CUP (Yang et al., 2022) was proposed to optimize the surrogate function with first-order approximation directly. Although PDO and CPO show great promise in online set- tings, it is challenging for them to handle the OOD problem in offfine settings due to their reliance on sampling actions under current policies.
Discounted Costs (C)
40-
C
False Positive
20
False Negutive
20
40
60
80
100
Reconstrucied Costs (G)
100
C<L
Falso Positive
50-
CLL
False Negative
20
40
60
80
100
Cumulative Costs (C)
Figure 1. Relationship between the discounted costs C, recon- structed costs Õ and the cumulative cost C' of randomly generated cost trajectories.
what consequences will they lead to; (3) How to address (2) with cost reconstruction model and incorporate it into CIQL; (4) How to integrate the constraint-conditionn abil- ity into CIQL to improve flexibility; and (5) Practical imple- mentation of our proposed C2IQL algorithm.
Safe Offline RL: Safe offline RL (SORL) integrates safety constraints into offline learning settings, addressing both safety requirements and limited online interaction with the environment. This emerging field has spawned two main ap- proaches: RL-based methods and generative methods. CPQ (Xu et al., 2022) is representative RL-based method that trains a CVAE model to mitigate the OOD problem based on the constraint-penalizer method. COptiDICE (Lee et al., 2022) utilizes stationary distribution correction to mitigate the distributional shift problem. OASIS (Yao et al., 2024) in- vestigates the influence of the unknown behavior policy and provides dataset distribution shaping method to address this problem. The alternative generative approaches (Liu et al., 2023b) transform SORL into a goal-conditioned gen- erative problem, where expected reward and cost returns are set as the input of the generative models. Neverthe- less, these methods struggle to balance the reward and cost conditions, making it hard to maximize the reward while satisfying the constraint. Different from the above two lines of research, our proposed method C2IQL is an RL-based method equipped with the IQL-style update to avoid the OOD problem and can achieve both constraint satisfaction and reward maximization.
3.1. Constrained Implicit Q-learning (CIQL)
To derive a concrete CIQL algorithm, we need to answer three questions: First, how to update the constrained re- ward value function following the IQL style? To address this problem, CIQL formulates a constraint-penin peeeaaiitt-e re- ward Q-value function following CPQ and utilizes a value function with expectile regression to approximate the maxi- mized Q-value function in the Bellman backup procedure. Second, how to update the cost value function under the same implicit policy since it is hidden in the reward value function? To address this problem, we rederive CIQL and obtain the formulation of the implicit policy in Theorem following IDQL, and then derive the formulation of the cost value function with this implicit policy. Third, How to extract the policy? We extract the policy in an expectile way following Equation (18).
Value Function Estimation.
To address the safety require- ments in offline RL, we first propose CIQL, which extends IQL to handle safety constraints while maintaining its OOD- avoidance property. We start by considering how to incor- porate the safety constraints in the value function. Among existing safe RL approaches, we adopt the shielding mecha- nism from CPQ, as naturally enables value updates with- out explicit policy representation. By injecting the shielding update to IQL, we obtain the constraint-penalize Q-value function as
3. Method
In this section, we introduce novel algorithm CIQL that re-derives IQL within constrained settings to address the OOD problem and propose an improved version, named C2IQL, by further addressing the inaccurate and inflexible estimation of constraints.
In particular, we will illustrate (1) how to obtain an SORL algorithm based on the constraint-penalize avoids the OOD problem by extending IQI. to CIQL:; (2) Why discounted formulation on the costs is inaccurate and
method that
Q7 = r. + max Q7, (s+1),)))),,
(11)
a
where Q (8, = ( s L.) (x,a))
4

---

C2IQL: Constraint-Conditione Implicit Q-lcarning for Safe Offline Reinforcement Learning
Similar to IQL, we use value function to approximate expectile of the constraint-penalized Q-value and the loss function of the value function is given
Algorithm 1 Cost Reconstruction Model
1: Network: Initialize the cost reconstruction model R
Parameter: Set discount factors as (?1, 72,, .... Ym
3: Calculate the cumulative cost CD and discounted cumulative cost under different discount factors CD. ,..., CD:r for each trajectory in D. 4: for iteration= 0...., N do
IV. (1f,a)>>(L5((00) a) (+)),
(12)
where (u) |x 1(u 0)]u² and K (0, 1).
More generally, the expectile regression loss (2) can be replaced by any arbitrary convex fiunction f(u) with f())) = 0 Slansen-Esttuche et al., 2023). In this case, Equa- tion (12) generalized to
5: Update with data from D:
6: Sample {(C ,.., CD } ~ D. C.Phhn) CP)).
LR- MSE(R(CCM))))
8: Update with randomly generated data:
9: Randomly generate {(co .... er)]
V.T (8) = arg min B(E.a)- (x)
(+))]
10: for all trajectories E {(co, ..., CT)) do
11:
C = Li=g Cr-
(13)
12:
CR. = Ei= 7 o for i = l,...,m. m.
By now, have obtained the formulation and loss function of the munstraint-penallzed reward value function given the cost Q-value function Q7 (s, a). The key issue is how to update Q a) in 27/c (s, a), where the policy π is implic- itly hidden in V, (s) and Q, (s, a). One straightforward solution is to extract the policy and then use it to update the cost Q-value function. However, it will result in the OOD problem since the update based on the extracted pol- icy may fall out of the dataset distribution. To maintain the OOD-avoiding property, we need to derive an update method implicitly following the policy in the reward (Q-) value function. By extending the results in IDQL, we have:
13: end for
14: LR= MSSE(
R. CR..
15:
end for
15 (ea))
where M
107,((000-
(7( = ([(s,,)=((sa)) (s).
(16)
The detailed proof is shown in Appendix A.2. Accordingly, the loss to update the cost value function is given as:
Lv = Ea (al
(a)=))(((())) ()(( a) V. (+))²
Theorem 1. For every state and any convex loss function f with f((0) = O, the solution to Equation (13) is also a solution to the optimization problem in Equation (14) where the implicit policy is a reweight of the behavior policy.
(17)
BBann(ass)A [M (Q, V2 (s))),
In the special case of f(u) (u), we have M 11 (O)) - V, (s) < 0)).
|
V.7 (s) -ar min Ba- (a) 9) ((07 VF (1)) (+)
Policy Extraction. We follow Theorem 1 and extract the policy in an expectile way with the loss function:
(14)
where Timp (als) x (00)) (V
La = Banu(ols) log [<|=))]
(18)
The detailed proof is shown in Appendix A.I.
where M = |K - 111--(Q90
(17 < 0)).
Theorem 1 provides us with relationship between the pesstraint-penalized reward function and the corresponding formulation of the implicit policy. When the implicit policy formulation follows Equation (14), the update of the value function under implicit policy is equivalent to the update formulation of the value function in Equation (13) under behavior policy (the policy for collecting the dataset). Thus, we can utilize the formulation of the implicit policy obtained in Theorem 1 to derive how to update the cost value function based on its definition in Theorem? 2.
3.2. Motivation for Non-discounted Cost Reconstruction
While introducing discount factors enhances convergence and stability, it causes a mismatch between the original non- discounted constraint in Equation (1) and their discounted approximations in Equation (2). Such mismatch becomes a pronounced problem in safe offline RL because the dis- counted cumulative cost depends not only on the accumu- lated cost but also erucially on the temporal distribution of costs within a trajectory.
Theorem 2. Based on Theorem 1, the cost (Q-) value functions are specified as Equation (15) and Equation (16).
To demonstrate this problem, we randomly generate various cost trajectories {T; = (c)- ..., ci i = 1,... 10000; of e 10,1):CC C(r)(r)) < 100} and calculate the cumulative cost C' and discounted cumulative cost Ò for each trajectory, as
ve (s) = Eawwin (als) (x(((a) (s)
(15)
= Earwials) [M Qº (s,
5

---

C2IQL: Constraint-Condittooee Implicit Q-lcarning for Safe Offline Reinforcement Learning
Algorithm 2 C2IQL
cost value (Q a)) is a linear summation of cost func- tions c(s, a). To address the discounting-induce mismatch described in Section 3.2, the objective is solving linear equation system:
1: Network: Initialize value function V,* (s, L), cost value function V f.). Q-value function Q (s, a. T.), cost Q-value function Q (s, a, L). policy x (a|s, s, L), cost reconstruction model R
oetain Q7 ar) =
Σ
d De4s Rt,,
2: Parameter: Initialize discount factors y for reward and
j=0
y = (2-72.... "2..... Ym for cost, expectile parameter K1 and 1i,, threshold set L.
(19)
T
s.t. (8,
Σ
plosst u445 xT,,
3: for iteration= 0..... ..... .N do
j=0
4: Sample transitions a,, s')} ~ D.
5: Randomly sample a threshold L E f.
where (5t, 0z) are known values estimated by dis- counted Q-value functions.
6: Cost Reconstruction: Obtain discounted cost val- ues and reconstruct the non-discounted value in Sec- tion 3.3.
However, solving Equation (19) exactly is challenging due to the massive number of variables. To address this problem, we relax the objective function and constrict the constraint to: min (8t ax) (g(().)))
7: (=,s,a, a, L) = Q (s, a, L)..... 25-7m (8, L). 8: Q7 a, f.) = RC a, L)).
T
9: Costraint Penalization: Obtain the constraint-
Σ j=0
c( Asts ~ T))
penalized valuue function with Equation (11).
10:
0% I.) = 1 (Q: (s, a, f.) S 1)) a, E).
11:
Update value function: Update value function with
s.t. (st,
Σ
cl Hc(ss+j, 04ss ~ x), Vi I,... m j=0
Equation (12).
12:
(s, a, L) (T D).. Update cost value function: Update cost value func-
LV = E(s,a)>D(Ly) (27 (OTie(s
(20)
13:
Then the goal becomes training an accurate cost reconstruc- tion model, denoted as R estimate the reconstructed cost Q from discounted cost values with different discount factors (?::?..... J: (s.a)..... A. a)).).
tion with Equation (17).
14:
Lv. [l 1(07 -)- VF L) < Ojl- (QI a, L) V. (s, [))?].
15:
Update Q-value function.
16:
Lq.- = (ss.a)> +
(s', L) +VV,
(=,, a, ?)
(s,
(21)
17:
Update cost Q-value function for each 7i E f..
Algorithm demonstrates the training process of the recon- struction model. Specifically, we use the MSE loss to update the cost reconstruction model fe and augment the original dataset with some randomly generated data to improve the generalizability of the model.
18:
LQ. = EEs.a)>Dl(c
(s', QT(s, a, D)).
19:
Policy Extraction.
20:
Ln = -Dlsa))DDIkg 100%
V, i) < 0)| log (als, L)].
21:
end for
The bottom figure of Figure demonstrates the recon- structed value of the generated trajectories in Section 3.2. Compared with the discounted value shown at the top, the false positive and negative trajectories with reconstructed cumulative cost are significantly reduced. Constraint satis- faction with the reconstructed cumulative cost provides a more accurate estimation of the cumulative cost compared to using the discounted value.
shown in the top figure of Figure 1. The results reveal that the mapping from cumulative costs to discounted values is many-to-one, creating two types of problematic cases when approximating constraint C(r²) < with C(r') S L. (I) False Positives: Trajectories with cady-cooncettraeed costs may satisfy the original constraint C(r²) < L but violate the discounted approximation C(r') < L, leading to overly conservative policies that reject safe trajectories. (2) False Negatives: Trajectories with delayed costs may satisfy the discounted constraint while violating the original one, potentially leading to unsafe policies.
3.4. mnnstraint-Conditoone Ability
CIQL employs a fixed threshold criterion 1(2 (se+1, a) I L) that fails to account for the dynamic nature of accumu- lated costs. This limitation prevents the agent from adapting its behavior based on the historical cost accumulation and remaining cost budget. For instance, when past costs low, the agent should be able to pursue more aggressive, reward maxumiii policies within the remaining cost bud- get. Conversely, when significant costs have already been incurred, the agent should adopt more conservative poli- cies to ensure overall constraint satisfaction. To address
3.3. Cost Reconstruction Model
Following the definition of Equation (2), the cost value func- tion (s, a)) is a linear summation of cost functions
c(s, a) weighted by 7t < 1. Similarly, the non-discounted
6

---

C2IQL: Cunssraint-Conditionn Implicit Q-lcarning for Safe Offliee Reinforcement Learning
Table 1. Normalized evaluation results. The normalized cost threshold is set to 1. Values in (.) represent the standard deviation. Each value represents the average performance over 10 evaluation episodes with seeds and thresholds. Black indicates safe results; gray indicates unsafe results; and blue indicates safe and best- rest-performing results. 1 (J) incidates that higher (lower) values are better.
Tasks
Algorithm
Metric
Run
Circle
Ant
Ball
Car
Drone
Ant
Ball
Car
Drone
Avg
reward f
0.56
m.27
0.92m.on)
0.76
0.76xa.5)
0.64
0.67
0.96xa.0
0.69
BCQ-Lag
cost 4.
0.28
0.39
0. 10a.30
1.62
1.73ma2
0. 0.98
1.23
2.09
1.05
reward t
(..4
0.22.o.5)
0.
0.4 0.02a.1s)
0.7 0.75
0.73
0.85
0.49
BEAR-Lag
cost J.
(.02
(.97
3.7 (e.59)
(.
1. 1.044.43)
1.50
1.59
1.49
reward f
(.57
0.47
0.90ma.z)
0.63
0. 19aas))
0.62
0.43
0.40
0.53
COptiDICE
cost L 0.22(e.11)
1.25
0.00
1 1.40
0.9 (0.19
0.1
1.20
0.42.o.10)
0.80
reward f
(0.000)
0.63
0 0.94)..0
0.33
0.02m.az)
0.63
0.59
0.13.ooo
0.41
CPQ
cost
(0.00.00)
1. 1.410
1 1.420.10
1 16220.41
0.06
(o19
1 1.03
0 0.19)
0.83
reward f
0. 1.18 81..03)
0.24
0.80ka.
p.20
0. 1 (ius))
0.25
0. O.23
0.46maa3)
0.31
FISOR
cost
(.00
0.00
0.01
0.41
0.00ka.00
0.03:a.3
0.00
0.00
0.06
reward f
O.32(a.03)
0.79
0.43 54)
0 0.48,o.9
0.00ke.00)
0.85
0.3 0.39 (..2)
0.1 o..0.
0.42
VOCE
cost
0.86(4.2)
1.04k..00
6.5 1,.00
1.58
1.0 Lo.05)
1.34,000
1.43:087>
0.57
1.80
reward f
0.25
0.80g.30)
0.86:0.99
0.66,.12)
0.40
0.69
(0.14))14)
0.02.a.01)
0.54
WSAC
cost +
0. 1
1. 1.980.33
0.40s..0)
2.52
0.9 0.98..12)
0.78
0.51 (0.14)
0.45.o.28)
0.97
reward f
0.72re.03)
0.56x0.01)
(.0000
0.66:0.0)
0.5460.6))
0.78:0..1
0.72ro.02)
0.75.v.01)
0.71
CDT
cost |
1.00
0.97...02)
0.80
0.73
0.9 1vo.13)
0.96:a.33
0.
0.97..00
0.91
reward f
0.74:a.01)
(.59
(.00)
0.71
0.66x0.0)
0.72:0.83)
0.74
0.78...02)
0.74
C2IQL
cost 4.
(.94(a.0))
0.95
0.08
0.73
0.76x..13
0.85
0.93
0.85
0.76
this problem, we include possible threshold conditions L in the input of policy and value functions in CIQL,, such VF (s, L) and V (s, L). By incorporating the threshold as an input, the agent can dynamically adjust its policy based on the remaining budget, enabling it to adapt to the evolving cost requirement.
Car, and Drone; and two kinds of tasks: Run and Circle. We use the DSRL (Liu et al., 2023a) dataset, which follows the D4RL (Fu et al., 2020) benchmark format. The details of the environments and more experiments on SufetyGymnasing are described in Appendices B. and B.3.
Baselines. We compare proposed C2IQL with the following baseline methods: (1) Primal-dual optimization (Stooke
3.5. Practical Implementation
et al., 2020): BCQ-Lag and BEAR-Lag: (2) Constraint
To sum up, Algorithm 2 demonstrates the overall training procedure of C2IQL. Notably, we use reconstrueted costs in line 10 for Q-values with more accurate constraint penalty. In practice, we update the value function and cost value function with the same expectile parameter K1 and update the policy function with another expectile parameter N2g This design maintains balance between IQL and IDQL approaches, with K2 serving as a temperature parameter to balance between behavior cloning and maximization of the value function. To reduce the computational complexity, instead of updating all constraint conditions, we predefine a threshold set L and sample thresholds randomly from it dur- ing training. More details of the implementation and hyper- parameters used in Section 4 are available in Appendix B.2.
penalized method: CPQ (Xu et al., 2022); (3) Distribution
correction estimation: COptiDICE (Lee et al., 2022); (4) Sequential modeling algorithms: CDT (Liu et al., 2023b) and FISOR (Zheng et al., 2024): (5) Variational optimization with conservative estimation: VOCE (Guan et al., 2023); (6) Weighted safe actor-critic: WSAC (Wei et al., 2024).
Metrics. We evaluate performance using normalized reward and cost returns. The normalized reward return is defined by -R,iimin where R. is the reward return under Rnom Rem.- -amin policy x, Ruin and Rmax are the minimum and maximum reward returns in dataset. The normalized cost return is defined by Cnorm = Gz, where C₂ is the cost return un- der policy л and L the selected threshold. To provide a comprehensive evaluation, we assess the performance of the algorithms across three cost thresholds: "small", "middle", and "large", calibrated to each environment's 's cost range. This evaluation scheme allows us to analyze both constraint satisfaction and reward maximization capabilities of algo- rithms across varying degrees of constraints. Detailed values are provided in Appendix B.2.
4. Experiments
Environments and Datasets. We evaluate C21QL in Buller- Safety-Gym (Gronauer, 2022), which includes various con- tinuous robot locomotion control tasks commonly used in previous works. We select four types of robots: Ant, Ball,
7

---

C2IQL: Constraint-Conditionee Implicit Q-learning for Safe Offline Reinforcement Learning
C21QL
CDT
1.0
1.0
1.0
1.0
0.8
0.8
0.8
Rowards
0.6
0.6
0.4
0.4
0.4
0.4
1.00
1.00
1.00
Costa
0.75
0.75
0.75
0.75
0.50
0.50
0.50
0.50
0.25
0.25
0.25
0.25
0.00
0.00
0.00
0.00
small
middle
large
small
middle
large
small
middle
large
small
middle
large
Thresholds
Thresholds
Thresholds
Thresholds
(a) CarRun
(b) CarCirele
(c) DroneRun
(d) DroneCircle
Figure Normalized evaluation results comparison between C2IQL and CDT with varying constraint thresholds.
4.1. Main Results
C2IQL w/o CC: C2IQL without Constraint-
Conditioned formulation introduced in Section 3.4.
Table presents the evaluation results. C21QL achieves the best and safe performance in most tasks, demonstrating its superiority in reward maximization and constraint satis- faction. Most baselines except FISOR and CDT generally suffer from constraint violations. Even when they show safe behaviors, the performance inferior compared to C2IQL. FISOR, on the other hand, achieves safe results across environments but exhibits overly conservative behavior due to its hard constraints property, which limits its ability to achieve high rewards.
CIQL: The algorithm proposed Section 3.1, where both the cost reconstruction model and constraint- conditioned ability are not included.
As shown in Figure 3, C2IQL w/o CR demonstrates inferior reward performance compared with C2IQL even though C2IQL w/o CR reaches the cumulative costs close to the threshold due to the constraint conitrainnt-cddiiionnddddd ability. This in- dicates that the cost reconstruction model helps to improve reward maximization. On the other hand, C2IQL w/o CC ex- hibits conservative behavior in terms of costs compared with C2IQL, which indicates that the mulation helps improve the constraint satisfication ability, CIQL, which lacks both the cost reconstruction model and constraint-conditioned ability, converges a safe but over- conservative policy for all thresholds. While the safety of CIQL benefits from avoiding the OOD problem and the over- conservative policy is caused by the inaccurate discounted constraint and the absence of constraint-conditoone ability. As a result, CIQL can only achieve very low performance with nearly zero cost, regardless of the scenario. Among all the variants, only C2IQL is able achieve the high- est rewards with the closest costs to the thresholds, which demonstrates the combination of the cost reconstruction model and strike a balance between maximizing rewards and satisfying constraints effectively.
Although CDT achieves similar results to C2IQL in Table I, a closer examination of their performance under different safety thresholds in Figure reveals that C2IQL can flexibly adjust its policy when dealing with different safety thresh- olds, while CDT does not. CDT fails to ensure safety in certain environments for small thresholds, whereas C2IQL consistently achieves safe results. In contrast, CDT cannot maximize rewards as effectively as C2IQL for large thresh- olds, even though both methods ensure safety. Notably, as the threshold increases, C2IQL progressively improves its performance, while CDT encounters limitations in specific environments such as DroneCirele.
for-
4.2. Ablation Study
To evaluate how much the cost reconstruction model and constraint-conditioned method contribute to the perfor- mance gain, we conduct an ablation study with the following variants of C2IQL:
ability enables C2IQL to
4.3. Hyper-parameter Analysis
C21QL w/o CR: C2IQL without Cost Reconstruction module introduced in Section 3.3. The discounted threshold follows the average approximation version,
We conduct analysis of two hyperparameters: the expectile parameter on the value function and the cost value func- tion 11 and the expectile parameter on policy extraction K2, shown in Figure 4. in primarily affects constraint satisfac-
which f-= 4 T-7
1-2""
8

---

C2IQL: Constraint-Condittonee Implicit Q-lcarning for Safe Offliee Reinforcement Learning
CIQL
C2IQL w/o CC
C2IQL w/o CR
C2IQL
440
500
500
0.9
435.65
428 22.88
441.92
400
Rewards
Rewards
400
400
K2
0.8
4433,98
427.60
439.22
420
200
300
300
0.7
400.06
409.31
411.05
-110
20
60
40
20
80
0.7
0.8
0.9
100
100
100
0.9
48.74
49.52
57.79
55
Costs
50
50
K2
0.8
47.85
48.96
55.62
Costs
-50
0.7
46.73
50.44
55.35
0-
0
20
40
60
80
20
40
60
80
20 40
60
80
0.7
0.8
0.9
Episodos (5c3)
Episodes (5c3)
Episodes (5c3)
K1
Figure 3. Ablation study on CarCirele with varying constraint thresholds. The three plots from left to right represent different threshold levels: small = 30, middle = 50, and high The dashed lines each plot indicate the corresponding threshold value for that case.
Figure 4. Hyperparameter analysis on CarCircle with threshold = 50.
Table Evaluation results of C2IQL,, whose CRM is trained different noise levels, task CarCircle. The constraint threshold
selected as 50. MSE indicates the training loss of CRM.
Noise
0.0
0.1
0.2
0.3
0.5
1.0
2.0
4.0
MSE
0.025
1.10
1.77
2.25
3.35
4.41
5.76
7.84
reward f
434.01
435.07
442.08
435.67
435.60
421.82
408.77
401.89
cost 1
46.60
47.20
48.30
47.30
47.89
42.10
34.90
36.20
tion where a smaller Ki leads to smaller costs. Excessive 81 values, such as 0.9, can cause the value function to be affected by some "luckily safe" transitions, also known "lucky samples" in IQL, and thus result in unsafe policy. K2 primarily affects reward maximization where larger K2 leads to higher performance with little effect safety. The main reason is that smaller K2 can make the policy closer to behavior cloning, degrading the performance. These find- ings suggest that selecting smaller Kn value and a larger K2 value are often good choice for achieving both safety and high performance.
Despite this, the reward and cost metrics degrade gracefully, indicating the robustness of C2IQL.
5. Conclusion
In this paper, we introduce C2IQL, novel approach to SORL that effectively addresses the OOD challenge in constrained settings by leveraging constraint-penalize im- plicit updates. Our findings also highlight the critical is- sues associated with inaccurate discounted approximations of constraints and the rigidity of fixed constraint thresh- olds. By implementing cost reconstruction model, C2IQL adeptly reconstructs non-discounted cumulatíve costs from discounted values across varying discount factors. Addi- tionally, the constraint-conditoones mechanism allows for dynamic adjustment constraint thresholds, significantly enhaneing the algorithm's 's adaptability. These innovations make C2IQL a promising approach for SORL, offering both improved OOD handling, and accurate and dynamic con- straint satisfaction abilities, paving the way for more robust and reliable SORL algorithms in the future.
4.4. Robustness of the Cost Reconstruction Model
To validate how the accuracy CRM influences the perfor- mance of C2IQL, we introduce the noise sampled from normal distribution, N0, noise), the input of CRM and train C2IQL on task CarCircle. As shown in Table 2, the performance slightly improves with constraint satisfaction as the noise level increases when the added noise is small. This occurs because small amounts of noise act similarly to data augmentation, which enhances the generalization capability of the cost reconstruction model. This is anal- ogous adding noise input data image processing to improve robustness. As noise increases from 1.0 to 4.0, the performance becomes progressively more conservative.
Acknowledgements
This work was supported by the Hong Kong Research Grants Council under the Areas of Excellence scheme grant

---

C2IQL: Constraint-Conddttonee Implicit Q-lcarning for Safe Offline Reinforcement Learning
A0E/E-601/22-R and NSFC/RGC Collaborative Research Scheme grant CRS HKUST603/22.
Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S. D4rl: Datasets for deep data-driven reinforcement learning, arXiv preprint arXiv:2004.07219 2020.
Impact Statement
Fujimoto, S., Meger, D., and Precup. D. Off-policy deep reinforcement learning without exploration. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2052-2062. PMLR, 09-15 Jun 2019. URL httpttttss///pptpetppteeteteeeee v97 fs7/fujiootolloo 9a. html.
This paper presents work whose goal is to advance the field of Safe Reinforcement Learning in offline settings, There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
mlr pre.srssss/p/////oomee
References
Achiam, Held, D., Tamar, A., and Abbeel, P. Con- strained policy optimization. In International conference on machine learning, pp. 22-31. PMLR, 2017.
Garcia, and Fernández, F. A comprehensive survey on safe reinforcement learning. Journa of Machine Learning Research, 16(1):1437-148), 2015.
Agnihotri, A., Jain, R., and Luo, H. ACPO: A policy opti- mization algorithm for average MDPs with constraints. In Forty-first International Conference on Machine Learning, 2024. URL httpttps://ppenreeew...oo forum? diddddidddnnuium
Gronauer, S. Bullet-safety- A framework for con- strained reinforcement learning. Technical report, TUM Department of Electrical and Computer Engineering, Jan 2022.
Altman, Constrained markov decision processes with total cost criteria: Lagrangian approach and dual linear program. Mathematical methods of operations research, 48:387-417, 1998.
Guan, J., Chen, G., Ji, J., Yang. I.., zhou, a., Li, Z., and jiang, c. Voce: Variational optimization with conservative estimation for offline safe reinforcement learning. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M.. and Levine, S. (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 33758-33780 Cur- ran Associates, Inc., 2023.
Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I. De- cision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084-15097, 2021.
Hansen-Esttuuhh P., Kostrikov, I., Janner, M., Kuba, J. G., and Levine, S. Idql: Implicit q-learning as an actor- critic method with diffusion policies. arXiv preprint arXiv:2304. 10573, 2023.
Chow, Y., Ghavamzadeh, M., Janson, L., and Pavone, M. Risk-constrained reinforcement learning with percentile risk criteria. Journal of Machine Learning Research, 18 (167):1-51, 2018.
Ji, J., Zhang, B., Zhou, J., Pan, X., Huang, W., Sun, R., Geng, Y., Zhong. Y., Dai, J., and Yang, Y. Safety gymma- sium: unified safe reinforcement learning benchmark. Advances in Neural Information Processing Systems, 36: 18964-18993, 2023.
Den Hengst, F., François-Lavet, V., Hoogendoorn, M., and van Harmelen, F. Planning for potential: efficient safe reinforcement learning. Machine Learning, 111(6):2255 2274, 2022.
Jiang. and Ye, Y. Achieving o(1/e) sample complexity for constrained markov decision process. In Globerson, A.,
Ding, D., Zhang, K., Basar, T., and Jovanovic, M. Natural policy gradient primal-dual method for con- strained markov decision processes. Advances in Neural Information Processing Systems, 33:8378 8390, 2020.
Mackey, L.,, Belgrave, D., Fan, A., Paquet, U., Tomezak, J., and Zhang, C. (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 78679-78714, Cur- ran Associates, Inc., 2024.
Fang, X., Zhang, Q. Gao, Y., and Zhao, D. Of- fline reinforcement learning for autonomous driving with real world driving data. In 2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC), pp. 3417-3422. IEEE, 2022.
Kim, J., Lee, S., Kim, W., and Sung, Y. Decision con- vformer: Local filtering in metaformer sufficient for de- cision making. In International Conference on Learning Representations, 2024.
Froot, K. A., Scharfstein, D. S., and Stein, J. C. Risk man- agement: Coordinating corporate investment and financ- ing policies. the Journal Finance, 4825):1629-1658, 1993.
Kostrikov, I., Fergus, R., Tompson, J., and Nachum, O. Of- fline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning, pp- 5774-5783. PMLR, 2021.
10

---

C2IQL: Constraint-Condittonee Implicit Q-lcarning for Safe Offline Reinforcement Learning
Kostrikov, I., Nair, A., and Levine, S. Offline re- inforcement learning with implicit q-learning. In International Conference on Learning Representations, 2022. URL stttpp://peezeevew..eee.. 2 B.
Weerakoon, K., Sathyamoorthy, A. J., Elnoor, M., and Manocha, D. Vapor: Legged robot navigation in structured outdoor environments using offline reinforce- ment learning. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 10344-10350. IEEE, 2024.
Kumar, A., Zhou, A., Tucker, G., and Levine, S. Con- servative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems. 33: 1179-1191, 2020.
Wei, H., Peng, X., Ghosh, A., and Liu, X. Adversarially trained weighted actor-critic for safe offline reinforcement learning. Advances in Neural Information Processing
Systems, 37:52806-52833 2024.
Le, H., Voloshin, C.,, and Yue, Y. Batch policy learning under constraints. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 3703-3712. PMLR, 09- 15 Jun 2019, URL https: pestps://proceste http:///oooeo//eeeeeeeeee mll.. press/v97 (wtp/////119.... html.
Wu, Y., Tucker, G., and Nachum, O. Behavior regularized offline reinforcement learning, 2020. URL https:// openreview net fooum? BJ=JJghTTKKH.
Wu, Y-H.,, Wang, X., and Hamaya, M. Elastic decision transformer. Advances in Neural Information Processing Systems, 36, 2024.
Lee, J., Paduraru, C., Mankowitz, D. J., Heess, N., Pre- cup, D., Kim, K.-E., and Guez, A. COptiDICE: Offline constrained reinforcement learning via stationary distribu- tion correction estimation. In International Conference on Learning Representations, 2022. URL https:// openreview.nor net Forum Wdw.inid---FPeeeeeee
Xu, H.. Zhan, X., and Zhu. X. Constraints penal- ized q-learning for safe offline reinforcement learn- ing. Proceedings of the AAAI Conference Artificial Intelligence, 36(8))8753-8760 Jun. 2022. doi: 10.1609/ www.vv66..20880 URL htt index. php/ AAAI htt.a//aaaiiceeeeee/ee///
Liu, Z., Guo, Z., Lin, H., Yao, Y., Zhu, J., Cen, Z., Hu, H., Yu, W., Zhang, T., Tan, J., et al. Datasets and benchmarks for offline safe reinforcement learning. arXiv preprint arXiv:2306.09303, 2023a.
Yang. L., Ji, J., Dai, J., Zhang, L., Zhou, B., Li, P., Yang, Y., and Pan, G. Constrained update projection ap- proach to safe policy optimization. Advances in Neural Information Processing Systems, 35:9111-91244 2022.
Liu, Z., Guo. Z., Yao, Y., Cen, Z., Yu, W., Zhang, T., and Zhao, D. Constrained decision transformer for offline safe reinforcement learning. In International Conference on Machine Learning, pp. 21611 1661-227 1-21630. PMLR, 2023b.
Yang, T.-Y,, Rosca, J., Narasimhan, K., and Ramadge, P. J. Projection- constrained policy optimization. In International Conference on Learning Representations, 2020. URL hwtph:ttt///./..... /wwp///pppe.eeeee/ee//e/e. net wtt.:/iwwww.ee....eeeeeeeeee forum? i id=rke3TJrtP.
Sohn, K., Lee, H., and Yan, X. Learning structured output representation using deep conditional generative models. In Cortes, C., Lawrence, N., Lee, D., Sugiyama, M., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015. URL https::/prrcceediinn neurips. co/paped filess/papee///011//i11e 8d55a24 2aleeeeaaaaa00722777222222250000 pdf.
Yao. Y., Cen, Z., Ding, W., Lin, H., Liu, S., Zhang, T., Yu, W., and Zhao, D. OASIS: Conditional dis- tribution shaping for offline safe reinforcement learn- ing. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https: //opereeveww htwpi/ewwwweeeeeee net hwt.:/eetteeoeeeeeeeee www.forrummndmnnmmmeeee
Zheng, Q., Zhang, A., and Grover, A. Online decision trans-
former. In international conference on machine learning, pp. 27042-27059. PMLR, 2022.
Stooke, A., Achiam, J., and Abbeel, P. Responsive safety in reinforcement learning by pid lagrangian methods. In International Conference on Machine Learning, pp. 9133-9143. PMLR, 2020.
Zheng, Y., Li, J., Yu, D., Yang, Y., Li, S. E., Zhan, X., and Liu, J. Safe offline reinforcement learning with feasibility- guuied diffusion model. In The Twelfth International Conference on Learning Representations, 2024. URL https: http:::ttt//pptttt //tp:::///peereview forum??
Sutton, R. and Barto, A. Reinforcement Learning, second edition: An Introduction. Adaptive Computation and Machine Learning series. MIT Press, 2018. ISBN
id=j5JvZCaDMO.
9780262039246. URL https: /tttpp::/bboks.. /books...........oo google. com com.hk wtt.kkkooooookssooooooooooo

---

C2IQL: Constraint-Condittonee Implicit Q-lcarning for Safe Offline Reinforcement Learning
A. Proof
A.I. Proof of Theorem 1.
(s) arg min (caao (F(+)
arg min E Ban V.7 (+)
For convex function f, the optimality is achieved when:
0
(V:(s) Ea
- VV)
Due to convexity of f and the assumption (0) = 0, we have f'(x) = (
-aaappaas - (7)) (7, (s)
(7,c( Vs]
(7, (s)
- | 1ld - v,( (3, (+)
(v.o) where Zmmp is a normalization factor to
We then define the implicit policy to be n. (als) = p(als) keep the sum the probability as 1.
7u 12 (8.0) (3)
/ TH im (s) -V7,))
mmp
-mmpp (1 -VV())
Zimp
(Q0 (;()))
2 OV
0
This means that (s) is also a solution for the optimization problem:
arg min Ea (a (s))) (s)))
(+)
A.2. Proof of Theorem 2.
V (8(()) Earain (a) (s)
Bavau (o) (g, (=()ll
Eggn (ala) (s, [s)
(a)s) v2()))
imp (a)( VV)
Following the definition of the implicit policy in Theorem 1: rimP(als) = ju(als) (aa normalization factor to keep the sum of the probability as 1.
7uo Q (V) where Zump is
1 /. u(als)
(s) V2)(
(s, Vs((s))
Zimp
(x) - V,7(
- V;)())
Eapplaa ((aaa
(x, V=(()] (Q. a) (+))
Zimp
|9(0 (s,a) V(((s))
E Bana(apss)
10, (s, Vs( (Q. (s))
20m
@V:(s)
12

---

C2IQL: Constraint Implicit Q-learning for Safe Offline Reinforcement Learning
This means that (2()) (s) solution for the optimization problem
[f(( [++1)
(s) arg min Ba-p((ass
Q (1) V;())| (Q, - (s))
we take f as L2, the loss function should be
Lv. = B(saa)=DIK 1 - (s) < - (Q) -
B. Experiment Details
Figure 5. Run task.
Figure Circle task.
B.1. Environment Details
Mallet-Safety-Gee -ym (Gronauer, 2022) consists of two types of tasks: Run and Circle. In Run tasks shown in Figure 5, the rewards of the agent come from running between two safety boundaries with high speed and the costs come from either reaching the boundaries or exceeding specific velocity related to different robots. Circle tasks shown in Figure 6, the rewards of the agent come from moving circularly within the circle area and the costs come from when the robot leaves the safety zone of the yellow boundaries. Besides two tasks, there are also four kinds of robots: Ant, Ball, Car, and Drone with different parameters, shown in Table 3.
Table 3. Environment parameters.
Tasks
Parameter
Run
Circle
Ant
Ball
Car
Drone
Ant
Ball
Car
Drone
Episode length (T)
200
100
200
200
500
200
300
300
Action space
8
4
2
2
State space
33
17
34
8
8
18
Cost range
150
80
40
140
200
80
100
100
B.2. Implementation Details on C2IQL
For C2IQL, the structure and most hyperparameters follow IQL (Kostrikov et al., 2022). Table shows the hyperparameters of our algorithms. The discount factor of the reward is fixed at 0.99 and the number of discount factors for the cost is 3. Table 5 demonstrates the discount factors for cost reconstruction and different thresholds for comprehensive comparison. For most environments, the selection of discount factors are based the episode length, where 7f' 0.05, 7/ ~ 0. and no 0.2, except for BallRun and AnrCircle. We select smaller discount factors for better stability and convergence for these two environments. Besides, we select three thresholds considering the cost range of each environment. In particular, we select "small" value, which is less than 50%; "middle" value, which is approximately 50%; and a "high" value, which is more than 50%.
For the cost reconstruction model, we use 5-layer MLP with hidden dimensions of 512 for each layer. We pre-train train the cost reconstruction model for each environment with the offline dataset and randomly generated data. Besides, to maintain
13

---

C2IQL: Coustraint-Conddttonee Implicit Q-learning for Safe Offline Reinforcement Learning
computational efficiency while computing multiple discounted estimates, employ parameter sharing across different discount factors by extending the cost value and Q-value functions to produce multiple outputs:
Qº (s, a) QU (s, ..,, Q 7m (s, a).
(22)
To prevent overfitting and improve robustness, we add the noise sampled from a normal distribution N(0, 0.1) to the input. We pre-train the reconstruction model for 1e6 epochs for each environment.
For the threshold set of each environment, we set L {small, middle, high, oc) the constraint condition of C2IQL for the training procedure. In the testing procedure, we scale the cost budget linearly based on horizon T, as shown in Equation (23), and then choose the closest value in L as the constraint condition.
L - C T T/(T - 1):
(23)
where C+ = - Cj is the cost budget at time step t.
Table 4. Hyperparameters Hyperparameters
Value
Ki
0.7
R2 (reward)
0.9
0.99
m Batch size Learning rate of V Learning rate of Q Learning rate of π Training steps
3
512
le-3
le-3
3e-4
4e5
Testing frequency
5e3
Table 5. Discount factors for cost reconstruction and constraint thresholds for experiments.
Tasks
Hyperparameters
Run
Circle
Ant
Ball
Car
Drone
Ant
Ball
Car
Drone
Episode length (T)
200
100
200
200
500
200
300
300
71
0.985
0.95
0.985
0.985
0.99
0.985
0.99
0.99
?2
0.99
0.96
0.99
0.99
0.993
0.99
0.993
0.993
73
0.993
0.97
0.993
0.993
0.995
0.993
0.995
0.995
Cost range
150
80
40
140
200
80
100
100
Thresholds
Small
30
30
10
30
40
30
30
30
Middle
70
50
20
70
100
50
50
50
High
110
70
40
110
160
70
70
70
B.3. Additional Experiments
To further validate the superiority of our proposed method, we incorporate seven additional complex tasks on SafetyGymna- sium (Ji et al., 2023). Specifically, we select the four "Point" and three "Velocity" tasks as additional benchmarks different from Sallet-Safety-Gym. We compare C2IQL with two well-performed baselines in Table 1: FISOR and CDT. As shown in Table 6, C2IQL achieves the best performance while satisfying constraints on all tasks. Among most tasks, C2IQL achieves nearly 10% improvement compared with other safe performance.
To further demonstrate the contributions of the cost reconstruction model and constraint-coondtioneed method to C2IQL,, we include training curves for task DroneCircle as an additional ablation study in Section 4.2. As shown in Figure 7, only C2IQL achieves the highest rewards while satisfying the constraint. C2IQL w/o CC and C2IQL w/o CR fail to satisfy the constraint for small or large constraints, and are more conservative other cases compared with C2IQL. CIQL overly conservative all cases even though much safer.
14

---

C2IQL: Constraint-ondditioee Implicit Q-lcarning for Safe Offline Reinforcement Learning
Table Normalized evaluation results on complex tasks. The normalized cost threshold is to 1. Values in (.) represent the standard deviation. Each value represents the average performance over 10 evaluation episodes with 5 seeds and 3 thresholds. Black indicates safe results; gray indicates unsafe results; and blue indicates safe and best best- performing results. 1 (J) incidates higher (lower)
values are better.
Task
Algorithm
Metric
Point
Velocity
Circlel
Goal
Pushl
Buttonl
HalfCheetah
Hopper
Ant
Avg
reward f
0.65
(.61
0.0z)
0.06
0.86
0.15
0.83
0.49
FISOR
cost
(.23()
0.48
0.1 1 (0.08)
0.02ia.z)
0.00ma.0
0.03x.oz)
0.00
0.27
reward t
0.56
0.69ro.01)
0.25
0.52
0. Dro00)
0.72(0.02)
0.99
0.67
CDT
cost +
(.63
1.03
(.45
1.63
0.06:o.04)
(.42(0.24)
0.46
0.67
reward f
(a69
(.75
o.34
0.34
0.99
(.79(..4)
1.00
0.56
C2IQL
cost +
0.77(a.1)
0.92(a.15)
0.46xa05)
(.91
(.5
0.34
0.58
0.64
CIQL
C2IQL w/o CC
C2IQL w/o CR
C21QL
750-
750
750
Rewards
500
500
500
250
250
250
20
40
80
20
40
80
20
60
80
100
100
100
Costs
50
50
50-
0
20
40
60
80
20
60
80
20
60
Episodes (5e3)
Episodes (5e3)
Episodes (5e3)
Figure 7. Ablation study DroneCircle with varying constraint thresholds. The three plots from left right represent different threshold
levels: small 30, middle = 50, and high 70. The dashed lines in each plot indicate the corresponding threshold value for that case.
B.4. Infrastructure and Runtime Analysis
Experiments carried out NVIDIA GeForce RTX 3080 GPUs. We have done the runtime analysis about C2IQL and corresponding baselines, as shown in Table 7. We follow the same iteration times, which is 400,000, for most algorithms. As for VOCE and WSAC, we follow the same setting with the corresponding works and make sure of their convergence. Overall, the training cost of C2IQL is reasonable when compared to other methods. While it takes slightly more time than methods like FISOR and VOCE, it is still significantly faster than CDT, which has the highest training time. Notably, C2IQL achieves a remarkable balance between computational efficiency and performance. The additional training time is justified by the significant performance gains provided by C2IQL, making a practical and effective solution.
Table
Runtime analysis of C2IQL and corresponding baselines task AntCircle.
Algorithm
BCQ-Lag
Bear-Lag
COptiDICE
CPQ
FISOR
VOCE
CDT
WSAC
C21QI.
Training Iterations
400,000
400,000
400,000
400,000
400,000
40,000
400,000
30,000
400,000
Training Time
4h23min
5h08min
2h57min
3h46min
2h05min
2h33min
11h32min
3h42min
Sh41min
15


Reviews:
[{"review_id": "NLETGiFPqK", "reviewer": "Reviewer_MiHo", "summary": "Offline RL has gained popularity recently as it can be trained with offline batched data without interacting with a simulation environment. Constrained offline RL further extends the idea by adding threshold penalty on the entire trajectory cost. While offline RL and constrained on-policy RL are relatively well studied problems, constrained offline RL still remains a challenging problem. The paper proposed to solve this problem by taking ideas from IQL and IDQL (to address OOD problem) and introducing a cost reconstruction model to address the problem with discounted cost in offline setting. Experimental results on Bullet-Safety-Gym toy simulation environment demonstrate that the proposed C2IQL can outperform other baseline methods in terms of improving the reward values while maintaining the cost budget.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1: My only concern is that the threshold penalty is chosen randomly. Adding experiments on real-world problem settings is appreciated.\nThank you for your suggestion. The threshold penalty is choose based on small (<50%), middle (50%), and large (>50%) constraint of the maximum cost, which is comprehensive to cover real-world constraint settings. \nBesides, we sincerely agree that addressing real-world problems is important. However, in most existing SORL benchmarks [1, 2], there are nearly no real-world simulators since SORL is a newly rising field in recent years. Thus we will consider it in our future work.\nTo address your concern, we have expand experiments on other environments in [2]. Please refer to Q1 of **reviewer ojm5** for detailed results due to character limit.\n\n[1] Gronauer, S. (2022). Bullet-safety-gym: A framework for constrained reinforcement learning.\n\n[2] Ji, J., Zhang, B., Zhou, J., Pan, X., Huang, W., Sun, R., ... & Yang, Y. (2023). Safety gymnasium: A unified safe reinforcement learning benchmark. *Advances in Neural Information Processing Systems*, *36*, 18964-18993.\n### Q2: Theorem 1 and 2 miss quality guarantees and convergence proofs.\nWe appreciate the reviewer recognizing the insights provided by theorem 1 and 2, which highlight our contribution avoid the OOD problem and address the discounted formulation problem in constrained settings. Regarding the quality guarantees and convergence proofs, we think our method share the same results as CPQ theoretically. The sketch of the proof is given as follows: First, we have that as the expectile parameter $\\kappa$ increase to 1,  $V^\\pi_r(s)$ can approach $\\max_aQ^\\pi_{r|c}(s,a)$ because the expectile regression is a convex function. Second, convergence proofs of our algorithm can follow similar steps as those in CPQ because CPQ utilize $\\max_aQ^\\pi_{r|c}(s,a)$ while C2IQL utilize $V^\\pi_r(s)$. \n### Q3: Adding a few real-world examples to motivate constraint offline RL in introduction.\nThank you for your suggestion. We add a few real-world examples on the right part of line 24, page 1, second paragraph, to further strengthen our motivation:\n\"… in safety-critical scenarios. **For example, unsafe operations could harm patients in healthcare, unsafe driving style may lead to accidents, and unsafe decisions may incur additional costs in financial investments.** In these situations, …\"\n### Q4: Adding a brief high-level summary will be helpful to follow the theoretical results and algorithms.\nThank you for your suggestion. We will add a high-level summary at the beginning of section 3.1:\n\"To derive a concrete CIQL algorithm, we need to answer three questions: **First, how to update the constrained reward value/Q-value function following IQL style?** To address this problem, CIQL formulates a constraint-penalized reward Q-value function following CPQ and utilize a value function with expectile regression to approximate the maximized Q-value function in Bellman backup procedure. **Second, how to update the cost value function under the same implicit policy since it is hidden in the reward value function?**  To address this problem, we rederive CIQL and obtained the formulation of the implicit policy in theorem 1 following IDQL, and then derive the formulation of the cost value function with this implicit policy. **Third, How to extract the policy?** We extract the policy in an expectile way following Equation 18.\"\n### Q5: Runtime analysis is needed.\nWe record the training time of our proposed C2IQL and baselines in AntCircle scenario in the following table:\n| Algorithm     | BCQ-Lag | Bear-Lag | COptiDICE | CPQ     | FISOR   | VOCE    | CDT      | C2IQL   |\n| ------------- | ------- | -------- | --------- | ------- | ------- | ------- | -------- | ------- |\n| Training Time | 4h23min | 5h08min  | 2h57min   | 3h46min | 2h05min | 2h33min | 11h32min | 5h41min |\n\nOverall, the training cost of C2IQL is reasonable when compared to other methods. While it takes slightly more time than methods like FISOR and VOCE, it is still significantly faster than CDT, which has the highest training time. Notably, C2IQL achieves a remarkable balance between computational efficiency and performance. The additional training time is justified by the significant performance gains provided by C2IQL, making it a practical and effective solution\n### Q6: About baseline configuration?\nAll baseline algorithms follow the default parameters, which are already finetuned in OSRL [1] projects for best performance. As for FISOR and VOCE, we finetuned the parameter to make sure the performance keeps consistent with the original paper. However, under the same environment but different constraints, we keep the same parameters for all methods including our proposed C2IQL.\n\n[1] Liu, Z., Guo, Z., Lin, H., Yao, Y., Zhu, J., Cen, Z., ... & Zhao, D. (2023). Datasets and benchmarks for offline safe reinforcement learning. *arXiv preprint arXiv:2306.09303*.", "claims_and_evidence": "Constrained offline RL is a challenging problem. The paper claims to solve this problem with cost reconstruction and IQL method. The major claims in improving the reward and cost satisfaction have been proven theoretical and experimentally.", "methods_and_evaluation_criteria": "The proposed evaluation criteria in safety-gym environment are a standard practice in offline RL problems. The evaluation criteria also make sense to me. My only concern is that the threshold penalty is chosen randomly in experiments. Adding experiments on real-world problem settings or a simulated version of some real problem would have been appreciated.", "theoretical_claims": "Theorem 1 and 2 (extended from IQL) gives reasonable insights on how to derive the algorithm, but quality guarantees and convergence proofs are missing.", "experimental_designs_or_analyses": "Experiments are designed carefully with standard practice on safety-gym environments. C2IQL is also compared with several SOTA techniques. Ablation studies were conducted to demonstrate the value for different steps proposed in C2IQL. Overall, the results seem complete, but adding some results on real-world problems with noisy offline transition data would have been more appreciated.", "relation_to_broader_scientific_literature": "Constrained offline RL is not a well explored problem yet, and the real-world applicability seems to be in niche areas. Therefore, it would be of interest to a limited scientific community.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Constrained offline RL is a challenging problem and the paper addressed the main challenges such as OOD problem and discounted cost reconstruction is a valuable contribution. Experimental results also seem to cover the key areas of evaluation. Here are some concerns though:\n1. The motivation behind constraint offline RL is somewhat missing. Adding a few real-world examples in the introduction will be helpful.\n2. Experiments are only conducted in simulation environment with randomly generated cost trajectories and threshold. Therefore, it is hard to predict how such techniques will perform in real-world scenarios with noisy transition sample trajectories. \n3. Convergence proof and computation complexity or runtime analysis is largely missing.", "other_comments_or_suggestions": "Overall, the paper is written well and easy to follow. However, it is hard to follow the theoretical results and algorithms, so adding a brief high-level summary will be helpful to readers.", "questions_for_authors": "1. Please provide a few real-world examples of constrained offline RL.\n2. Have you done any runtime analysis of C2IQL? Can you provide the results for different methods considered in experiments. \n3. Have you finetuned parameters of different baseline algorithms or just took some random configuration?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r0sAyg0I8A", "reviewer": "Reviewer_ojm5", "summary": "This work focuses on the offline safe RL, where the existing baseline methods often suffer from the OOD issues (as in the general offline RL). To do so, this paper proposes the C2IQL method that employs the cost reconstruction model to derive non-discounted cumulative\ncosts from discounted values and incorporates a flexible, constraint-conditioned mechanism to accommodate dynamic safety constraints. This work also provides some empirical evidence on the Bullet-Safety-Gym environments.\n\n## update after rebuttal\n\nMy concerns has been addressed by the authors and I have updated my score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1: More complex benchmark is needed.\nThank you for your suggestion. To address the reviewer’s suggestion, we have also incorporated additional experiments on the SafetyGymnasium benchmark to diversify the test tasks. Specifically, we have selected the 4 Point and 3 Velocity tasks as additional benchmarks different from Bullet-Safety-Gym. Given limited time, we have included results of C2IQL and two strong baselines (FISOR and CDT). The bold results are **unsafe** cases.\n|Env||C2IQL|CDT|FISOR|\n|-|-|-|-|-|\n|PointCircle1|R|0.666 (0.09)|0.562 (0.04)|0.651 (0.04)|\n||C|0.772 (0.11)|0.628 (0.23)|**1.230 (0.44)**|\n|PointGoal1|R|0.745 (0.01)|0.689 (0.01)|0.612 (0.01)|\n||C|0.924 (0.05)|**1.033 (0.26)**|0.474 (0.03)|\n|PointPush1|R|0.343 (0.02)|0.252 (0.02)|0.244 (0.02)|\n||C|0.464 (0.05)|0.447 (0.12)|0.114 (0.08)|\n|PointButton1|R|0.343 (0.09)|0.519 (0.02)|0.063 (0.01)|\n||C|0.913 (0.33)|**1.631 (0.87)**|0.018 (0.02)|\n|HalfCheetahVelocity|R|0.985 (0.02)|0.975 (0.02)|0.855 (0.01)|\n||C|0.512 (0.11)|0.061 (0.04)|0 (0)|\n|HopperVelocity|R|0.791 (0.04)|0.715 (0.02)|0.153 (0.02)|\n||C|0.344 (0.08)|0.422 (0.24)|0.029 (0.02)|\n|AntVelocity|R|0.996 (0.01)|0.986 (0)|0.832 (0)|\n||C|0.575 (0.18)|0.46 (0.12)|0 (0)|\n\nThe additional evaluation on SafetyGymnasium demonstrates that our method, C2IQL, achieves nearly a 10% improvement in performance for most tasks while satisfying constraints, further validating its superior performance.\n### Q2: No discussions for Theorems.\nThank you for your suggestion. We will an explanation of Theorem 1 in line 255 to illustrate how it can be utilized in Theorem 2: \n\"Theorem 1 provides us with a relationship between the constraint-penalized reward function and corresponding formulation of the implicit policy. When the implicit policy formulation follows Equation (14), the update of the value function under implicit policy $\\pi_{imp}(a|s)$ is equivalent to the update formulation of the value function in Equation (13) under behavior policy (the policy for collecting the dataset). Thus we can utilize the formulation of the implicitly policy obtained in Theorem 1 to derive how to update the cost value function based on its definition in Theorem 2.\"\n### Q3: How do the authors obtain the results of baselines? \nFor BCQ-Lag, BEAR-Lag, COptiDICE, CPQ, and CDT, we use the code provided by the OSRL library. For FISOR and VOCE, we use the official code released by the respective papers. To ensure a fair comparison, we run each method across 10 evaluation episodes, using 5 random seeds and 3 different constraint thresholds. \n### Q4: The proposed method has very limited improvement to be honest. \nThe performance gain only appears marginal to CDT because the unsafe results in smaller constraints is averaged by safe results in larger constraints since Table 1 is averaged out 3 constraint thresholds following the style of most existing work. Thus we add Figure 2 as supplement of Table 1 to illustrate that:\n1. CDT **cannot achieve safe policy for small constraints (like L<30)** under some tasks. This is very fatal disadvantage since satisfying the constraint is the foundation of safety.\n2. CDT **cannot achieve reward maximization for large constraints** (like L<70) compared to C2IQL.\n3. C2IQL achieves **best and safe performance for all three constraints**. This indicates C2IQL provides substantial benefits in handling a wider range of constraints and reward maximization accordingly.\n### Q5: The proposed method is not beating baselines in other OSRL work [1].\nThank you for bringing up this concurrent work. We would like to address the reviewer’s concern in the following aspects:\n\n**Baseline Algorithm Selection Criteria:** We carefully select strong-performing methods (e.g, CDT (2023), VOCE (2023), FISOR (2024), WSAC (2024) from reviewer qPih) from recent publications in SORL field to ensure a comprehensive comparison. Extensive experiments on Bullet-Safety-Gym and SafetyGymnasium benchmarks have demonstrated the superior performance of our proposed C2IQL. Regarding the concurrent work [1], we excluded it from our comparison because it was only posted on arXiv in 12/2024 and has not undergone the peer review process.\n\n**Compare with [1]:**\n1. **Performance**: By directly comparing the results from Table 5 in [1] and Table 1 in our paper, we observe that C2IQL provides stronger performance compared to [1]: \n- C2IQL achieves better results on AR, AC, BR, BC, CC, DR, and DC, demonstrating its strong performance across a variety of tasks.\n- C2IQL exhibits slightly worse performance on CarRun compared to [1].\n2. **Method**: [1] addresses issues with poorly behaved policies under global cost constraints. In contrast, C2IQL focuses on the OOD problem and is the first method designed to avoid the OOD problem in SORL. Besides, we also address the discounted cost formulation problem, which further improves constraint satisfaction and policy effectiveness.\n\n[1] Offline Safe Reinforcement Learning Using Trajectory Classification.", "claims_and_evidence": "The experimental results are less convincing. Please see the weakness and questions below.", "methods_and_evaluation_criteria": "The benchmark environments are commonly considered in OSRL. However, the authors use Bullet-Safety-Gym only. Other more complicated environments like SafetyGymnasium, MetaDrive should be considered as well.", "theoretical_claims": "Theorems look reasonable, but I didn't carefully check the proofs.", "experimental_designs_or_analyses": "Some of the experimental results are less convincing and shows limited improvements than the baselines. Please see the weakness below.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "1. The idea is interesting.\n\n2. I think the main weakness is that the proposed method is only evaluated on Bullet-Safety-Gym, which is relatively simple. DSRL has other more complicated environments like SafetyGymnasium, MetaDrive.\n\n3. There are no discussions/comments for the Theorems.\n\n4. How do the authors obtain the results of baselines, e.g., in Table 1? a) The proposed method has very limited improvement to be honest. b) The proposed method is not beating the baselines that are mentioned in other OSRL work, e.g., see Table 5 of the below work\n\nGong, Z., Kumar, A., & Varakantham, P. (2024). Offline Safe Reinforcement Learning Using Trajectory Classification. arXiv preprint arXiv:2412.15429.", "other_comments_or_suggestions": "1. I suggest that the authors should focus on C2IQL, maybe no need to say that much about CIQL, especially maybe don't treat it as a new method but part of the C2IQL?", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ltc3B5ED3v", "reviewer": "Reviewer_QM2i", "summary": "The paper introduces Constraint-Conditioned Implicit Q-Learning (C2IQL), a novel approach for Safe Offline Reinforcement Learning (SORL) that improves constraint satisfaction while maximizing rewards. The key innovations include a Cost Reconstruction Model (CRM), which estimates non-discounted cumulative costs to improve safety, and constraint-conditioned learning, which allows policies to dynamically adapt to different safety thresholds. The authors provide theoretical results supporting their approach and conduct experiments on Bullet-Safety-Gym tasks, demonstrating that C2IQL outperforms existing safe RL baselines in terms of both constraint satisfaction and reward maximization.\n\n## update after rebuttal\nI appreciate the authors’ detailed response and the additional experiments. The new SafetyGymnasium results and the clarifications on the theoretical derivations (particularly how Theorem 1 supports cost value updates in Theorem 2) help address my main concerns. The runtime and baseline setup explanations are also helpful. In short, I find the core contributions meaningful for the SORL community. I maintain my score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1: An experiment of how errors/inaccuracies in cost estimation impact safety guarantees is needed.\nThank you for your suggestion. To address your concern, we have conducted additional experiments to analyze the impact of cost reconstruction error on policy performance and safety guarantees. \nTo simulate increasing cost reconstruction error, we introduced noise sampled from a normal distribution, $N(0,Noise)$, to the input of the cost reconstruction model. When the added noise is small, the performance (reward) slightly improves as the noise level increases. This occurs because small amounts of noise act similarly to data augmentation, which enhances the generalization capability of the cost reconstruction model. This is analogous to adding noise to input data in image processing to improve robustness. As noise increases from 1.0 to 4.0, the performance becomes progressively more conservative. Despite this, the reward and cost metrics degrade gracefully, indicating the robustness of our method.\n| Noise | 0.0 | 0.1 | 0.2 | 0.3 | 0.5 | 1.0 | 2.0 | 4.0 |\n| - | - | - | - | - | - | - | - | - |\n| Error | 0.05 | 1.05 | 1.33 | 1.50 | 1.83 | 2.1 | 2.4 | 2.8 |\n| R $\\uparrow$ | 434.01 | 435.07 | 442.08 | 435.67 | 435.60 | 421.82 | 408.77 | 401.89 |\n| C $\\downarrow$ | 46.60 | 47.20 | 48.30 | 47.30 | 47.89 | 42.10 | 34.90 | 36.20 |\n\n### Q2: The paper suggests that C2IQL mitigates the OOD problem, there is no theoretical guarantee on how well it generalizes to unseen cost distributions beyond the dataset.\nThank you for your question. \n\n1, we would like to clarify that C2IQL **avoids the OOD problem rather than merely mitigating it**. In SORL, the OOD problem is defined as \"Since the policy may produce OOD actions, the Q-value may be wrongly estimated\". Previous methods address it by constraining the target policy to be close to behavior policy, thus only mitigating the problem. In contrast, C2IQL, following IQL, completely avoids the OOD problem by training the value function using expectile regression, which inherently focuses on in-distribution data during training.\n\n2, **C2IQL does not require generalization to unseen cost distribution** because: 1) C2IQL avoids the OOD problem entirely by ensuring that the training process operates strictly within the dataset distribution. When training, C2IQL avoids relying on any distributions outside the dataset. Even during testing phase, C2IQL naturally tends to select actions that align with the dataset distribution, as it is trained to do so. 2) As for the cost reconstruction model (CRM), they will not encounter unseen cost distribution because CRM is only used during training. In testing phase, CRM is no longer utilized—only the policy is used to generate actions based on the current state and constraint conditions.\n### Q3: The results in Table 1 are a bit difficult to interpret without learning curves.\nThank you for your suggestions. We will add some more learning curves to the appendix. \nTable 1 follows a common style in most SORL papers (e.g., CDT, FISOR, OASIS), where results are presented as final values after training. Additionally, we use color coding to improve interpretability.\nThe primary reason for not including learning curves is due to the nature of offline RL. In SORL, the agent is trained on a pre-collected and fixed dataset without interacting with the environment during training. As a result, monitoring performance during intermediate training steps is not typically necessary, since the agent's performance is only evaluated after the training process is complete.\nThat said, we have included some learning curves in Figure 3 to provide a visualization of policy evolution.\n### Q4: Evaluation of sample efficiency is needed.\nThank you for your suggestions. In Offline RL, the agent is trained on a fixed and pre-collected dataset without any interactions with environments. This indicates that all methods (C2IQL and baselines) require the same amount of data without any new data added into the dataset. The dataset in DSRL usually contains thousands of trajectories. \n### Q5: Algorithm 2 is a bit difficult to interpret, which could consider to improve clarity.\nThank you for your suggestion. Algorithm 2 is indeed a collection of equations previously introduced in Sections 3.1–3.4, but now extended to the constraint-conditioned version. We realize that we did not explicitly indicate their connections to the earlier sections, which may have caused confusion.\n\nTo improve clarity, we will provide additional explanations in Algorithm 2 to the corresponding sections and equations:\n\nLine 6: Cost Reconstruction -> Obtain discounted cost values and reconstruct the non-discounted value with $R^c$ in Section 3.3\n\nLine 9: Constraint Penalization -> Obtain the constraint-penalized Q-value function with Equation (11)\n\nLine 11: Update value function -> Update value function with Equation (12)\n\nLine 14: Update cost value function -> Update cost value function with Equation (17)", "claims_and_evidence": "1. The claim that C2IQL improves constraint satisfaction is supported by experiments where it achieves better safety-performance tradeoffs than baselines.\n2. The claim that CRM improves cost estimation is justified through its design, but there is no formal analysis of how errors in cost reconstruction affect policy learning.\n3. The paper suggests that C2IQL mitigates the Out-of-Distribution (OOD) problem, but there is no theoretical guarantee on how well it generalizes to unseen cost distributions beyond the dataset.", "methods_and_evaluation_criteria": "1. The use of Bullet-Safety-Gym tasks is reasonable for evaluating safety-constrained RL.\n2. The choice of baselines (BCQ-Lag, BEAR-Lag, COptiDICE, CPQ, FISOR, CDT, and VOCE) is appropriate for comparing constraint satisfaction and reward maximization", "theoretical_claims": "1. Theoretical results provide constraint satisfaction guarantees and introduce a cost reconstruction model (CRM) to improve estimation of non-discounted cumulative costs. The approach enhances policy learning by refining constraint enforcement without requiring explicit policy extraction.\n2. However, the method relies on the assumption that cost reconstruction provides sufficiently accurate estimates, and the paper does not formally analyze how errors in this model may propagate through policy learning. A deeper analysis of how inaccuracies in cost estimation impact safety guarantees could strengthen the theoretical foundation.", "experimental_designs_or_analyses": "Advantages:\n1. The experiments demonstrate that C2IQL outperforms baselines in both reward maximization and constraint satisfaction, showing its effectiveness in safe offline RL settings.\n2. The evaluation covers a range of environments from Bullet-Safety-Gym, providing insights into how C2IQL handles safety constraints in practical scenarios.\n\nLimitations:\n1. While the results show improvements, the study does not analyze how well the cost reconstruction model generalizes to unseen cost distributions.\n2. The results in Table 1 are comprehensive but a bit difficult to interpret, as they only provide final values without learning curves. A visualization of policy evolution would improve clarity.", "relation_to_broader_scientific_literature": "1. The paper builds on prior work in offline RL (IQL, COptiDICE) and safe RL (CDT, VOCE).\n2. The CRM approach is novel compared to prior pessimistic offline RL approaches.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "1. The impact of cost reconstruction errors on policy learning is not analyzed, making it unclear how estimation inaccuracies influence performance.\n2. There is no evaluation of sample efficiency, leaving the question of how much data C2IQL requires compared to baselines unanswered.", "other_comments_or_suggestions": "1. Algorithm 2 is a bit difficult to interpret, as the equations are dense. The authors could consider placing key equations in an appendix or breaking them down incrementally within the main text to improve clarity. A more structured explanation before introducing the full algorithm would help readers understand its components more intuitively.\n2. I wonder if the heavy use of abbreviations in the abstract and introduction is necessay, as it affects readability.", "questions_for_authors": "See the above sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Veh8UnR6IF", "reviewer": "Reviewer_qPih", "summary": "This paper considers offline CMDP problem where one needs to maximize the expected cumulative reward subject to the constraint that the expected total cost is below a certain threshold. The paper then proposed constrained conditioned implicit Q learning for safe offline RL. The main novelty is redefining the reward function in a way that only when the policy is safe it is getting reward.  Empirical results have shown that their result is better compared to the state-of-the-art CDT results in most cases.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1: The main novelty is redefining the reward function that only gets rewards when the policy is safe. The contribution part is not clear.\nIn SORL, the most important concern is the OOD problem, while existing methods can only mitigate it via policy constraining but cannot avoid it. The first novelty of this paper is proposing C2IQL to completely avoid the OOD problem in SORL.\n\nThe second novelty is that we may be the first one to propose and analyze the problem of discounted cost value formulation in detail. The contribution is illustrated in line 87, page 2 while redefining the reward function is the novelty of CPQ (2022).\n\nFor better clarity, we will add new content to line 88 page 2 and line 100 page 2:\n\n\"We first propose Constrained IQL (CIQL) to address the OOD problem, which can only be mitigated for existing RL methods in constrained settings.\"\n\n\"As far as we know, we may be the first one to propose and analyze the problem of discounted cost value formulation in detail.\"\n### Q2: References [A1] is not discussed/compared.\nThank you for bringing this paper to our attention. We agree that [A1] is relevant to SORL, and we will include it in the related work section and as a baseline in the experiments. Specifically, we will add the following to line 205, page 4:\n\n\"... problem. WSAC (Wei et al., 2024) focuses on the policy inferior problem and proposes a refined adversarial objective function. OASIS ...\"\n\nAdditionally, we compared WSAC with C2IQL across all environments. The results are summarized below where bold results are unsafe:\n\n|Algo||AR|BR|CR|DR|AC|BC|CC|DC|Avg|\n|-|-|-|-|-|-|-|-|-|-|-|\n|WSAC|R|0.25|0.804|0.86|0.659|0.40|0.69|0.61|0.024|0.537|\n||C|0.179|**1.98**|0.40|**2.518**|0.98|0.78|0.51|0.45|0.97|\n|C2IQL|R|0.74|0.59|0.95|0.71|0.66|0.72|0.74|0.78|0.74|\n||C|0.94|0.95|0.08|0.73|0.76|0.85|0.93|0.85|0.76|\n\nC2IQL achieves better performance while satisfying constraints on all tasks.\n### Q3: The improvement compared to the CDT seems to be marginal. \nThe performance gain only appears marginal because **the unsafe results in smaller constraints are averaged by safe results in larger constraints** since Table 1 is averaged out 3 constraint thresholds following the style of most existing work. Thus we add Figure 2 as a supplement of Table 1 to illustrate that:\n1. CDT **cannot achieve safe policy for small constraints (like L<30)** under some cases. This is very fatal disadvantage since satisfying the constraint is the foundation of safety.\n2. CDT **cannot achieve reward maximization for large constraints (like L<70)** compared to C2IQL.\n3. C2IQL achieves best and safe performance for all three constraints. This indicates C2IQL provides substantial benefits in handling a wider range of constraints and reward maximization accordingly.\n### Q4: The cost reconstruction part is not clear. How did the paper reconstruct the cost by finding the value function corresponding to different discount factors?\nWe would like to provide the following clarifications:\nThe definition of the cost value function is a linear function where $\\gamma$  is the constant and  $c(.,.)$ are variables.\n\n$V_c^\\pi(s_t)=\\sum_{j=0}^T\\gamma^jc(s_{t+j},a_{t+j}\\sim\\pi)$\n\nThe goal is to reconstruct the non-discounted cost value:\n\n$\\hat C^{\\pi}(s_t)=\\sum_{j=0}^Tc(s_{t+j},a_{t+j}\\sim\\pi)$\n\nTo approximate the non-discounted cost value, we construct a set of cost value functions with different discount factors:\n\n$V_c^{\\pi,i}(s_t)=\\sum_{j=0}^T\\gamma^j_ic(s_{t+j},a_{t+j}\\sim\\pi),i=1,...,m$\n\nUsing these multiple discounted value functions, we train a supervised learning model to estimate the non-discounted cost value, where the inputs are estimated $V_c^{\\pi,i}(s_t)$ and the output is $\\hat C^{\\pi}(s_t)$. \n### Q5: Should provide theoretical guarantee.\nPlease refer to **Q2 of reviewer MiHo** for theoretical analysis due to character limit. The mathematical proof are out of the scope of this paper. Besides, C2IQL has good empirical results including additional experiments on SafetyGymnasium from **Q1 reviewer ojm5**.\n### Q6: Question about equation (3) and (4).\nYes, equation (3) lets the infeasible reward Q-value to be 0 and equation (4) indicates only feasible policy is updated, which are the key ideas of CPQ (2022). However, this is not what C2IQL should be focused on. While your suggestion to multiply the indicator function with the reward is reasonable and could improve CPQ, it is outside the focus of our work, as we adopt CPQ’s framework without modification. Investigating this idea further would be interesting for future work.\n### Q7: About 4 different loss functions.\nCompared with existing safe RL algorithms, we only add one loss to train the cost reconstruction model (CRM). In safe RL, value loss, cost value loss, and policy loss are essential. For the CRM, it is trained separately with the dataset and some randomly generated data. It is used in C2IQL after the loss is minimized and stable. Thus the training stage is not influenced.", "claims_and_evidence": "The paper seeks to show the value of the constrained condition implicit Q learning for finding safe policies with good reward. The main difficulty is how to train the new model. In particular, they use a novel reward and a novel cost reconstruction for different parameters of the discount function. The claim is that they achieve a better policy compared to the state-of-the-art approaches has been backed up by empirical results.", "methods_and_evaluation_criteria": "They used detailed experimentation on well-known benchmark problems to validate their claims. The evaluation criteria are the rewards attained and whether the policy is safe or not. I do not find any weakness in the evaluation criteria.", "theoretical_claims": "The two theoretical results are proved. They are pretty straightforward and there is no issue.", "experimental_designs_or_analyses": "Experimental design is solid. However, the improvement over the CDT approach is very minimal. See the questions for detailed explanation.", "relation_to_broader_scientific_literature": "The paper seeks to contribute to the space of the safe offline RL algorithm. However, the paper has not compared with one of the most prominent literature. Hence, the contribution part is not clear.", "essential_references_not_discussed": "The main drawback of this paper is that essential references have not been discussed. In particular, the paper did not compare with the following paper [A1] which also proposed a novel approach for the safe offline RL and has shown better results compared to CDT.\n\nFurther, [A1] considered the following objective $V_r^{\\pi}+\\lambda (V_{c}^{\\pi}-b)_+$,\n\nhence, only when the policy is feasible it adds to the objective, otherwise, it is not which is in the similar flavor of the approach proposed in the paper. It is true that this paper uses the Implicit Q learning, however, [A1] shows provable safety and sub-optimality gap. **Overall, the contribution part is not clear.**\n\n\n[A1]. Wei, Honghao, Xiyue Peng, Arnob Ghosh, and Xin Liu. \"Adversarially Trained Weighted Actor-Critic for Safe Offline Reinforcement Learning.\" In The Thirty-eighth Annual Conference on Neural Information Processing Systems.", "other_strengths_and_weaknesses": "Weakness:\n\nThe improvement compared to the CDT seems to be marginal. Hence, it is not clear the benefits of the proposed approach.\n\nThe cost reconstruction part is not clear. \n\nThe paper did not provide any theoretical guarantee on the safety or the sub-optimality gap.", "other_comments_or_suggestions": "**Post-Rebuttal**\n\nI am happy with the responses provided by the authors and am more confident about the contributions. Thus, I have raised my score. I would encourage the authors to add those discussions (especially, the need for reconstruction of cost, and the augmentation of the state) in the final version. It would be super important.", "questions_for_authors": "1. How did the paper reconstruct the cost by finding the value function corresponding to different discount factors?\n\n2. The overall idea is to solve equation (4). However, how is it related to (3), should the reward be $0$ if the policy is not feasible? In particular, the indicator function should be multiplied with the reward as well. \n\n3. The paper has 4 different loss functions which might be tedious for training.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper addresses the offline learning problem in constrained Markov Decision Processes (MDPs), where the objective is to maximize the expected cumulative reward while ensuring that the expected total cost remains below a specified threshold. The novel contribution is a Cost Reconstruction Model (CRM), which estimates non-discounted cumulative costs, allowing policies to adapt dynamically to varying safety thresholds.

Overall, the reviewers agree that this work is technically sound and has the potential to contribute to the existing reinforcement learning literature. However, they noted that the simulations were conducted in a simple synthetic environment, raising concerns about the applicability of the results to more complex settings. Additionally, there was insufficient discussion regarding the proposed theorems. The authors have partially addressed these concerns by including additional experiments on more complex environments and providing a more detailed comparison with baseline methods during their discussion. It is recommended that the authors incorporate these additional details into the camera-ready manuscript.
Decision: Accept (poster)