Offline Guarded Safe Reinforcement Learning for Medical Treatment Optimization Strategies
Authors: Runze Yan
Xun Shen
Akifumi Wachi
Sebastien Gros
Anni Zhao
Xiao Hu
Venue: NeurIPS 2025 spotlight ()
Keywords: Health informatics; Offline reinforcement learning; Safe reinforcement learning; Mathematical optimization;
Primary area: machine_learning_for_sciences
TL;DR: We propose an offline guarded safe RL framework for medical treatment, providing theoretical safety guarantees and improving both clinical consistency and treatment outcomes.

Abstract: When applying offline reinforcement learning (RL) in healthcare scenarios, the out-of-distribution (OOD) issues pose significant risks, as inappropriate generalization beyond clinical expertise can result in potentially harmful recommendations. 
While existing methods like conservative Q-learning (CQL) attempt to address the OOD issue, their effectiveness is limited by only constraining action selection by suppressing uncertain actions. This action-only regularization imitates clinician actions that prioritize short-term rewards, but it fails to regulate downstream state trajectories, thereby limiting the discovery of improved long-term treatment strategies. To safely improve policy beyond clinician recommendations while ensuring that state-action trajectories remain in-distribution, we propose \textit{Offline Guarded Safe Reinforcement Learning} ($\mathsf{OGSRL}$), a theoretically grounded model-based offline RL framework. $\mathsf{OGSRL}$ introduces a novel dual constraint mechanism for improving policy with reliability and safety. First, the OOD guardian is established to specify clinically validated regions for safe policy exploration. By constraining optimization within these regions, it enables the reliable exploration of treatment strategies that outperform clinician behavior by leveraging the full patient state history, without drifting into unsupported state-action trajectories. Second, we introduce a safety cost constraint that encodes medical knowledge about physiological safety boundaries, providing domain-specific safeguards even in areas where training data might contain potentially unsafe interventions. Notably, we provide theoretical guarantees on safety and near-optimality: policies that satisfy these constraints remain in safe and reliable regions and achieve performance close to the best possible policy supported by the data. When evaluated on the MIMIC-III sepsis treatment dataset, $\mathsf{OGSRL}$ demonstrated significantly better OOD handling than baselines. $\mathsf{OGSRL}$ achieved a 78\% reduction in mortality estimates and a 51\% increase in reward compared to clinician decisions.

Offline Guarded Safe Reinforcement Learning for Medical Treatment Optimization Strategies
Runze Yan
Akifumi Wachi Sebastien Gros* Anni Zhao Emory University, Tooyo University of Agriculture and Technology, 3 'LY Corporation, "Norwegian University of Science and Technology runze yandemory. edu, shendgo. tuat. tuat.ac.
Xun Shen
Xiao Hu
Abstract
When applying offline reinforcement learning (RL) in healthcare scenarios, the out-of-distribution (OOD) issues pose significant risks, as inappropriate generaliza- tion beyond clinical expertise can result in potentially harmful recommendations. While existing methods like conservative Q-learning (CQL) attempt to address the OOD issue, their effectiveness is limited by only constraining action selection by suppressing uncertain actions. This action-only regularization imitates clinician actions that prioritize short-term rewards, but it fails regulate downstream state trajectories, thereby limiting the discovery of improved long-term treatment strate- gies. To safely improve policy beyond clinician recommendations while ensuring that state-action trajectories remain in-distribution, we propose Offline Guarded Safe Reinforcement Learning (OGSRL), a theoretically grounded model- based offline RL framework. OGSRL introduces a novel dual constraint mechanism for improving policy with reliability and safety. First, the OOD guardian is established to specify clinically validated regions for safe policy exploration. By constraining optimization within these regions, it enables the reliable exploration treatment strategies that outperform clinician behavior by leveraging the full patient state history, without drifting into unsupported state-action trajectories. Second, we introduce safety cost constraint that encodes medical knowledge about physiolog- ical safety boundaries, providing domain-specifer safeguards even in areas where training data might contain potentially unsafe interventions. Notably, provide theoretical guarantees on safety and near- searr-ptimality: policies that satisfy these constraints remain in safe and reliable regions and achieve performance close the best possible policy supported by the data, When evaluated on the MIMIC-III sepsis treatment dataset, OGSRL demonstrated significantly better OOD handling than baselines. OGSRL achieved a 78% reduction in mortality estimates and a 51% increase reward compared to clinician decisions.
Introduction
Deep reinforcement lcarning (RL) has been widely applied in many safity-critical domains, such fine-tuning of language models 9 35 robotics 7 8 and autonomous driving (21). Given its
capacity to learn from large-scale real-world datasets, there is growing interest in leveraging deep RL for decision support in medical treatment. Notably, deep RL has been explored for treatment opti- mization various clinical conditions, including sepsis 22 37], cancer [47], and type 2 diabetes [56). In medical applications, unlike conventional RL, two additional challenges must be addressed. First, medical treatment optimization is not amenable to learning via active interaction; that is, online exploration of treatment alternatives for patients is strictly prohibited. Second, medical treatments
R. Yan and X. Shen contributed equally this work.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

multi-faceted: we need to incorporate (possibly conflicting) safety constraints and reward function. Even if treatment is highly effective, therapies with severe side effects are undesirable for patients.
Offline RL learns policies from pre-collected datasets without further environment interaction (28) making it ideal for medical treatment optimization, where real-time experimentation is ethically constrained. Early healthcare applications relied on value-based off-policy methods such as
17 39 53). They face challenges in offline settings due to OOD actions [25 and Q-value overestimation for unseen actions (2), leading to unsafe or suboptimal decisions. Conservative Q-learning (CQL) [26] mitigates OOD action overestimation by penalizing value estimates for actions not present in the dataset and has been applied to clinical [3] CQL focuses solely on suppressing OOD actions, leaving OOD states unaddressed [1 As policies evolve, even in di-striibuutinn actions can lead to state trajectories that diverge from data distribution. This is problematic in healthcare, where accurate modeling of state transitions is critical, and OOD states may correspond to unsafe or clinically invalid patient conditions. Prior methods fail to fully leverage clinician expertise embedded in dataset. CQL can only encourage policies that imitate clinician actions but cannot improve upon them because it lacks mechanisms to safely explore or optimize within the full state-action support derived from expert trajectories.
DQN 22 37 54 and its variants [4
Contributions. We propose Offline Guarded Safe Reinforcement Learning (OGSRL), a theoretically grounded framework for learning safe and effective treatment policies from offline clinical data. Our key contributions are as follows. (1) We introduce an OOD guardian that jointly constrains policies to remain within the state-action support and enables optimization within this region. Unlike prior methods such as CQL. that only suppress OOD actions, OGSRL explicitly restricts both states and actions, fully leveraging clinician knowledge embedded in the dataset and incorporating explicit safety cost constraints to avoid risky recommendations. (2) We provide theoretical guarantees that any policy satisfying the OOD cost constraint remains in-distribution with high probability. When combined with model-based RL, OGSRL further offers probabilistic guarantees on safety and near-optimality, and quantifies the effect of dataset size on policy reliability. (3) We demonstrate the practical effectiveness of OGSRL on real-world sepsis treatment data. OGSRL consistently outperforms strong offline RL baselines in cumulative reward, safety constraint satisfaction, and alignment with clinical behavior.
2 Problem Statement
Modeling medical treatment as a CMDP. We define the patient state as s E S C IR" and the permissible treatment action as a E A S R The patient state evolves according to transition dynamics T(s* s, a), which specifies the distribution over the next state s* given the current state s and action a. A reward function r : S x A A- [0, rmax] is defined based on clinical health indicators, which reflects the treatment objective of improving patient health. In addition to reward, certain safety indicators must be considered during treatment. These are encoded by vector-valued safety cost function S x A- -> [0, cL.xx) x ... x [0, Ct.max), where Cmax = [C1, (C1.max:--> max) denotes the upper bounds for safety-related quantities. At each decision step h, a clinician observes the current patient state S; and selects treatment as aimed at improving the patient's 's condition (maximizing r) while avoiding unsafe outcomes (ensuring each component of c remains within safe limits). Thus, medical treatment can be formulated as a constrained Markov decision process (CMDP) by M := (S. A. T, r. C, y, Pu), where y E (0, 1] is a discount factor and Po is the probability density of the initial patient state So, typically reflecting the variety of conditions the time of ICU admission or treatment onset. A treatment policy is a stochastic mapping from the state to the probability density over admissible treatment actions. Let T(. s) denote the probability density of a when the state is s, represent a trajectory II. The value function associated with a bounded function o : S A IR (e.g., reward r or a safety cost component 6,) under policy π and transition dynamics T is defined by (s) = E, E > o (Sn, an)) 8o = s). Here, o is assumed to be bounded by omax- The expected value across patients is defined as (P)r(po) == Es-pu (s)].
and let IT be the space of all such policies. Let r := (So, a Sh. -
induced by policy
Example scenario. Consider the treatment of sepsis. Conventional studies of using RL to optimize sepsis treatment have not considered safety constraints either for the action or for physiological
states that a safe treatment action should always maintain. Early studies [22 36 45 of sepsis treatment used mortality as the only penalty (negative reward) to guide the learning process, but recent
work 14 15 19 53 started using composite scores, such as the Sequential Organ Failure Assessment
(SOFA), as the negative or the reciprocal of the reward. While SOFA combines multiple organ function

---

Algorithm 1 OGSRL: Offline Guarded Safe Reinforcement Learning for Treatment Recommendation
1: Input Initial dataset Db collected under standard treatment
2: Learn classifier 9 of the guardian from D to detect safe state-action pairs (see Sec. 3.1
3: Construct guarded treatment model Ma using 9 and Do (see Def.
4: à <- ConOpt(Mg) to compute safe and effective treatment policy
5:
end for
indicators and hence encourages actions that move patient towards normal physiological states, the learning algorithm cannot guarantee that every intermittent physiological state of a patient is indeed safe. In addition, there are readily available variables that used to produce physiologically sound and clinically interpretable safety constraints. Hence, a novel algorithm that capable of learning policies that explicitly obey safety constraints needed.
part of SOFA but can be
The primary objective this paper is to maximize the value function (po), while ensuring that the adopted treatment policy æ should satisfy the safety cost constraints: V I Cj j E [e], where Cj E [0, Cj,max] is the upper constraint for the j-th expected cumulative safety cost. The safe RL problem associated with Po we shall solve written as
Goal.
max V s.t. VG, Ve,,T(P)) EG, #Ell
Vj [4]
(ESRL)
3 Method
We propose framework called Offline Guarded Safe Reinforcement Learning (OGSRL) to learn a treatment policy under safety constraints. The workflow of OGSRL outlined Algorithm [ First, the offline dataset D, = {(s, a. r, c)} is used to estimate the reward function, safety cost function, and transition dynamics, which together define an estimated constrained Markav decision process (E-CMDP). To address the risk of unsafe generalization, we incorporate a guardian into the model-based offline safe RL and construct a guarded E-CMDP. The guardian plays two roles: classification and rejection. A PSoS-based classifier is trained to identify OOD state-action pairs. Using the learned classifier 9. we formulate an OOD cost constraint and insert it into the CMDP. The OOD cost constraint explicitly eliminates policies with a high probability of visiting state-action pairs outside the dataset support. Unlike CQL that primarily suppresses OOD actions, our constraint jointly addresses both OOD states and actions, leading improved generalization and safety. A constrained policy optimizer, denoted as ConOpt, is then used to solve the guarded E-CMDP and compute a policy (+) that maximizes the expected clinical outcome while satisfying the predefined safety constraints and additional OOD constraint. While we employed constrained policy optimization (CPO, I as ConOpt, other constrained RL algorithms are not prohibited from being used.
3.1 Constructing Guardian Classifier
The state-action space 2 := S x A can be partitioned into two regions: the in on-distribution (ID) set Us and the OOD set Ulod == U \ Uig The estimated model is only guaranteed to converge in the II) region Lia To prevent unsafe generalization, we introduce a guardian that classifies whether state-action pair lies outside the support of the data and then restricts policy learning to ID regions.
We first introduce an important notion called polynomial sublevel set, defined as follows.
Definition 1 (Polynomial sublevel set). Let = (s, a) R. with np n + m. Let e(x) denote the vector of all monomials of x up degree d 0, e(x) = [1, II, In, "i, 2d parameter vector 0, define the polynomial function: g(x, 0) == e (x)))((()e(x) where P(0) is
Given
symmetric, positive semidefinite Gram matrix fully determined by O. The degree of q is 2d, and we require q(x,00 0) Z 0 for all x, making it a polynomial sum-of-squan
(SoS) function 124 41
Then, the polynomial sublevel set is given by: Uo := {x E U g(xx 0) S 1}..
Ideally, we desire to obtain the following classifier g S x + {0,1), defined as g(s, a) II ((ssaa) a) & Uha). Unfortunately, the perfect classifier is unknown in practice. We thus aim to approximate this set using a polynomial sum-of-squares (PSoS) classifier, which enables explicit theoretical analysis of the OOD guarantee due to its structured mathematical form. While PSoS
3

---

provides analytic tractability for safety proofs. we use kernel-based approximation in practice to improve scalability and ease of implementation. As such, by learning a polynomial sublevel set Us satisfying Uaea S Ui with high probability, we obtain a conservatively approximated classifier, denoted as 9 : A + {0, 1]: 0(s, a) = 1l { (ss a) & Ug }. where Ug,a is a degree-d polynomial sublevel set parameterized by 0 RR Learning the PSoS guardian g involves estimating the polynomial sublevel set Ug from the dataset Db. Let XN = {x(0) -
1=1 denote the collection of N state-action pairs sampled from Da- Optimization problem for constructing Üa given by:
min L(D)= := log det (0) s.t.
Σ Li (o(x((), .0)(())))))))) S ace N
1
(GCL)
where ae E (0, 1) is an empirical coverage threshold and (z) 1 and otherwise. The objective minimizes the volume of the set, forming a tight envelope around the in-support data. This set later used detect whether a state-action pair out-of-distribution. In practice, In is replaced with smooth surrogate for tractability. While we adopt this PSoS-based classifier for theoretical guarantees, alternative methods such as Kernel Density Estimation (KDE) [6 or Neighes-- (k-NN) scoring 6 can approximate the support and are used in our experiments (Appendix G.3 Let A. denote the solution to this problem, and define the learned set as 2gx "y,."' .d'
Probability bound of guardian classifier learning. medical applications, it is particularly important to use an algorithm with favorable theoretical properties. We now provide a probabilistic guarantee on the accuracy of the learned classifier used the guardian.
Theorem For any probability level a > 0 and any ae > a, there exists a polynomial degree d such that the following holds: Pr (Ugs, .d Uta) S exp (-2N" -a)) That is, with high probability. all points within Ug a lie in the in-distribution region Uig.
The proof of Theorem is provided in Appendix C Theorem I implies that the learned guardian classifier provides high-confidence rejection region, whose conservativeness is explicitly tunable via de and improves with more data. Although our proposed method embeds the guardian classifier into the model-based offline RL, it can also be applied to model-free offline RL.
3.2 Model-based Offline RL with Guardian
Model-based offline RL. In our model-based reinforcement learning framework, we first estimate the following from the offline dataset Dp; reward model f, a vectur-valued safety cost model é, and transition dynamics model T. These models define what refer to Estimated Constrained Markov Decision Process (E- CE-CMDP): M == (S, A. C. po). Given M, the model-based safe reinforcement learning problem is formulated as:
max TEIl
(Po)
s.t.
VT V2, S E3n Vj E [.]
(MSRL)
where V (Po) denotes the expected clinical outcome (e.g., improvement in SOFA score), and each constraint ensures that the expected safety-related cost (e.g., risk of hypotension, organ failure, etc.) remains below a clinically acceptable threshold ēy. Problem MSRRess from the ideal formulation using the true CMDP M, because it relies entirely on estimated models. In practice, the reward and cost functions can be learned using Gaussian process regression (GPR) [48] 49), while the transition dynamics can be estimated using techniques such as, c.g., Gaussian process models [3], generative models 43.. A critical challenge in medical applications is that the offline dataset often covers only a limited subset of the state-action space-i.c.. treatments observed under the standard of care [52] Consequently, the learned policies are reliable only within the distribution of data induced by the behavior policy. Naively applying constrained policy optimization to this E-CMDP can result in over-optimistic value estimates and unsafe treatment decisions, especially in regions not well-covered by the data (30 31. To address this, we introduce state-action guardian in the next step.
Guarded C-CMPP With the learned PSoS classifier g, we define a guarded E- E-CMDP by embedding the OOD-aware safety mechanism directly into the model:

---

Definition 2. A guarded E-CMDP is defined as M3 := == (S, (S, FA,,T....... f. 9. Cg)g), where Eg is threshold limiting the out-of-distribution (OOD) cost. The OOD cost constraint is formulated as:
v (Pu) == Es [V S cgg.
Given this structure, the guarded policy optimization problem is formulated as:
max
V (,7(P)) s.t.
(Po( S Cg. V ( S Cg Vj E E
(GSRL)
TELL
The motivation for introducing the OOD cost constraint is to discourage policies that frequently visit state-action pairs outside the support of the dataset. When the support of the true transition dynamics is unbounded, it is often impractical to enforce striet avoidance of OOD state-action pairs. Instead, a more tractable goal is to ensure that the policy remains within the data support with high probability over finite horizon, formalized as the following joint chance constraint: Pr (ū(s; an) = 0, Ah < H) > 3. However, directly incorporating this joint chance constraint into policy optimization is intractable in most safe RL frameworks. Following the approach of Shen et al. [40], we approximate it conservatively via the OOD cost constraint ( The key idea is that, for given risk level 53, one can select a sufficiently large discount factor ry so that feasibility under the cost constraint implies feasibility under the joint chance constraint. A discussion of this approximation
strategy and practical guidance choosing is provided in Appendix B
With the above notations, we extend the result of Theorem
into policy-level guarantee:
Corollary 1. Let itz be any feasible solution to Problem
GSRL
Then, for a desired confidence
level 8 E (0, 1). if the number of samples satisfies N >
log(1/6 /6
the policy ft ensures that, with
20
probability 1 - 8 B, the agent remains within Uh for all steps h H.
The proof of Corollary is summarized in Appendix D
guarantee safety during online environmental interaction by intervening when unsafe actions are about occur. Our guardian in OGSRL shares a similar goal of constraining behavior that causes OOD issues, but operates entirely offline. Instead of correcting actions during execution, the guardian restricts the feasible policy space during offline optimization, ensuring that learned policies, with high probability, keep state-action trajectories within the dataset support safety during online interactions, our approach provides probabilistic safety guarantees in the offline setting, which is crucial for medical applications where real-time corrections are infeasible.
Connections to shielding methods. Shielding methods 3 23 32
finite horizon. Thus, while shielding ensures pointwise
Practical significance. In the context of medical treatment optimization, Theorem T and Corollary D provide essential probabilistic guarantees: only policies that maintain high probability of remaining within the dataset support over a finite horizon H are considered feasible. Crucially, any policy satisfying the OOD cost constraint operates entirely within regions where the estimated dynamics, value functions, and action-value functions are reliable. This is especially important in clinical settings, where learned policies must avoid poorly supported regions; otherwise, inaccurate modeling in such areas could lead to unsafe or ineffective treatment recommendations. Moreover, the OOD cost constraint is a data-driven proxy for clinical knowledge. Because the dataset reflects real- world clinician behavior, constraining policies to remain within support implicitly aligns the learned strategies with accepted medical practices, enhancing both interpretability and trustworthiness for deployment. However, it is important to note that clinician behavior often reflects safe individual treatment decisions, rather than globally optimal long-term strategies. Human decision-maan making may rely on heuristics or short-term goals, with limited integration of the patients' full historical state. A capable offline RL policy with an effective OOD cost constraint can leverage the full patient state to optimize long-term outcomes, while still adhering to the safe local actions reflected in clinical data. While methods like CQL 266 effectively suppress OOD actions, they do not constrain state transitions. This can be particularly problematic in clinical settings, where clinicians make decisions based on observed patient state trajectories. CQL lacks mechanism to encode this temporal structure, leaving it unable to control or reason about OOD states that may emerge downstream. In contrast, our OOD guardian enables safe policy learning by jointly constraining states and actions, making better aligned with clinical reasoning and safer for real-world deployment.
5

---

3.3 Safety and Sub-optimality with Finite Samples
Value function error. We begin by analyzing the error bound of the estimated value function associated with a function o (e.g., reward r or safety cost eÿ). This section assumes that the tramsition dynamics T are estimated using kernel density estimation (KDE). At the same time, the reward and safety cost functions are known, i.e., 6 = o. This assumption is reasonable in many medical treatment settings, where both reward and safety cost functions are predefined, as is the case in our application study in S For settings where the reward and safety cost functions are unknown, we provide a generalized theoretical analysis in Appendix E where these functions are estimated using GPR. Let h be the bandwidth of the KDE, and assume that the joint density of (s", ,, a) and the marginal density of (s, a) are Hölder continuous with exponent Ç E (0, 1].
Theorem 2. Let be any feasible solution of Problem GSRL Assume the standard KDE conditions
Nhutm + oo and h + 0 as N + Jc. Then, with probability at least 1 - 23 46, the following
holds: V (po) (po) S Ek + EH, where:
n 7max EH (1 - 7)2
8k Omax (7 - -2
log(1/6)
1S (I++v
(1 - 7)
Nh2n+m
Here, Cden is a positive constant depending on the smoothness of the densities, the choice of kernel, and the dimensionality 2n + m.
Theorem 2 can be directly obtained from Theorem 6 in Appendix E by setting 6 6 o = for any (s, a). This bound decomposes the total value function error into parts; (1) Ek from approximation of T. which vanishes asymptotically; (2) EH, due to state-action pairs that fall outside the support of the dataset beyond horizon H. By selecting a sufficiently large dataset size N and conservative OOD threshold Cg. we can ensure small 3 in the chance constraint and thus make ey negligible.
Method of choosing Cg with respect to desired H/ follows 40 50
Safety and sub-optimality. We now define conditions under which the policy output by ConOpt is safe and near-optimal with respect to the true model. We say a policy Tout is & if:
maxj Cy Ve
e, (po) 2 E.. Let #* be the optimal solution to Problem GSRL with safety threshold
Cg. If Taut is computed using tightened threshold é E, and satisfies: V ((o)) we obtain the following guarantee for the true system:
ViTaa S Er,
R,
Theorem 3. I E, Ek + EH, and nout is E,- for Problem GSRL
then Tout is safe
and (E, + 25 + 2E ( -ub-optimall for Problem ESRL with probability at least 23 46.
Practical significance. Theorem guarantees that the learned policy remains safe and optrmml with high probability, even under model approximation and conservative constraints. This is essential in clinical contexts, where decisions must be not only effective but also verifiably safe. Crucially, our approach constrains learning within the support of the dataset, where expert treatment trajectories reside, thus fully leveraging clinician knowledge while avoiding unsafe extrapolation. Unlike prior methods relying on unverifiable assumptions, our result explicitly links dataset size and model error to performance bounds, making well-suited for reliable deployment in clinical workflows. Finally, while our approach and Off-Dynamics RL both use classifiers to influence learning, the goals differ. Our guardian is designed to restrict policy optimization to the in-distribution region for safety guarantees, whereas Off-Dynamics RL uses them for reward shaping or domain adaptation [10].
4 Experimental Validations
We evaluate OGSRL through comprehensive experiments on real-world clinical data validate three key aspects of our framework: (1) the effectiveness of the OOD guardian in constraining policies to in-distribution regions, (2) the ability learn safe and effective treatment policies that improve upon clinician behavior while satisfying physiological safety constraints, and (3) the generalizability across different critical care conditions. We conduct detailed evaluation on sepsis treatment using the MIMIC- III dataset (Sections 4.1 4.2). To demonstrate broader applicability, we validate generalizability on the Synthetic Acute Hypotension Dataset (Section [.3). which represents a different critical care condition with distinct physiological dynamics, temporal resolution, and clinical objectives.

---

Across both validation studies, we instantiate OGSRL using GMB-CPO, a model baaedd variant of Constrained Policy Optimization equipped with our OOD guardian mechanism.
4.1 Sepsis Treatment: Formulation and Experimental Setup
We evaluated OGSRL using 18, 923 ICU stays with sepsis diagnosis from the MIMIC-III dataset and established protocols in Komorowski et al. [22] Patient data were encoded as multidimensional time series with 4-hour intervals, capturing up to 72 hours around the estimated onset of sepsis. Our implementation addresses key limitations in previous approaches to sepsis treatment optimization. Rather than discretizing interventions or combining multiple treatments into a single dimension, we developed a continuous wwo-dimensional action space that separately models intravenous fluid administration (IFA) and maximum vasopressor dosage (MVD), namely [IFA, MVD] E R22 This representation enables more nuanced treatment recommendations, reflecting the clinical reality where physicians simultaneously titrate multiple interventions based on patient response. The state representation emerged from a clinically informed feature selection process, incorporating variables significantly correlated with organ dysfunction. This balanced representation captures essential physiological dynamics while enabling personalized treatment strategies. Totally 13 features are selected the dynamic state, namely, s R Departing from previous work that employed mortality as a terminal reward [22). we adapted the Sequential Organ Failure Assessment (SOFA) score into an instantancous reward signal by setting r S x A + SOFA More details about the definitions for selected dynamic and static features, actions and reward can be found in Appendix G.2 This approach provides more frequent feedback on treatment efficacy and better aligns with contemporary clinical practice. Our approach implements two distinct but complementary safety mechanisms. First, explicit safety constraints are appllied to physiological states by enforeing
18
minimum physiological thresholds for oxygen saturation (SpO2) (2 92%) (38) and urine output (2 0.5 mL/kg/hour) (20). These constraints directly encode clinical knowledge about vital parameter ranges necessary for patient safety. Second, our OOD guardian mechanism addresses a fundamentally different safety concern-th the reliability of model predictions when encountering state-action pairs insufficiently represented in training data. While clinical constraints ensure physiological safety within the model's s assumptions, the OOD guardian prevents the policy from recommending treatments in regions where the model itself may be unreliable, regardless of the predicted clinical outcomes. We implement OGSRL as described in Algorithm [ approximating the PSoS guardian classifier ğ using a kernel-based method (see Appendix G.3) to identify OOD state-action pairs efficiently. For
transition dynamics T, employed a k-nearest neighbor (k-NN) approach as an approximation of KDE, which maintains theoretical consistency while offering practical advantages for clinical time-series data, particularly its robustness to sparse regions in the state space [4 (5I. We use
CPO 1 as the constrained policy optimizer ConOpt, resulting in our full implementation referred to as GMB-CPO, which is model-based (MB) variant of CPO equipped with the OOD Guardian (G). Additional details are provided in Appendix G.4 Note that GMB-CPO is a specific instantiation of the proposed OGSRL framework. As discussed the beginning of Section 3 other constrained reinforcement learning algorithms can also be employed as ConOpt within our framework.
Baseline Algorithms and Evaluation Metries. We evaluated OGSRL against seven baseline algo- rithms spanning model-free and model-based offline RL approaches, and their guardian-enhanced variants prefixed with G: (1) CQL: (2) CQL with Guardian (GCQL); (3) CQL variant (CCQL) pre- sented in [3]: (4) CCQL with constraint satisfaction (GCCQL); (5) MB- TRPO (2);: (6) MB-TRPO with Guardian (GMB-TRPO)
(7) MB-CPO The implementation details for guardian integration with each algorithm are summarized in Appendix G.4 We assess OGSRL against baseline methods across four eritical dimensions that follow logical progression essential for clinical deployment: (I) OOD state avoidance: establishing whether guardian mechanism effectively constrains policies to remain within the clinical data support: (2) clinical alignment: measuring how closely learned policies match clinician decision-making patterns, a prerequisite for interpretability and trust; (3) treatment effectiveness: quantifying improvements in patient outcomes compared to standard care; (4) physiological safety: verifying that policies maintain vital parameters within safe ranges throughout treatment trajectories. For quantitative evaluation, we employed four clinically relevant metrics: Model Concordance Rate (MCR) measuring alignment with clinician decisions, Appropriate Intensifi- cation Rate (AIR) assessing treatment escalation in response to physiological deterioration, Mortality
2 Our source code is available at http:tttsss:::thaaaaaa www htt.a//.....c.ttuooo//aaaaaae---
MIMIC- III dataset: htttp:://pysi.nnt ottg//oottent//nnnicc////////
7

---

Table Performance comparison across methods showing Model Concordance Rate (MCR), Appro- priate Intensification Rate (AIR), Mortality Estimate (ME), and Action Change Penalties (ACP) for vasopressor dosage (MVD) and fluid administration (IFA) (mean + Standard Deviation (SD)). Mean and SD were computed from the results of five different seeds. The symbol f indicates that higher values are better, + indicates that lower values are better, and ++ denotes that closer alignment with the standard of care (SOC) is preferred. MCR should align with SOC within reasonable range.
Method
MCR(1,10
AIR(T. 3)
MEG, 10 10-2)
ACP: MVD(++)
ACP: IFA(+>, 10")
CQL
789 + 5.61
13++0.540
4.86 0.540
4.18 0.129
5.43 + 0.083
GCQL
909 = 2.52
1.17
0.211
3.13 0.033
1.51 +: 0.034
CCQL
827 = 3.12
3.24
0.339
3.74 0.066
4.60 0.027
GCCQL
827 + 3.50
30.2
0.142
3.23 3.23
2.73
MB-TRPO
0.04
0.280
48.1
1670
GMB-TRPO
571 3.37
3.19
2.32 0.491
1.24 0.026
9.85 0.063
MB-CPO
0.055
49.6
50.5
492 1.12
GMB-CPO
549 2.56
44.8
1.38 0.482
4.34 0.0552
6.47
SOC
6.32
4.34
6.48
Estimate (ME) projecting survival outcomes, and Action Change Penalty (ACP) quantifying treatment smoothness over time. Detailed definitions and computational methodology for these metrics are provided in Appendix G.6
4.2 Sepsis Treatment: Results and Discussions
We present results OOD state avoidance and summarize key insights regarding clinical efficacy, defined here as the ability of learned policies to simultancously achieve clinician alignment, treatment effectiveness, and physiological safety. This organization allows focus on the core techni- cal contribution while providing essential context on its downstream clinical implications, with comprehensive quantitative analyses available in Appendix G.7
180
SOC
ME-TRPO
ME-CPO
75
33
-39
-30
(a) CQL
(b) MB- TRPO
(c) CCQL
(d) MB-CPO
GCOL
CNB-TAPO
-108
GCOOL
GHB-CPO
160
-106
(e) GCQL
( GMB- TRPO
(g) GCCQL
(h) GMB-CPO
Figure Results on state distributions by learned policies via different algorithms. Blue points represent the original offline dataset; orange points represent the states visited by the learned policies.
OOD State Avoidance. To visualize the high- ligh-dimensional state distributions, we apply t-SNE dimensionality reduction to project the policy-generaan states and the original clinical dataset onto 2D manifold. commprres the distributions across all evaluated algorithms. Policies learned without the guardian (CQL, MB- TRPO, CCQL, MB-CPO) exhibit significant divergence from the support of the offline dataset, with many states falling outside the distribution of the training data. In contrast, guardian-augmented policies (GCQL, GMB-TRPO, GCCQL, GMB-CPO) maintain state distributions tightly concentrated around the dataset support, visually validating our theoretical guarantees OOD state avoidance (Theorem I and Corollary D.. This visualization confirms the

---

AG4
COO $750
(a)
(c)
(d)
Figure 2: Comparison of cumulative reward distributions between the SOC (green) and policies by different algorithms with guard mechanisms (blue). Each subplot shows the estimated reward density for trajectories in the test set. Dashed vertical lines indicate the mean rewards. (a) CQL vs. GCQL;(b)
CCQL vs. GCCQL; (c) MB-TRPO vs. GMB-TRPO, (d) MB-CPO vs. GMB-CPO.
core premise of our approach. Despite effective mitigation of OOD actions by CQL and CCQL, without explicit guardian mechanisms, their learned policies still induce OOD states during trajectory rollouts. Integrating the proposed guardian restricts policies to operate within regions where model predictions remain reliable, enhancing the generalization capability of existing approaches.
Clinical Efficacy. Table T presents quantitative performance metrics across all RI. algorithms eval- uated in our study. We observed that the incorporation of guardian mechanism led significant performance improvements. regardless of which underlying RL algorithm was implemented. Specifi- cally, we observed marked improvements in clinician decision alignment (MCR increased from 0.789 to 0.909 in GCQL and from approximately zero to 0.549 in GMB-CPO), as did appropriate interven- tion timing (AIR increased from 0.130 0.305 in GCQL). These improvements directly reflect the guardian's 's ability to constrain policies within climically relevant state-action regions (Corollary [ When comparing model-free versus model-based approaches, we observed complementary strengths. Model-free methods with guardians (GCQL, GCCQL) achieved superior clinician concordance, while model-based guardian approaches (GMB-TRPO GMB-CPO) demonstrated enhanced physiological responsiveness and more concentrated reward distributions (Figure 2). indicating greater robust- ness to patient variability. Notably, GMB-CPO achieved the lowest mortality estimate (0.0138), representing 78.2% reduction compared to the standard of care (0.0632), while simultaneously improving cumulative rewards by 51% compared to SOC. The explicit incorporation safety con- straints physiological states demonstrated effectiveness even without guardian integration. As evidenced by Figure Bal MB-CPO) reduced the number of unsafe states through explicit constraints on SpO2 (2 92%) and urine output (2 0.5 mL/kg/hour), whereas TR-T exhibits concerning deterioration in both physiological states. These results illustrate how explicitly encoded clinical constraints preserve physiological stability throughout treatment. Interestingly, GMB-TRPO despite lacking explicit clinical constraints, also decreased the number of unsafe states by using only the guardian mechanism. The dual-safety mechanism in GMB-CPO, combining explicit physiological with distribution-awan guardian safety constraints, achieved the greatest decrease in unsafe states for urine output and the second-bes, decrease for SpO2. This dual-safety mechanism further enabled GMB-CPO to maintain near-identical action smoothness to clinical practice (ACP of 4.34 versus 4.34 for standard care), as shown in Table [ These improved outcomes demonstrate that GMB-CPO improves policy performance through the OOD cost constraint, consistent with Theorem 3
4.3 Cross-Disease Validation: Acute Hypotension
To evaluate whether OGSRL generalizes beyond sepsis management, we validate our framework on the Synthetic Acute Hypotension Dataset [27]. This dataset represents a different critical care condition with distinct physiological dynamics and clinical objectives. providing meaningful test of cross-disease applicability. The hypotension cohort contains 3,910 ICU stays with 187,680 hourly state-action pairs over 48 hours. Unlike sepsis experiments using 4- 4-hour intervals with 13-dimensional states and SOFA- rewards, hypotension operates on hourly intervals with 18-dimensional states and piecewise linear MAP- rewards. Safety constraints also differ: urine output and lactate levels. We apply the same guardian mechanism and baseline algorithmss with appropriately adjusted hyperparameters. Complete experimental setup details are provided in Appendix H
Results and Analysis Table 2 summarizes performance across key metrics. The guardian mechanism demonstrates consistent benefits across both datasets. For model-free approaches, GCQL achieved

---

488
150
100
Chunge in of Suno Dompand to
12.
15.
of Linaate
-.6
-127
-100
ocat
68-1WP0
MB-CPD
cool
accat
GMa TRP
MB-OPD
GMB-CPD
68
SpO2
(b) Urine Output
Figure 3: Physiological safety assessment of learned policies. We evaluate the safety of learned policies by analyzing critical physiological states: SpO, and urine output. Our assessment compares the percentage of states below defined safety thresholds against SOC. Positive values represent reduction in unsafe states compared SOC, while negative values an increase.
near-perfect clinician aligmment (MCR: 0.973 + 0.002), representing 18% improvement over CQL (0.824 0.004). The impact was even more pronounced for GMB-CPO increased concordance from near-zero (0.060 + 0.008) clinically meaningful alignment (0.700 + 0.063). In terms of clinical safety, GMB-CPO achieved the highest AIR at 0.482 0.071-1 a 17% improvement over MB-CPO (0.411 0.036) and substantially higher than both CQL (0.281 = 0.031) and GCQL (0.301 + 0.046). Regarding cumulative rewards, GMB-CPO achieved the best performance (mean: 14.88, median: 16. 14), outperforming MB-CPO (mean: 3.9, median: 5.38), GCQL (mean: 12.42, median: 12.69), and standard of care (mean: 10.37, median: 11.21).
Cross-Disease Con- sistency These results demonstrate three key aspects of cross-disease generalizability: (1) Consistent guardian Guarddan aug-
Table 2: Comparison on Acute Hypotension Dataset (mean +: SD).
Method
MCR (t)
AIR (1)
Reward Mean (f)
CQL
0.824 + 0.004
0.281 + 0.031
10.15 + 1.23
GCQL
0.973 1 0.002
0.301 + 0.046
12.42 1 0.98
MB-CPO
0.060 1 0.008
0.411 + 0.036
3.90 +: 2.14
GMB-CPO
0.700 +:: 0.063
0.482 0.071
14.88 1 1.45
mentation consistently improves all methods across both diseases, with model-based approaches benefiting most dramatically (concordance improved from near-zero to 0.700 for GMB-CPQ CPB both datasets). (2) Robust best per- former-GMB-CPA achieves the best balance of clinician alignment, clinical safety, and treatment effectiveness in both sepsis and hypotension. (3) Mechanism transferability
SOC
10.37 = 1.87
dual-safety mechanism (explicit physiological constraints OOD guardian) proves robust to differences in disease pathophysiology. temporal resolution (hourly vs. 4-hour intervals), state dimensionality (18 vs. 13 features), reward structure (continuous MAP-based vs. discrete SOFA-basede and safety constraints (urine + lactate vs. SpO2 + urine). These cross-disease results establish OGSRL as generalizable framework for safe offline RL in critical care settings.
5 Conclusion
We introduced OGSRL, model-based offline reinforcement learning framework designed for safe and effective medical treatment optimization. By jointly enforcing OOD and safety cost constraints, OGSRL ensures policy learning remains within clinically supported regions while allowing safe performance improvement observed clinician behavior. We established theoretical guarantees safety, near-optimality, and in-distribution containment. We validated OGSRL by evaluating one of its instantiations, GMB-CPO, on real-world sepsis treatment data, showing substantial gains in reward, safety, and clinical consistency, demonstrating the promise of OGSRL for reliable deployment in safety-critical healthcare domains.
10

---

References
[1] Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International Conference on Machine Learning (ICML), 2017.
[2] R. Agarwal, D. Schuurmans, and M. Norouzi. An optimistic perspective on offline reinforcement learning. Proceedings of the 40th International Conference on Machine Learning, PMLR, 37: 104-172 114, 2020.
[3] M. Alshiekh, R. Bloem, Ehlers, Könighofer, S. Niekum, and U. Topeu. Safe reinforcement learning via shielding, In Proceedings of the AAAI Conference on Artificial Intelligence, AAAI-2018, volume 32, pages 5739 0049057799000 2018. doi: 1111699aaai.v32221.11.
[4] Arko Banerjee, Kia Rahmani, Joydeep Biswas, and Isil Dilli, Dynamic model predictive shielding for provably safe reinforcement learning. In Neural Information Processing Systems (NeurIPS), 2024.
[5] Steven Boyd and Lieven Vandenberghe. Convex Optimization. Cambridge University Press, 2004.
[6] M. Döring, L. Györfi, and H. Walk. Rate of convergence of k-nearest-neet neighbr classification rule. Journal of Machine Learning Research,
2017.
[7] L.C. Garaffa, M. Basso, A.A. Konzen, and E.P. de Freitas. Reinforcement learning for mobile robotics exploration: A survey. IEEE Transactions on Neural Networks and Learning Systems, 34(8):3796-3810 2023.
[8] Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off off-policy updates. In 2017 IEEE international conference on robotics and automation (ICRA), pages 3389-3396. IEEE, 2017.
[9] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501. 2025.
[10] Yihong Guo, Yixuan Wang, Yuanyuan Shi, Pan Xu, and Anqi Liu. Off-dynamics reinforcement learning via domain adaptation and reward augmented imitation. In Proceedings of the 38th International Conference on Neural Information Processing Systems (NeurIPS), pages 136326- 136360. Curran Associates, Inc., 2024.
[11] Mason Hargrave, Alex Spaeth, and Logan Grosenick. Epicare: a reinforcement learning bench- mark for dynamic treatment regimes. In Neural Information Processing Systems (NeurIPS), 2024.
[12] W. Hoeffding. Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association, 58:13-30, 1963.
[13] Linbin Huang, John Lygeros, and Florian Dorfler. Robust and kernelized data-enabled predictive control for nonlinear systems. IEEE Transactions on Control Systems Technology, 32(2):61 1- 624, 2024.
[14] Yong Huang, Rui Cao, and Amir Rahmani. Reinforcement learning for sepsis treatment: A continuous action space solution. In Proceedings of the 7th Machine Learning for Healthcare Conference, volume 182 of Proceedings of Machine Learning Research, pages 631-647. PMLR, 2022.
[15] Christina X Ji. Michael Oberst, Sanjat Kanjilal, and David Sontag, Trajectory inspection: A method for iterative clinician-driven design of reinforcement learning studies. AMIA Summits on Translational Science Proceedings, 0001: 2021.
[16] Heinrich Jiang. Uniform convergence rates for kernel density estimation. Proceedings of the 34th International Conference Machine Learning, PMLR, 70: 1694-1777,, 2017.
11

---

[17] Simi Job. Xiaohui Tao, Lin Li, Haoran Xie, Taotao Cai, Jianming Yong, and Qing Li. Optimal treatment strategies for critical patients with deep reinforcement learning. ACM Transactions on Intelligent Systems and Technology, 15(2):1-22, 2024.
[18] A. Johnson, T.J. Pollard. L. Shen, L.H. Hehman, M. Feng, G.Gasemmi, B. Moody, Szolovits, L.A. Celi, and R.G. Mark. Mimic-iii, a freely accessible critical care database. (may. 2016), 2016. URL https:/:://do orgoooo//// 10. (003//saata... 2016.35
[19] Alexandre Kalimouttou, Jason N Kennedy, Jean Feng, Harvineet Singh, Suchi Saria, Derek C Angus, Christopher W Seymour, and Romain Pirracchio. Optimal vasopressin initiation in septic shock: the oviss reinforcement learning study. JAMA, 2025.
[20] John A Kellum, Norbert Lameire, and KDIGO AKI Guideline Work Group. Diagnosis, evaluation, and management of acute kidney injury: kdigo summary (part 1). Critical care, 17:1-15, 2013.
[21] B.R. Kiran, I. Sobh, V. Talpaert, Mannion, A.A. Al Sallab, and Yogamani. Deep re- inforcement learning for autonomous driving: a survey. IEEE Transactions on Intelligent Trunsportation Systems, 23(6):4909 4926,, 2022.
[22] M. Komorowski, L.A. Celi, O. Badawi, A.C. Gordon, and A.A. Faisal. The artificial intelligence clinician learns optimal treatment strategies for sepsis in intensive care. Nature Medicine, 24: 1716-1720, 2018.
[23] Bettina Könighofer, Roderick Bloem, Sebastian Junges, Nils Jansen, and Alex Serban. Safe reinforcement learning using probabilistic shields. In International Conference on Concurrency T'heory: CONCUR, 2020.
[24] K. Kozhasov and J.B. Lasserre. Nonnegative forms with sublevel sets of minimal volume. Mathematical Programming, E93-D(3):583-594 2010.
[25] A. Kumar, Fu, G. Tucker, and S. Levine. Stabilizing off off-policy q-learning via bootstrapping error reduction. Proceedings of the 33th International Conference on Neural Information Processing Systems, 32:1 12:11784-11794, 2019.
[26] A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforcement learning. Proceedings of the 34th International Conference on Neural Information Processing Systems, 1911179-1991 2020.
[27] N Kuo, S Finfer, L Jorm, and S Barbieri. Synthetic acute hypotension and sepsis datasets based on mimic-iii and published as part of the health gym project (version 1.0. 0), 2022.
[28] S. Lange, T. Gabel, and M. Riedmiller. Batch reinforcement learning. Reinforcement learning: state-of-the-art, pages 45-73, 2012.
[29] Yuping Luo, Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, and Tengyu Ma. Algo- rithmic framework for model-based deep reinforcement learning with theoretical guarantees. In International Conference on Representation Learning (ICRL.), 2019.
[30] Y. Mao, H. Zhang, C. Chen, Y. Xu, and X. Ji. Supported trust region optimization for offline reinforcement learning. Proceedings of the 40th International Conference on Machine Learning. PMLR, 002:23829-23851 2023.
[31] Y. Mao, Q. Wang, C. Chen, Y. Qu, and X. Ji. Offline reinforcement learning with ood state correction and ood action suppression. Proceedings of the 37th Advances in Neural Information Processing Systems, 2024.
[32] Daniel Melcer, Christopher Amato, and Stavros Tripakis. Shield decentralization for safe multi-agent reinforcement learning. In Neural Information Processing Systems (NeurIPS), 2022.
[33] M. Nambiar, S. Ghosh, P. Ong, Y.E. Chan, Y.M. Bee, and Krishnaswamy. Deep offline reinforcement learning for real-world treatment optimization applications. KDD '23: Proceed- ings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 2023.
12

---

[34] Kenny Falkær Olsen, Rasmus Malik Høegh Lindrup, and Morten Mørup. Think global, adapt local: Learning locally adaptive k-nearest neighbor kernel density estimators. In Proceedings of the 27th International Conference on Artificial Intelligence and Statistics (AISTATS), volume
238 of Proceedings of Machine Learning Research, pages 41 4114-4122, 2024.
[35] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[36] Aniruddh Raghu, Matthieu Komorowski, Leo Anthony Celi, Peter Szolovits, and Marzyeh Ghassemi. Continuous state-space models for optimal sepsis treatment: a deep reinforcement learning approach. In Machine Learning for Healthcare Conference, pages 147-163. PMLR.
ISBN 2640-3498.
[37] Aniruddh Raghu, Matthieu Komorowski, Leo Anthony Celi, Peter Szolovits, and Marzyeh Ghassemi. Continuous state-space models for optimal sepsis treatment: deep reinforcement for learning approach. In Proceedings of the 2nd Machine Learning for Healthcare Conference, volume 68 of Proceedings of Machine Learning Research, pages 147-163, PMLR, 2017.
[38] Andrew Rhodes, Laura E Evans, Waleed Alhazzani, Mitchell M Levy, Massimo Antonelli, Ricard Ferrer, Anand Kumar, Jonathan E Sevransky, Charles L. Sprung, Mark E Nunnally, et al. Surviving sepsis campaign: international guidelines for management of sepsis and septic shock: Intensive care medicine, 43:304-377, 2017.
2016.
[39] Luca Roggeveen, Ali el Hassouni, Jonas Ahrendt, Tingjie Guo, Lucas Fleuren, Patrick Thoral, Armand RJ Girbes, Mark Hoogendoorn, and Paul WG Elbers. Transatlantic transferability of a new reinforcement learning model for optimizing haemodynamic treatment for critically ill patients with sepsis. Artificial Intelligence in Medicine, 112:102003, 2021.
[40] Xun Shen, Shuo Jiang, Akifumi Wachi, Kazumune Hashimoto, and Sebastien Gros. Flipping- based policy for chance-constrained markov decision processes. Proceedings of the 37th Advances in Neural Information Processing Systems, 2024.
[41] Xun Shen, Tinghui Ouyang, Kazumune Hashimoto, and Yuhu Wu. Sample-based continuous approximate method for constructing interval neural network. IEEE Trunsactions on Neural Networks and Learning Systems, 36(4):5974-5987, 2025.
[42] Xun Shen, Ye Wang, Kazumune Hashimoto, Yuhu Wu, and Sebastien Gros. Probabilistic reachable sets of stochastic nonlinear systems with contextual uncertainties. Automatica, 176: 1 112237, 2025.
[43] M. Sugiyama, I. Takeuchi, T. Suzuki, T. Kanamori, H. Hachiya, and D. Okanohara. Conditional density estimation via least-squares density ratio estimation. Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, PMLR, 9:781-7888 2010.
[44] X. Sun, Y.M. Bee, S.W. Lam, Z. Liu, W. Zhao S.Y. Chia, H.A. Kadir, J.T. Wu, B.Y Ang, N. Liu, Z. Lei, Z.Xu, T. Zhao, G. Hu, and G. Xie. Effective treatment recommendations for type 2 diabetes management using reinforcement learning: treatment recommendation model development and validation. Journal of Med. Internet Res., 23(7)x22758,, 2021.
[45] Shengpu Tang and Jenna Wiens. Model selection for offline reinforcement learning: Practical considerations for healthcare settings. In Machine Learning for Healthcare Conference, pages
2-35. PMLR. ISBN 2640-3498.
[46] Stephen Trzeciak, R Phillip Dellinger, Michael E Chansky, Ryan C Arnold, Christa Schorr, Barry Milcarck, Steven M Hollenberg, and Joseph E Parrillo. Serum lactate predictor of mortality in patients with infection. Intensive care medicine, 33(6))970-99) 2007.
[47] H.H. Tseng, Y. Luo, S. Cui, J.T. Chien, R.K.T. Haken, and I.E. Naqa. Deep reinforcement learning for automated radiation adaptation lung cancer, Medical Physics, 44:6690-6705, 2017.
13

---

[48] Akifumi Wachi and Yanan Sui. Safe reinforcement learning in constrained markov decision processes. Proceedings of the 37th International Conference on Machine Learning, PMLR, 119: 9797-9806, 2020.
[49] Akifumi Wachi, Wataru Hashimoto, Xun Shen, and Kazumune Hashimoto. Safe exploration in reinforcement learning: a generalized formulation and algorithms. Proceedings of the 37th International Conference Neural Information Processing Systems, 36:29252-29925 2023.
[50] Akifumi Wachi, Xun Shen, and Yanan Sui. A survey of constraint formulations in safe reinforcement learning. Proceedings of the Thirly-Third International Joint Conference on Artificial Intelligence (IJCAI24), 913:8262-8271, 2024.
[51] N. Wang, M. Wang, Y. Zhou, Liu, L. Wei, X. Fei, and H. Chen. Sequential data-based patient similarity framework for patient outcome prediction: Algorithm development. Journal of Medical Internet Research, 24(1)=e30720, 2022.
[52] Miriam Wolff, Hans Georg Schaathun, Sebastien Gros, Rune Volden, Martin Steinert, and Anders L. Fougner. Blood glucose prediction algorithms require clinically relevant performance criteria beyond accuracy. Diabetes Technology Therapeutics, 2025.
[53] XiaoDan Wu, RuiChang Li, Zhen Hc, TianZhi Yu, and ChangQing Cheng. value-based deep reinforcement learning model with human expertise in optimal treatment of sepsis. NPJ Digital Medicine, 611) 2023. ISSN 2398-6352.
[54] C. Yu, G. Ren, and J. Liu. Deep inverse reinforcement learning for sepsis treatment. Proceedings of 2019 IEEE International Conference on Healthcare Informatics, pages 1-3, 2019.
[55] Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Proceedings of the 34th International Conference on Neurul Information Processing Systems, 33:14129-14144 2020.
[56] H. Zheng, I.O. Rozhov, W. Xie, and J. Zhong. Personalized multimorbidity management for patients with type 2 diabetes using reinforcement learning of electronic health records. Drugs, 4821,448, 2021.
14

---

A Assumption on the Probability Density
Note that the data points {(st, s.a)} } in Db can be samples extracted from S Ui according
to joint probability density f(s, x) associated with the behavior policy, initial state distribution, and the transition dynamics. Here, with an abusement of notation, we replace st by s for simplicity. Let p(s]x) be the condition density from f(s, x). Let (x) be the marginal density. We have the following assumption regarding the underlying real density.
Assumption 1. Suppose that both joint density f(s, x) and the marginal density fx(x) are Holder continuous with the parameter E (0, 1]. Namely, there exists Cc such that [[x((x) - fx(x') S
Ccl[x- x'I and [fss x) f(s', x')l S Cxl (s',x')lf hold. Besides, the marginal density
fx(x) (x) satisfies
fx(x) 2 smin: Vx Via-
Both joint density and the murginal density satisfy exponential decays.
The lower bound the marginal density given by 22) is a strong assumption general sense. However, it is reasonable and practical in the problem setting of offline reinforcement learning with partial coverage. this setting, we should not consider the area with an extremely low probability of having state-action pairs.
B About the Choice of
Lemma 1. For any risk level 3 > there exist constants 7(3) (0, and (1) E N such that,
for all y 2 (3) and H S HB), the OOD cost constraint
20
9.7 (po) = EasorP
En"g(hh an) S
(3)
Dz=
serves conservative approximation of the joint chance constraint:
Pr (g an)) = 0, vh S H} > 11 3.
(4)
Proof. We begin by considering the joint chance constraint under the learned dynamics model T: Pr [(),, an) = 0, vh <
f 1-3.
(5)
Using the definition of ÿ. this is equivalent to:
2 1--B. B. Pr pr(ate (xn,, 0) 0, vh < H 7
(6)
Define the violation probability under '/ (po) = Pr
{ 3h & H : g( [1, (5,,0 0) > 0 17
3.7
Then c is equivalent to V. 5.7 (Po) B. Applying Boole's inequality gives:
H
y WHA (po) 6.
E
g(sh an)
7 IE
Σ g(S an 1
H. 5.
(po).
(7)
Thus, T.. (po) (PO) < 3 serves as a conservative approximation for the original joint chance constraint. 8
Next,
consider the infnite-horizon discounted surrogate:
Va (Po) (Po) 8.7
Exso~-pa Σn 2 an)) 7 Let 7 (po) = E o >' (St an)
7 The error
between the infinite-horizon horizon discounted cost and the cumulative cost can be expressed as:
H.H (po) = v (po) Vl (po) + TH a.7 ( T7. (po) 9.7
Esh (", == 6.7 (po)
H
E
7 g( an)
+ E
( ()) an))
Aum #+1
A=
tail error
discount bias
15

---

This error term (y, H) is strietly increasing in with:
E" (0, H) < 0, ETH (1, > 0.
Therefore, for any fixed H and policy n, there exists (im H) such that y > Mim H) implies:
- 8.7 (po) S 9.7 (pn)-
If is parameterized over a compact set, we can define ((,,3) := sup> Nim (T, H) such that this inequality holds for all feasible policies.
We now discuss approximation under the true model T. Let V H (Po) denote the cumulative violation 8.T
cost under true dynamics. The gap between the learned-model discounted cost and the true cumulative cost is:
E H) == 12,7((0) - V.H (po) 3.T
T. a. (po)
0.T (po) + V 6.7
H (po)
(po)-
5.7
The second term ?222 is already controlled as before. For the first term 5,1- which reflects model mismatch, we note:
In practice, the approximation error En (7, Hf) may be controlled by selecting a modest value of Hl (to limit propagation of model error) and choosing y sufficiently close to I (to amplify the tail contribution in E722) While this argument is heuristic, it aligns with common assumptions in model- based reinforcement learning, where shorter planning horizons and conservative discounting reduce the impact of model misspecification. Meanwhile, since the state-action pair is within the dataset support, the model misspecification can be small. Under these conditions, is reasonable to expect Eg (7, H) 2 0, ensuring:
7. V (po) S B.
Thus, the discounted OOD cost constraint conservatively approximates the joint chance constraint under the true model. If the policy class is compact, define constants ((3) and ( such that the approximation holds uniformly for all feasible policies.
C Proof of Theorem 1
A chance-constrained optimization problem can be formulated as min T.(0)= = log det p (0)
(Pa.))
s.t. Pr (g(g 0) S 1] 2 a-
(8)
The solution of Problem P a.d is defined by 0z. The corresponding polynomial sublevel set is tæ We have the following lemma regarding Problem Pa.d
Lemma 2. Assume that Uh is compact set. If a and the degree d - co, we have
Üor
Lig-
(9)
Proof. Define a distance function ya (x) associated with tha in the following way:
gs(x) - dist(x, Hldd + 1, if E Lid. dist(x, OWha) + 1, otherwise.
(10)
Here, dist(x, OUsa) is defined by
dist(x, 0144) == inf
||x y112.
(11)
yoöld
Note that tha can be specified by (x) < 1 and (x) is continuous. By the Stone-Weierstral theorem, for any s > 0, we can find a such that the following holds:
sup 1gs(x) o' (x)) xedl.
(12)
16

---

Here, Ue is compact set satisfying that Uia C Ue and (x)xx is d degree polynomial function. Define three sets by
Up = {x (x(x
<1=s}. ūa == fx (x)) S 11. Uj == fx (x(xx s 1+
For any e, let d be chosen as the value that makes 122 holds. Then, we have
Üg C Ui C Ut.
(13)
Note that, as E and d is corresponding chosen to satisty
12 we have
lim Up
Ut Ve 0 Uhs üa Uud
lim Uā = Ud Ae > 0, C Ua = U 2 Ud. c0
Thus, 02 = Ula
Let (ex)en be sequence converging zero and {d 2 be sequence chosen to satisfy lgx(x) - dh (x) < Ek, Ak E N+.
sup
xel4
Define problem for any given k by
min L(0) == log det P-)
(Pr,d.)
"
s.t. (x. 0) < 1, Vx E Uia
(14)
Note that Problem Prody equivalent to Problem Pard with the feasible set of Problem
and d da- For all k, let Of be Construct a set of polynomial sublevel sets by
r,dy
uf. = {uo... : @f)
we have Tht E Uf for every E N+ since every x E Ui also satisfies x E U As + oo, U converges U1 Ubs and thus Lid U
By the definition of 7 de
Let Od be one optimal solution of Problem Pr.de with k = cc. Then, we continue to prove Uha -Mgan dx d First, know that Lia U de due to the constraint g(x, Od..) < 1, Vx Mia- Note that we have already proved that Uha E Ug and let 0g be the parameter corresponding to the polynomial sublevel set that is identical with Lig. Since 24a S &U we have L(0,)= S [(f...) due to the the monotonicity of log-det inverse for positive semidefinite matrices [5]. Besides, Problem Prdde is convex optimization with strictly convex objective function and thus the problem attains an unique solution. Namely, 0g is identical with Od...,.'' which implies that Via = =U -ga do Since Ug do is identical with Uag
p holds as d + Do.
Theorem 4. The set Uha need not be compact. For any >> O, there exists degree d such that Üog C Via-
(15)
Proof. By Assumption 1 we know that the probability density defined on Uha is positive and continu- ous on Vig
Assume that 0* a.d is the solution of Problem aud with d = doc The corresponding polynomial sublevel set is Ug. do Note that Ulo No doc is compact. Define the following sets:
Ucom = Uto as don Aliis 2m : Lid Uo: dn doo-
Assume that the volume of Um is not zero. Note that Ucom is compact. Replacing Uia in Problem Pr.de by Ucom, we have Ucom Ugs by Lemma 2. Moreover, 8 02 is feasible solution to Problem Pride by Ucom and thus L(O* L(0*.am)) L(Ōa=) ) holds due to the uniqueness of the optimal solution to Problem Pr.de by Ucom Note that Oa is also a feasible solution of Problem Pa.d with d do due to Pr(x E Ucom) } = 1 Therefore, by [( L(Qu,e) .e it contradicts with that
8" a. is optimal solution. Thus, the volume of Um is zero and 155 holds.
17

---

Theorem only relies on positive value of a to ensure the subset relationship Uo a S Uis. Assumption 2. The parameters a and d are appropriately tuned such that Theorem A holdd. Theorem 5. Suppose that Assumption holds. Let Ote > a. Then, the probability that Ugg a subset of Uia can be bounded
is
Pr (uay a & Uia) eep exp (-22 -a))
(16)
As N + oo, the bound converges
Proof. It is reasonable to assume that we seek the optimal solution of Problem Pa.d within a compact set O, which includes a solution that satisfies (12) for some with correspondingly chosen d. Besides,
also assume that (xx()) () is well-justified ((x(+), ,0) (x((9) 8) E.
Let Y; I1 (9(x("),)) 0)) for i = I..... N. then Pr E [0, 1] 1 and B(Y, Pr (q f) 1}.
Given that 0 is a feasible solution for Problem P a.d and Pr (4(x, 0) > 1} S a. We consider the event not feasible for Problem GCL implying
that a feasible solution 9 (E [Y4] S a)) for Problem
1
11 (x()), .)) 2 ae
1
(Y, - E EY) de N [Yi
N
N
Σ -Y-BBly.) 2 N(ac-o) a). N (V E [i]) 2 ac N
Thus, we have
Pr M 11 (x())
2 ac Pr Σv -V--EMDD 2 N(ac - a) expl -2N(-2N(ao - =
where the last inequality holds due to Hoeffding's inequality [2]. Note that fi.d d is also feasible solution of Problem P a. which one realization of those mentioned above O. If 07,d is feasible
solution of Problem GCL we have that L( 2 L(ON). Then, Ugy S _ūo= By Theorem 4
have Ügx d Vid Thus, the probability violation is bounded by exp( (a - =a
D Proof of Corollary 1
From Theorem 4
know that the inclusion Ugy Uha holds with probability least 5 if the
log(1/8) 2 In addition, recall that the discount factor y and the OOD cost
sample size satisfies >
threshold ēg are chosen such that the following joint chance constraint holds (Lemma T
Pr (g(sh, S 0, vh < H) > 1-3. 3.
(17)
Note that the event U1g a C Und together with {Ō(Sk, a,) S 0, vh < H) forms sufficient con- dition to ensure that the agent remains within the support Via for all steps h < H. Therefore, applying Boole's inequality yields:
Pr {
[ & Una v [3h H, g(S,, an) > 0 < 8
(18)
Hence, the corollary is established: with probability at least 1 1- 8 B, all state-action pairs along the trajectory remain within the support Uis for the first H steps. This concludes the proof of Corollary
E Safety and Sub-Optimality with Finite Samples Considering o's Estimation Error
Value Function Error. We first analyze the error bound of the estimated value function associated with function (c.g., reward safety cost c;). Following Wachi et al. [4), we estimate ó using
18

---

GPR and estimate 7 using KDE [16]. While our theoretical analysis is grounded in these choices, our results also apply to other estimation methods, provided they ensure asymptotic consistency. Let be the bandwidth of the KDE, and assume that the joint density of (st ss a) and the marginal density of (s. a) are Hölder continuous with exponent E (0, 1]. Let a N (x) denote the posterior standard deviation of the GP estimate (()), and define 'n 1/2 == max + 4 UN + + log(1/6) where w is a kernel scaling constant and UN is the GP information capacity. Define the maximum standard
deviation at training points
o maXxelly xx(x). UN := {x()) Ji=1
Theorem 6. Let be any feasible solution of Problem GSRL Assume the standard KDE conditions
Nha+m + 0o and h + 0 as N + Jc. Then, with probability at least 1 - 23 - 46, the following
holds:
va v VorT(po) S EB Ek EH, where:
Eg = NN max
EH := 7 HH+1 - -7)0max (1 71-
mmax - 7
log(1/6)
Ek
(1-7)²
hS +
2h2n+m
Here, Cden is a positive constant depending on the smoothness of the densities, the choice of kernel, and the dimensionality 2n + m.
The proof of Theorem 6 is provided in Appendix F By selecting a sufficiently large dataset size N and conservative OOD threshold čg. we can ensure small 3 in the chance constraint (4. and thus make εµ negligible. Method of choosing čg for desired H follows [50].
Safety and Sub-optimality. We now define conditions under which the policy output by ConOpt is safe and near optrmmll with respect to the true model. We say a policy Tout is safe if:
max; Cj V (po) 2 Eg. Let 4" be the optimal solution to Problem GSRL with safety threshold
Cy. If Tout is computed using a tightened threshold éÿ E, and satisfies: V we obtain the following guarantee for the true system:
F F,j (po)
1,7 (po) I Er.
Theorem 7. E Ss &g + Ek +EH, and Tout is e, e--sub-pttmal for Problem GSRL then Tout safe and (s, + 2gg + 2sk + 22 --ub-opptimll for Problem ESRL with probability at least - 233 46.
F Proof of Theorem 6
The proof follows these main steps:
1. Bounding Errors for Supported Policies: We assume uniform upper bounds on the errors of conditional density estimation and reward or cost functions. Based on this, we establish the error bound for policy evaluation, limited to policies that not visit state-action pairs outside the support of the behavior policy.
2. Relating Sample Size and Estimation Errors: We analyze how the sample size influences the errors both conditional density estimation and function approximation, showing the dependency between the two.
3. Deriving the Probabilistic Bound: Combining the results from steps (1), (2) and Theorem [ we deduce a probabilistic error bound for policy evaluation, demonstrating how the evaluation accuracy improves with larger datasets.
First, we give the revised telescoping Lemma by introducing the estimation error of 6.
Lemma 3. Define a function G (s, a) by
GT a) = E (t.a)
V [oT(s*)]-
E
[=*
(19)
x T(,a)))))
Then, we have
v (Po) -Va (po)
Σ
E
(6(s) - o(sj, a;)]+
744 E
G3
G7 aj)
j=0
8 NT,
j=0
(20)
19

---

Proof. Following the pattern of the proof of Lemma 4.1 in [55], define W as the expected return when executing on Ma for the j steps, then switching to M for the remainder, written by
W
E as (sA)) Bompp h (sn, 8=8 82 Bhh 6=0
Σ 1A=0
7 6s an)
Write
W, Dj-1
E
7³ aj):
E a)H,n)()),,s))
Vo,T (8)+1))
8o By mT
j1 D
E
E
(
$3 my
T (sy,n,)
Here, DT- the expected return the first j time steps, which are taken with respect to 7 and 6. Then, we have
W W = 7d
E [6s a) o(s, + 7
E
G C7 (5)
8,,a, N
8,
Note that Wo Vo.T and Wo
V2,7 (po), and have
x (W - -W))
1/a (2,((0)) 1/- (Po)
j=0
20
E Sy,
(6(s,a) oss a)] + Σ j=0
i+1
E sy
G (S))
j=0
which completes the proof.
One practical strategy is to use the kernel density estimation to give the estimations f(s, x) and fx (x) and then obtain the estimation of p(s x). The estimation p(s x) is defined by
f(s
p(s[x)
(21)
fx(x)
where f(s, x) and fx( denote the estimated joint density and marginal density, respectively. Kernel density estimation can be used for f(s,, x) and fx(x), denoting by
f(s, N. -m2+m
1
Σ K
(())
K
x
x(()
(22)
fx(x) N. hn+m
N
K
x
x
(23)
We have the following lemma for the estimation error the conditional density estimation. Lemma 4. Suppose that Assumption hoold.. The kernel function K(u) satisfies:
I K(u)du 1, sup sup]K(u)] oo, / [2² Ku)ddu co. "
(24)
The bandwidth satisfies the standard kernel density estimation condition such that Ni - 00 and h 0 hold as N - co. Then, let Ep(s, x) = l0(s)x) p(s|x)) with probability at least 1 8, we have
(s) < (G++
((s)x) Cm)) -
log 1/5 N1,2+mm
(25)
S
Here, Oj is positive constant which depends on the kernel, joint density smoothness, and dimen- sionality 2n, + m. Besides, Cm is positive constant which depends on the kernel, marginal density smoothness, and dimensionality m.
20

---

Proof. Compute the absolute error by
p(s|x) p((s)) = f(s,x) [(s, x)
[(s,x)
fx(x) - fx(x)
[(s,x) - f(s. x)
fxx (x)
fx(xx
fx(x)
fx(x)
f(s,x) f(s, x)
- -x(x))fx(x))
-((sx))
[x(x)
[x(x)
Then, we have
f(s.x) - f(,
fx(x) (xx)
(8, x)
<
f(ssx)
fx(x)
(x(
(p.1
(8.2(8.))
Then, According to the sup-norm bound for kernel density estimation given by Theorem 2 in [66]
with probability at least
6, we have
Ey (s, x) < 1 Lg.
log 1/8 Nh2n+m
Cj.
h V
(26)
Cm fmin
(15
log 1/6 2/2n+m
Ep.2(s,)) S ((s)x)
(27)
By 26 and 27
obtain
25
L
Based on the above discussions, we give the proof of Theorem a follows.
Proof. (Theorem 6 We first discuss n, which ensures that (s,, a) pij stays in Lia with probability 1. Using x for (s, a), rewrite (20) into the following case:
(P)) - (PD) E
[x(x)) - o(x)))
+
J+1
Gx
G
(28)
f
i=0
i=0
We first discuss E& 's bound based on the GPR-based estimation 6, We have
Eg
[0(x,) - o(x))
j=0
X
[6(x;) - o(x;)]
(6(x)) o(x,))
X
H++
Xy
In distribution w.p.1-0-6
Out of distribation
H
(ō(x,) - o(x))] +
J Omax
1=0
Xymun?
j 1+1
H
Omax 7 HH++1
E X
[1((x,) - -(x;)]
T
1-7
1=0
We use the Gaussian process regression to approximate the scalar function o(x). By Theorem 6.1 we further have Ea < " d
-x,)) + Sma H-1
of 49
E
w.p.1 3 26. Here,
1/2 = Omax u 1 + 1og(1/8) (x1x) the posterior standard deviation at point x, and 7N
with as scaling factor accounting for kernel parameters, vN is the information capacity associated
21

---

with kernel. Since we consider the in-distribution posterior standard deviation xx is reasonable assume that xx) is bounded in Un xx S max Note that oN can be approximately chosen as o Nax se maxxEMy ON (x). dx(x). Thus, we have
max VN ON
max H+1
86 S
w.p.1 - B - -22.
(29)
1
We then discuss the bound of eq. We have
D0
0c
Eq
70
(x()
S
7
(x))
G
+
Σ
7
(x(xx)
1
Xy
Xynua,
j
X
H+1
X
In distribution (1p(1---8)
Out of distribution
11
++
E
G (x,)
+
++1
max
1 1=0
XyNoT,
i= +1
H
+ 7 #+1 +1 Oaxx
7
G (x,)
(1 7)-
==
X
7 1- maax
Σ ==
7
Vo7 [o)] snT(X,)
[VTT(s)]
(1 7)2
Xynu,
(~x((x))
H
Vo (s)((s) / (S)((( ds
+ H+ Omax
E
7)
X
/
+ 7 11+1 Omax
Σ
Vo, (o)(()) (f(s)x,) p(s/x;)) ds
E
(1 7)
1=0
X
1/
+ 7 H++ maxx
E
Vo- [o)(
(p(slx)) p(ssx,))) ds
(1 7)2
x
H
Ug Ep(s, ds
+ 7 H+ Omax
E x
(1 (1-7)
jm0
(Use Lemma
4 to proceed to the next)
mmax
j+1
Gj p(s)x,) Cm
( ) (ns +
log 1/6
ds +
fmin
Nh2ntm
j=0
X x
7
H Omax
w.p. 1 - 3 28
71-
max
7(1 (1111 (9 (G +9
hc
log 1/5 2h2
+
7
1-7
smin
7 1++1 +1 Omax
w.p. 1 B 28
(1 - 7)-
Omax (7 (++2) ( (C Om
H+1 Omax w.p. 3 3-2 26. log 1/8 2h2n+m
18
- 7)-
min
7)
(30)
22

---

Combine 29 and 30 we have that, with probability 1 23 - -6, the following holds
1 (,7(p0) - (Po) -
NN
max ON
max
(7 - 7 71+2
Cden
(15 +
log 1/6
+
1
(1 - 7)2
2h2
nh+ 12 7) -max
(31)
(1 7)
where Cden (Cj + Cm)/Smin
G Sepsis Treatment Experimental Details
G.1 Dataset Description
The MIMIC-III (Medical Information Mart for Intensive Care) database serves as comprehensive repository containing detailed clinical records from over 40,000 intensive care admissions (18). This extensive dataset our methods maintain clinical relevance and algorithmic robustness across diverse patient populations and treatment scenarios. Its widespread adoption in healthcare machine learning research, combined with its real-world clinical variability and detailed documentation, establishes MIMIC-III an appropriate benchmark for treatment policy evaluation. We implemented a five-fold cross-validation approach, randomly dividing the data into training (60%), validation (20%), and test (20%) partitions for each seed.
G.2 Sepsis Treatment Formulation for RL
The MIMIC-III Sepsis dataset provides 44 variables, comprising both dynamic physiological mea- surements and static patient attributes, which form the foundation for our state space construction. Following the protocols as mentioned Section 4.1 we obtained cohort of septic patients by iden- tifying those who developed sepsis at some point during their ICU stay and including all observations from 24 hours before until 48 hours after the presumed onset of sepsis. The protocols organized data into 4-hour windows, creating sequence of 20 time windows in total. Table [ below summarizes the selected features, treatment actions, and reward signal used our study.
Table 3: Summary of Feature Space, Actions, and Reward
Category
Variables
Description
Dynamic Features
Mechanical ventilation, GCS, FiOz, PaO2, PaO,/FiOz. Total bilirubin, Urine output (4h), Cu- mulative urine output, Cumulative fluid input, SpO₂
Time-varying physiological indica- tors with moderate or strong corre- lation to SOFA, capturing real-time sepsis severity.
Static Features
Age, Gender, Readmission status
Fixed patient characteristies offering essential context for modeling het- erogeneity and enabling personalized treatment.
Actions
Intravenous fluids (per 4h), Max vasopressor dose (per 4h)
Core interventions for sepsis manage- ment aimed at stabilizing blood pres- and perfusion.
Reward
SOFA score
Quantifies the extent of organ dys- function and guides the policy toward clinically meaningful improvement.
Reward Function. The Sequential Organ Failure Assessment (SOFA) score was selected as reward function for its clinical advantages over mortality as a terminal reward. SOFA provides instan- taneous assessment of organ dysfunction across six physiological systems: respiratory, coagulation, hepatic, cardiovascular, central nervous system, and renal. SOFA scores range from 0-24, with each
23

---

subsystem contributing 0-4 points. This granularity supports more nuanced policy optimization compared to binary mortality outcomes.
State Space. For treatment optimization, we selected state variables through a clinically grounded and data-driven approach. Our model incorporates two treatment actions: intravenous fluid volume administered every 4 hours and maximum vasopressor dosage within the same interval. These interventions were chosen for their prevalence in early sepsis management protocols, where they restore blood pressure and tissue perfusion [33] The state representation was designed to include features highly correlated with SOFA, capturing critical aspects of patient health relevant to clinical outcomes.
Dynamic Feature Selection. We prioritized dynamic variables that change during treatment by calcu- lating Pearson correlation coefficients between each variable and the SOFA score. Both synchronous (lag = 0) and asynchronous correlations (lags of 1, 2, and 3 time steps) were examined to account for delayed physiological responses. Features with absolute correlation exceeding 0.2 in at least one lag setting were retained. This threshold balances inclusivity and relevance-sstingeet stringent enough exclude weak associations while preserving moderately meaningful relationships to patient severity.
Inclusion of Static Features. To model patient heterogeneity and personalize treatment decisions, we incorporated static patient attributes that remain constant during ICU stays. Although limited to age, gender, and readmission status in this dataset, these features provide essential clinical context: age represents a known risk factor for sepsis severity, gender may influence physiological responses, and readmission could indicate chronic conditions recent complications. Their inclusion enables policy generalization across diverse patient populations.
Final State Space. The final state representation comprises 13 features (10 dynamic and 3 static), creating a compact yet expressive state space that captures key clinical indicators while maintaining computational efficiency. Both state and action spaces are continuous, supporting fine-grained policy learning and clinical interpretability while remaining feasible for real-world implementation.
Explicit Clinical Safety Constraints. We justify that our safety oxygen saturation (SpO2) maintained at or above 92% prevents hypoxemia and ensures adequate tissue oxy geyaetttton. Urine output of at least 0.5 mL/kg/hour preserves sufficient renal perfusion and detects early signs of kidney injury. We selected these constraints to monitor distinct yet vulnerable organ systems sepsis while providing continuous measurement capability in ICU settings, ensuring both clinical interpretability and practical implementation.
G.3 Guardian Construction
Solving Problem GCL becomes computationally complex when the dataset is large (c.g.. exceeding fifty thousand state-action pairs). To enable scalable implementation, we propose a kernel density- based approximation for the guardian set. Let fpa Xy) be a kernel density estimate built from the dataset Xy of state-action pairs. For any given density threshold fehs. define the corresponding empirical outlier probability by (fts)) := ffa //N, where Nour(ftte) is the number of samples in XN whose estimated density is below fths- To find a threshold corresponding given confidence level a, we perform binary search over sehs: Initialize smin and fmaa such that Pout (fui))) < a < (ffaa))))) - Iteratively update the midpoint fmii = (femin + fmmax )/2 and evaluate ( - (f > a, update fins == fmid, otherwise, set fms == ==fsss
smid After a
fixed number of iterations, the binary search converges to a threshold f such that Pour (ft%s) ) Rs a.
We then define the approximate guardian: a state-action pair x is considered inside the guardian fpa (x, Xx) > fihs and outside otherwise.
G.4 Policy Learning Algorithms
We explain how the guardian is applied in GCQL. CQL trains the Q-function using an offline dataset ((s, a, ).). During the Bellman backup step in CQL, the target is computed as = wa((ss)) | )
r ++7
(Staggets* a+)], where the expectation approximated by sampling. Note that although s+ lies within the dataset, the sampled action a may result a state-action pair (st, ) that falls outside the guardian set. In GCQL, if (s+,a+) a+) is not within the guardian, we replace Qtarget (s+ at) with a large negative penalty value. For MB-TRPO, GMP-TRPO, MB-CPO, and GMB-CPO, we use the training data to fit a k-ncarest neighbor (k-NN) model of the transition
24

---

dynamics. These online algorithms are then trained by interacting with the environment simulated using the k-NN based transition model. The reward (i.e.. the SOFA score) and the cost (i.e., SpO,) are computed directly from the estimated state using predefined rules, without requiring additional model estimation. GMB-TRPO, the guardian incorporated by modifying the reward function: state-action pair falls outside the guardian set, the reward is penalized by assigning large negative value.
G.5 Transition Dynamics Model
To comprchensively evaluate learned policies while maintaining safety, employ k-NN model to estimate transition dynamics. The k-NN model learns transition dynamics from historical patient trajectories, capturing the complex relationships between medical states, treatment actions, and subsequent patient outcomes. In our experiment design, we maintain two separate k--N models:
k-NN-train. Trained the 60% training partition, used during policy learning in model-based algorithms (GMB- TRPO and GMB-CPO). As the agent learns, takes actions this simulated environment, with the k-NN model providing plausible next states based on historical patterns from the training data. This creates a realistic training environment that remains anchored to observed clinical behavior, preventing the policy from exploring dangerously unfamiliar territories.
k-NN-eval. Trained on the full dataset (training validation test), used exclusively for policy evaluation. This approach plays crucial role in enabling safe off-policy evaluation by simulating patient trajectories without actual patient interaction. When evaluating a learned policy, we start with real patient states from our test set and let the policy choose treatments. The k--N model then predicts the most likely next state by finding similar historical cases in the dataset. This process continues, creating synthetic patient trajectories that mirror realistic clinical progressions while keeping actual patients safe from experimental policies. The k-NN-eval model, having access to broader range of state transitions, provides more demanding test of the guardian's 's ability to prevent OOD exploration.
This dual-model approach ensures methodological rigor: policy learning uses only training data (k-NN-train), while evaluation leverages the full dataset (k-NN-eval) to comprehensively assess generalization. This configuration is consistent across both sepsis and hypotension experiments.
G.6 Evaluation Metrics
Model Concordance Rate (MCR). The Model Concordance Rate measures the proportion of instances where the model's recommended action matches the clinician's 's action in the offline dataset. Formally, the MCR is defined as:
Ex, (TTS) TRI. (S..:))
MCR
E.T.
where (s,,()),)) and TRL (s),.) denote the actions taken by the standard-of-cat (SoC) and the learned policy at state Si,, respectively, and T; is the number of timesteps for patient i. For continuous action spaces, a match is determined if the Euclidean distance between the two actions is less than a pre-specified threshold that is,
(ST,))) - (s..)) <
Appropriate Intensification Rate (AIR). The Appropriate Intensification Rate evaluates whether the model appropriately escalates treatment in response physiological deterioration. We define the Urine Output Rate (UOR) as the volume of urine output normalized by patient weight per hour (mL//g/hrr). A need for intensification arises when either the oxygen saturation (SpO.)) the UOR falls below a clinically significant threshold. Formally, AIR is defined as:
Ex, T (SpO2( < Tspo V UOR(i, f) < TUOR) A Intensified (i,1) AIR
E., TSSO (1, < Tspo, v UOR(i, t) TUOR
where TSpO, is set 92%, TUOR set to 0.5 ml /kgg/r and Intensified(i, (i, 1) is an indicator function equal to if the model recommends an increased treatment intensity at time t.
25

---

Mortality Estimate (ME). The Mortality Estimate measures the likelihood of patient death under the model's policy by simulating patient trajectories using learned transition model. Starting from an initial state, actions are selected according to the policy, and next states are predicted by the transition model. The simulation continues until either the maximum trajectory length reached terminal "dead" state is encountered. The ME is defined as:
N N Σ (Diet))))
ME
where N is the number of simulated trajectories. and Diet(i) equals if patient i equals 1 if patient enters a dead state before reaching the end of the simulation horizon.
Action Change Penalty (ACP). The Action Change Penalty quantifies the abruptness of the model's recommended actions across consecutive time steps within a trajectory. Formally, ACP is defined as:
ACP Σ, E |0,,411 a1,1122
Σ.T.-1)
where dg denotes the action taken at time for patient i, and || 1/22 denotes the Euclidean norm. Lower ACP values indicate smoother and more consistent treatment recommendations over time.
G.7 Detailed Validation Results and Discussions
G.7.1 Physiological State Distribution Analysis
This comprehensive analysis examines the temporal evolution of physiological states across 20-step treatment trajectories, extending beyond the safety constraints demonstrated in (Spo,, and urine output) to encompass the broader spectrum of clinical variables. Through systematic evalua- tion, we reveal how different reinforcement learning implementations influence patient physiology throughout the treatment continuum.
G.7.2 Temporal Evolution of Clinical Variables
MB-TRPO demonstrates progressive divergence from standard care protocols, manifesting physio- logical deterioration across multiple organ systems illustrated in Figure 4 Mechanical ventilation patterns exhibit marked volatility, while Glasgow Coma Scale trajectories deviate substantially from clinical norms. The PaOz/FiO, ratio reveals concerning instability that intensifies temporally, sug- gesting compromised respiratory efficiency. This systemic divergence indicates that unrestricted exploration permits physiologically implausible treatment strategies, compounding adverse effects across interconnected organ systems.
MB-CPO, despite explicit constraints limited to SpO2 and urine output, achieves remarkable sta- bilization of unconstrained variables. This effeet extends across multiple physiological domains: hepatic function markers demonstrate reduced variability, respiratory parameters beyond SpO2 ex- hibit enhanced stability, and cumulative fluid balance follows more physiological trajectories. The mechanism reflects the interconnected nature organ systems in sepsis pathophysiology. where preserved oxygenation prevents cascading organ dysfunction.
GMB-TRPO undergoes fundamental transformation, producing state distributions closely approx- imating standard clinical practice. This metamorphosis manifests across all monitored variables, with pronounced stabilization evident respiratory mechanics, neurological status indicators, and fluid homeostasis parameters. The synergistic combination of explicit constraints and guardian mechanisms yields the optimal configuration, demonstrating unprecedented aligmment with standard care patterns while maintaining physiological parameters within clinically appropriate ranges.
As visualized in Figure MB-CQL demonstrates inherent safety properties, maintaining closer alignment with standard care practices. This alignment extends beyond action
CQL achieves a MCR of 0.789 Table encompass state-space dynamics shown in Figure The algorithm's 's conservative value function regularization naturally constrains exploration to clinically validated regions. producing physiological trajectories significantly more stable than those generated by baseline model-based approaches, such as MB-TRPO and MB-CPO. Guardian augmentation builds upon this foundation, achieving enhanced stability particularly cumulative urine output patterns, where the combined approach maintains tighter alignment with clinical practice.
26

---

MB-TRPO
GMB-TRPO
MB-CPO
GMB-CPO 50C
Machamtcol
Glangove Scale
Fracctin
Inspited
Partal Mrzm
PLOZIFIO satlo
T0 ailruban
Urion PoR 6
WiltE of
Figure Physiological state progression under moodel-based reinforcement learning methods (RP-TRPO, GMB-TRPO, MB-CPO, and GMB-CPO). Each step is represented by a box plot, where each box shows the interquartile range (25th-75th percentiles) with the horizontal line in- dicating the median. Whiskers extend to 1.5 x.RR and black dots represent outliers individual measurements falling outside this range.
G.7.3 Clinical Significance Discussion
Physiological System Interconnectivity. The analysis substantiates established clinical principles regarding organ system interdependence. Maintaining critical parameters within therapeutic ranges generates beneficial cascade effects throughout multiple organ systems, corroborating the therapeutic strategy of prioritizing hemodynamic stability and respiratory function fundamental interventions sepsis management.
Complementary Safety Architectures. Comparative evaluation reveals synergistic benefits be- tween constraint-based and guardian-based protective strategies. Explicit constraints provide robust safeguards for designated variables, while guardian mechanisms furnish comprehensive trajectory
27

---

COL
GCQL
CCOL
GCCQL
S0C
Machamtcol
Giangowe Coma
Scale
Fraccton Inspited Diggen
Port Prozo
PLOXIFIO natio
Todaa aitrrta
Tal of
Voluma of Urion
Figure 5: Physiological state progression under model-free reinforcement learning methods (CQL, GCQL, CCQL, and GCCQL). Each step is represented by box plot, where each box shows the interquartile range (25th- 75th percentiles) with the horizontal line indicating the median. Whiskers extend to 1.5 1.5xIQR, and black dots represent outliers individual measurements falling outside this range.
protection, mitigating exploration of unsafe state combinations beyond the scope of limited constraint sets. The superior performance of GMB CPO demonstrates that clinical implementation should incorporate both protective modalities.
Temporal Stability Considerations. Given sepsis pathophysiology's progressive nature, treatment protocols must maintain stability across extended periods. Guardian-enhanced methodologies exhibit superior temporal resilience, particularly crucial during the initial 20-hour therapeutic window. This consistency translates to reduced risk of abrupt physiological deterioration cardinal concern in intensive settings.
28

---

Clinical Integration Perspectives. Aligmment between
enhanced policies and established care patterns facilitates integration within existing clinical workflows. Reduced variability in physio- logical trajectories enhances predictability, essential for clinical acceptance and real-time decision support deployment. 'The demonstrated capacity to identify beneficial deviations from standard protocols while maintaining safety parameters suggests potential for discovering innovative treatment approaches within established safety boundaries.
G.7.4 Consistency Validation
Comparison of OOD state avoidance with different seeds.
NE--TRPO
BB-CPO
-15
-30
-50
-350
SOC
COL
-350
-32
-105
SDc
CCOL
-130
-12
100
-100
-40
-190
-90
131
-100
-93
1000
(a) CQL
(b) MB- TRPO
(c) CCQL
(d) MB-CPP CPO
100
OCOL
S0c
-122
SOC
GMB-TRPO
-322
S0c
GMB-CPO
(e) GCQL
(I) GMB-GMMB TRPO
(g) GCCQL
(h) GMB-CPO
Figure 6: Results on state distributions generated using the second seed. The patterns observed here consistent with those shown in Figure]
100
SOC
ME-TRPO
ICC
S0C
CCOL
IEC
SOC
ME-CIPO
30
-50
-100
-200
20
50
300
-100
-50
30
-100
-30
30
10c
-200
-52
30
100
(a) CQL
(b) MB- TRPO
(c) CCQL.
(d) MB-CPO
100
-2
SDC GCOL
SOC
GMB-TRPO
-222
SDC
OCCOL
S0C GMB-CPO
(e) GCQL
(I) GMB-GMBBR TRPO
(g) GCCQL
(h) GMB-CPO
Figure Results on state distributions generated using the third seed. The patterns observed here consistent with those shown in Figure
Comparison of Cumulative Reward Distributions with Different Seeds.

---

10C
SOC
MB-TRPO
MB-CPO
-50
50c
-35
50c
-35
COL
-152
CCOL
1100
100
-19
-10
100
-100
116
-100
-52
50
593
(a) CQL
(b) MB- TRPO
(c) CCQL
(d) MB- IPO-
122
50c
SBc GMB-CPP
OCCL
-32
OCCOL
(e) GCQL
(I) EMBB-TRPO
(g)
GCCQL
(h) GMB-CPO
Figure 8: Results on state distributions generated using the fourth seed. The patterns observed here consistent with those shown in Figure
IEC
IEC
ICC
IEG
SOC
MB-CPO
30-
-50
-50
5OC
S0C
-302
-165
-100
-91
-190
(a) CQL
(b) MB-TRPO
(c) CCQL
(d) MB ABOCPO
SDC
SOC
CCOL
-322
GCCOL
110
159
-100
+10
755
-000
15
500
-193
-10
(e) GCQL
(I) GMB-TRPP
g
GCCQL
(h) GMB-GMPBPP
Figure 9: Results on state distributions generated using the fifth seed. The patterns observed here
consistent with those shown in Figure
Meare TELS
4CM
Book
piaicee
604
6000
201
200
(a)
(b)
(c)
(d)
Figure 10: Comparison of cumulative reward distributions between the SOC (green) and various RL policies with guard mechanisms (blue) across different algorithms using the second seed. The patterns observed here are consistent with those shown in Figure 2 (a) CQL vs. GCQL:(b) CCQL vs.
GCCQL: (c)
GMB-TRPO; (d) MB-CPO GMB-CPO.
30

---

541
CILAY
201
(a)
(b)
(c)
(d)
Figure 11: Comparison of cumulative reward distributions between the SOC (green) and various RL policies with guard mechanisms (blue) across different algorithms using the third seed. The patterns observed here are consistent with those shown in Figure 2 (a) CQL vs. GCQL:(b) CCQL GCCQL; (c) MB- TRPO vs. GMB- TRB-; (d) MB-CPO vs. GMB-CPO.
OUT
PAEK OOLLE
215.1
SFBICI CONAS
241
(a)
(b)
(c)
(d)
Figure 12: Comparison of cumulative reward distributions between the SOC (green) and various RL policies with guard mechanisms (blue) across different algorithms using the fourth seed. The patterns observed here consistent with those shown in Figure 2 (a) CQL vs. GCQL:(b) CCQL vs. GCCQL; (c) MB- TR-O vs. GMB-TRPO; (d) MB-CPO vs. GMB-CPO.
SIV
SVC
2.22
No Sot Maat
208.30
PRID $15.32
Bass COAP
Newerd Interbutton
Hemard
(a)
(b)
(c)
(d)
Figure 13: Comparison of cumulative reward distributions between the SOC (green) and various RL policies with guard mechanisms (blue) across different algorithms using the fifth seed. The patterns observed here consistent with those shown in Figure 2
(a) CQL GCQL:(b) CCQL GCCQL;
(c) MB- TRPO vs. GMB- TRB-RRPO; (d) MB-CPO vs. GMB-CPO.
G.7.5 Comprehensive Clinical Efficacy Analysis
Clinician Policy Alignment. Table I
presents quantitative comparison of clinical alignment across methods using four metrics: MCR, AIR, ME, and ACP.
MCR. The alignment between policy recommendations and clinician decisions is reflected by MCR. Higher MCR values indicate stronger behavioral mimicry, which enhances clinical interpretability and acceptance. Model-free approaches such as CQL and CCQL demonstrated moderate concordance (0.789 and 0.827, respectively), while model-based methods including MB-TRPO and MB-CPO showed near-zero concordance, highlighting instability in unconstrained model-based training. The integration of OOD guardians significantly improved all model variants. GCQL achieved the highest concordance (0.909), while GMB- TRPO and GMB-CPO substantially outperformed their baseline counterparts, demonstrating the guardian's 's efficacy in promoting clinically familiar behavior.
AIR. AIR measures learned policy's ability to intensify treatment when physiological deterioration occurs (e.g., decreased SpO2 or urine output). Policies with high AIR values respond appropriately to emerging risks. Without guardian augmentation, AIR remained low across all base methods, CQL (0. 130), CCQL (0.039), and MB methods (< 0.05) i.ddininn redeosieeponsive policies. Guardian- augmented approaches substantially improved AIR, with GMB-CPO (0.448) and MB-CPO (0.496)
31

---

demonstrating the greatest responsiveness, suggesting more adaptive and clinically aligned escalation behavior.
ME. ME predicts expected mortality outcomes, with lower values indicating improved survival rates. Compared to standard of care (SOC: 0.0632), most baseline methods showed slight improvements (CQL: 0.0486, CCQL: 0.0481). Guardian cenrdian-enhnneed policies further reduced mortality estimates, with GMB-CPO achieving the lowest value (0.0138), followed by GMB-TRPO (0.0232). This suggests that guardian integration not only enhances safety but also improves health outcomes. Model-based methods without guardians produced missing or undefined mortality estimates due trajectory instability.
ACP. ACP quantifies the magnitude of change in policy recommendations between consecutive time points. Lower values indicate smoother, more stable treatment suggestions- critical property for clinical implementation,
abrupt changes in medication dosage or fluid administration can cause physiological disruption or compromise patient safety. For MVD and IFA, CQL and CCQL produced relatively stable policies (ACP of 4.18 and 3.74; ACP of 543 and 460, respectively). In contrast, MB-TRPO and MB-CPO exhibited substantially higher ACP values-48. to 50.5 for MVD and up 4.92e for indiinain erratic treatment recommendations unsuitable for clinical application.
Guardian integration dramatically improved action smoothness. GMB-CPO (ACP.MVD 4.34, ACP:1AA: 647) closely matched standard-of-cant values (4.34 and 648), while GCQL and GCCQL maintained consistently low ACP values. These findings suggest that OOD guardians effectively regularize learned policies by discouraging unstable transitions in treatment trajectories, resulting in smoother, safer interventions that better align with clinical expectations and practices.
In summary, muardian-enhancee methodologies outperform their baseline counterparts across all metrics. Model-based approaches without guardian constraints prove unreliable due to excessive generalization, while guardian integration restores alignment with clinical norms. Among all eval- uated methods, GMB-CPO delivers the most balanced performance, demonstrating strong MCR, AIR, ME, and treatment prescriptions closely resembling real- wollw clinical practice (ACP). These findings validate the proposed OOD guardian as an effective mechanism for ensuring safe, effective, and trustworthy policy learning in offline medical reinforcement learning.
Moreover, implicit policy (sequence of actions) adopted by clinicians might not be optimal, but individual decisions are the least safe. This is because even human experts have limited ability to integrate a patient's full historical state into consideration. Individual action is safe, but it is derived from a rigid rule or greedy fashion to achieve short-term goal. What capable and safe offline RL learns from observational data a "dynamic" policy that considers the full state history of a patient with safe individual actions.
Treatment Effectiveness and Reward Distribution.
Figure 2 compares the cumulative reward distributions of policies learned by different RL algorithms with guardian (blue) against algorithms without guardian (red) and the standard clinical policy (green). Across all algorithms, policies trained with the guardian achieve substantially higher mean cumulative rewards than the policies trained without the guardian, highlighting the potential of the guardian to improve RL-based treatment outcomes. Notably, model-based methods such as GMB-TRPO and GMB-CPO not only yield higher reward means but also exhibit more concentrated reward distributions compared to model-free approaches (GCQL and GCCQL), demonstrating improved robustness. Furthermore, while GMB-TRPO and GMB-CPO exhibit similar performance in reward, the latter achieves lower safety costs (see Figures Ba and 3bj, confirming its superior balance between reward optimization and safety compliance. Besides, for the safety cost, compared to the model-fre free methods GCQL and GCCQL, GM - CPO shows a better similarity with the standard of care (see Figure 3 Combined with the MCR and AIR results (Tables TT. which show stronger alignment between guardian-enhanced policies and clinician decisions, these findings suggest that the OOD guardian improves not only consistency with expert behavior but also outcome quality. Among all methods, GMB-CPO achieves the best trade-off between safety and reward, addresses both OOD action and state issues, and produces robust, high-quality policies aligned with clinical practices.
Physiological Safety.
As shown in Figure 3 we evaluate the physiological safety of our learned treatment policies by analyzing SpO2 and unine output-specfficall chosen because they serve as our explicitly constrained
32

---

physiological safety states in the OGSRL framework. The results clearly demonstrate that our proposed guardian consistently reduced unsafe states compared to their non-guardian counterparts. For SpO2, GCQL achieved the most substantial improvement (59.4% reduction in unsafe states), while GCCQL and GMB CPQ demonstrated strong performance with 28.88 35.8% and 49.8% reductions, respectively. Only MB TRPO significantly worsened respiratory safety with 90.7% increase in unsafe states, highlighting the danger of unconstrained exploration in high-stakes domains. For urine output, aradd methods again outperformed their counterparts, with GMB-TRPO and GMB-CPO achieving 19.0% and 19.6% reductions in unsafe states. The dual-safety mechanism in GMB-CPO, combining explicit physiological constraints with the OOD guardian, demonstrated balanced performance across both measures. Notably, even GMB-TRPO, which lacks explicit safety constraints, significantly improved safety through guardian- based restriction of OOD regions. This highlights how the guardian mechanism indirectly preserves physiological safery by constraining policies to clinically validated regions. Interestingly, observe that model-free methods (CQL, GCQL, CCQL, GGCQL) improve SpO2 safety but increase unsafe states for urine output. This pattern likely originates from SpO, responding quickly interventions, while urine output depends on complex, delayed effects of fluid management and hemodynamic stability. Without explicit modeling of physiological dynamics, model-free methods struggle capture these delayed treatment effects, despite successfully constraining actions to clinically observed patterns through the guardian mechanism.
H
Acute Hypotension Experimental Details
H.1 Dataset Description
The Synthetic Acute Hypotension Dataset 27
contains 3,910 ICU stays with 187,680 hourly state- action pairs over 48 hours. Acute hypotension (mean arterial pressure below 65 mmHg) represents critical hemodynamic emergency distinct from sepsis's 's multi-organ pathophysiology. Patient-level 5-fold cross-validation follows the same methodology as sepsis: 60% training, 20% validation, 20% testing.
H.2
Acute Hypotension Treatment Formulation for RL
State Space. The state space comprises 18 features: 11 continuous physiological measurements and 7 binary data availability indicators (Table
Table 4: Acute hypotension state space features
Category
Features
Type/Unit
Hemodynamic
MAP, Systolic BP, Diastolic BP
numeric (mmHg)
Respiratory
PaO2. FiO,
numeric (mmlHg), categorical
Renal
Urine output, Creatinine
numeric (mL), numeric (mg/dL)
Hepatic
ALT, AST
numeric (IU/L)
Metabolic
Lactate
numeric (mmol/L)
Neurological
GCS
binary
Mcasurement
Urine (M), ALT/AST (M), FIO2 (M),
binary indicators
Indicators
GCS (M), PaO2 (M), Lactate (M), Creatinine (M)
The 7 binary variables (with suffix (M)) indicate whether a variable was measured at a specific point in time, which in medical time series is usually highly informative. Key differences from sepsis include explicit separation of MAP, systolic, and diastolic blood pressure as primary hemodynamic indicators, hepatic function markers (ALT, AST), and metabolic marker (lactate) as direct indicator of tissue perfusion adequacy.
Action Space. The action space is 2-dimensional, representing hourly fluid boluses (mL/hour) and vasopressor dosage (meg/kg/min). While originally categorical in the dataset, treat them as con- tinuous to maintain consistency with sepsis experiments and enable fine-grained policy optimization.
Reward Function. We define piecewise linear reward based solely MAP:
33

---

1.0
if MAP 2 65 mmHg
0.2 x (65 - MAP)
60 S MAP < 65 mmHg
(32)
1.0 1.0 (60 MAP)
MAP 60 mmHg
This continuous single-variable reward contrasts with the discrete multi-organ SOFA score used in sepsis. The piecewise structure ensures continuity at the breakpoint (60 mmHg) while imposing steeper penalties for critically low blood pressure. Missing MAP values are assigned zero reward. This MAP-focused reward deliberately simplifies hypotension management to test OGSRL's effectiveness across fundamentally different reward structures.
Two physiological safety costs ensure adequate organ perfusion. The urine output constraint flags states where urine production falls below 0.5 /l/////ouur 20 The lactate constraint flags states where lactate exceeds 2.0 mmol/L. [46]. The urine threshold ensures adequate renal perfusion. The lactate threshold represents a clinically established cutoff for detecting tissue typoperfusion-tle primary pathophysiologial consequence of severe hypotension. These two safety constraints provide a rigorous test of OGSRL's adaptability to speecic--peccfic physiological boundaries.
Safety Constraints to Physiological States.
Guardian Construction. KDE-based classifier is applied on 20-dimensional state-action pairs with Gaussian kernel. Complete methodology is described in Appendix G.3
Transition Dynamies Model. Following the sepsis treatment methodology (Appendix G.S maintain two separate models: k-NN-train (trained on 60% training partition, used during policy learning) and k-NN-eval (trained full dataset, used for policy evaluation). The k-NN-eval model provides more demanding test of the guardian's 's ability to prevent OOD exploration by having broader range of state transitions.
Policy Learning Algorithms To maintain conciseness while demonstrating cross-disease consistency, Table 2 reports results for representative methods: CQL and GCQL (model-free), MB-CPO and GMB-CPO (model-based), and SOC. We focus on MCR and AIR as these metrics directly assess clinical alignment and safety responsiveness the key dimensions for validating cross-disease gener- alizability. Policy evaluation follows the same protocol as sepsis: sample initial states from the test set, select actions according the learned policy, predict next states using i-NN-eval, and continue rollout for up to 48 steps. We compute MCR, AIR, and cumulative reward for each trajectory. All results are averaged over random seeds account for stochasticity in policy initialization and training.
H.3 Comparison with Sepsis Experiments
Table 5: Key differences between validation domains
Characteristic
Sepsis
Hypotension
Dataset source
MIMIC-III (real)
Health Gym (synthetic) 3,910 187,680
ICU stays
18,923
Statc-action pairs
247,733
Temporal resolution
4-hour intervals 72 hours (18 steps)
Hourly intervals 48 hours (48 steps)
Episode duration
State dimension
13 features
18 features Hemodynamic instability
Primary condition
Multi-organ dysfunction
Reward type
Discrete composite (SOFA -1)
Continuous variable (MAP)
Safety constraints
SpO² urine
Urine + lactate
Clinical objective
Reduce organ failure
Restore blood pressure
These substantial differences test OGSRL's cross-disease generalizability across multiple dimensions: (1) Disease pathophysiology sepsis involves systemic inflammation and multi-organ organ dysfunction, while hypotension focuses on acute hemodynamic compromise; (2) Temporal dynamics-hourry vs. 4-hour resolution tests adaptability to different decision frequencies; (3) State complexity vs. 13 features tests guardian scalability; (4) Reward structure discrree multi-organ composite (SOFA) vs.
continuous single-
single-variable variable (MAP) tests effectiveness across fundamentally different optimization
34

---

objectives; (5) Safety constraints different physiological boundaries test accommodation of disease- specific requirements; (6) Dataset sizelemaller cohort tests performance with limited data coverage. Despite these variations, consistent guardian benefits across both domains (Section 4.3) validate OGSRL as a robust framework for safe offline RI. in critical care.
I Limitations
The OGSRL framework exhibits several constraints that merit consideration. Its conservative ap- proach, while ensuring safety, potentially restricts the discovery of innovative treatment strategies beyond observed clinical practices paritess---pticularlly relevant in evolving sepsis management. Despite advancing toward continuous representation, the implementation still simplifies the multifaceted nature of sepsis interventions, which typically encompass antibiotics, ventilation adjustments, and nutritional support beyond the modeled fluid and vasopressor dimensions. The fixed 4-hour dis- cretization window fails capture the rapid physiological fluctuations that might necessitate more frequent clinical interventions. Generalizability concerns arise from the MIMIC-III dataset's limited institutional scope, as treatment patterns from a single hospital system may not translate across diverse healthcare settings with varying protocols and patient demographics. Moreover, the guardian mecha- nism sacrifices interpretability for statistical robustness, creating potential barriers to clinical trust since its safety boundaries emerge from complex statistical properties rather than transparent medical reasoning. All these limitations are practical challenges in applications, and appropriate adaptations to the proposed OGSRL framework will be implemented for real-world clinical deployment.
J Experiments Compute Resources
All experiments were conducted on a high performance computing (HPC) cluster equipped with NVIDIA A100 and V100 GPUs.
K
Broader Impact
The proposed framework, Offline Guarded Safe Reinforcement Learning (OGSRL), aims to improve treatment decision-making in high-stakes clinical settings using offline reinforcement learning. By introducing an OOD guardian and explicit safety cost constraints, OGSRL enables the development of safe and reliable treatment policies that remain grounded in observed clinical data. This is particularly impactful in domains such as ICU treatment, where policy optimization must adhere to strict safety boundaries due to patient risk.
The primary benefit of this work lies in its ability to learn treatment strategies that outperform clinician policies while preserving safety and trustworthiness. Since method constrains policy learning within the support of historical clinician decisions, it ensures that learned interventions do not extrapolate dangerously beyond medical expertise. Furthermore, including theoretical safety guarantees makes our framework more suitable for deployment in clinical decision-support tools than prior offline RL approaches that lack such safeguards.
However, like all machine learning methods applied to healthcare, there are risks. Improper interpre- tation or deployment of learned policies without proper clinical oversight could lead to misuse. We strongly emphasize that OGSRL designed as a decision-suu -spppot tool, not a substitute for human medical judgment.
To mitigate potential negative impacts, we advocate for responsible deployment in collaboration with healthcare professionals, rigorous post-hoc evaluation in simulated environments, and continuous monitoring in real-world applications. By combining domain knowledge with safe offline learning, we believe our framework contributes positively to the development of transparent, interpretable, and trustworthy AI systems for healthcare.
35

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims in the abstract and introduction precisely state the paper's 's con- tribution in Offline RL for medical treatment with theoretical and experimental results.
reflect
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We give separate "Limitations" part Appendix D
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
36

---

Answer: [Yes]
Justification: We have provided the full set of assumptions and a complete proof. The complete proofs are included in the Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We have fully disclosed all the information needed to reproduce the main experimental results of the paper in Section and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
37

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We provide code for documentation for environment setup, data preparation, implementation of all RI. algorithms and reproducing all experiments and figures presented in the paper in the supplementary material. Our experiments use the publicly available MIMIC-III dataset, which can be accessed through PhysioNet (https:/ https:://phsiinee org/ content/mimicoo after completing the required CITI training for protected health information.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis. cc/
sablic/guides/ httee/guiids//oeeeuuiiieeee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https http:////ooeeuuiiisiiooooool.oe
//nips. cttpsssccppbbccccc/cp guwdes/ddd/eee httpp////aaataaeeeeeoo/taaaaaaaa
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide the experimental details in Appendix G
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We provide the error bars to show the statistical significance of the experiments, which are given in Figure Besides, we present the results the experiments with different seeds in the Appendix G.7
Guidelines:
The answer NA means that the paper does not include experiments.
38

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the information on the computer resources in Appendix]
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We conducted the research conforming in every respect with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
39

---

Justification: In the final paragraph of the Introduction and the Conclusion, we discuss the societal impacts of our work. Our approach improves the reliability, safety, and reward performance of offline RI algorithms by fully leveraging the clinician data's knowledge. This advancement promotes the application of offline RI. in healthcare scenarios. Besides, we have found some negative societal impacts of the work, such risks for healthcare applications, which normal for all machine learning methods. The details are given in Appendix K
Guidelines:
The answer NA means that there is no socictal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeet of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
This paper does not include any data or models with a high risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer:[YYas Yes]
Justification: The creators or original owners of assets used in the paper are properly credited and are the license and terms of explicitly mentioned and properly respected.
40

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets. We use the existing toolbox or data sets for the experiment to validate our theory.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not include any research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
41

---

Answer:
[NA]
Justification:
This paper does not include any research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: LLM is not used as an important, original, or non-standard component of the core methods in this research.
Guidelines:
The answer NA means that the core method development in this research does not non non- sta-standard components.
involve LLMs as any important, original,
Please refer to our LLM policy for what should or should not be described.
42


Reviews:
[{"review_id": "qdxPnJ6VbH", "reviewer": "Reviewer_oUug", "summary": "This paper studies safe reinforcement learning (RL) in the medical context, specifically constraining policies to trace trajectories that are within a safe region, as well as satisfy numerical safety constraints. The paper introduces a new framework, Offline Guarded Safe RL (OGSRL), that combines the constraints into a dual constraint mechanism. Theoretical guarantees on safety satisfaction and high-probability optimal performance are proved, and empirical results validate theory insights on a sepsis treatment dataset.", "questions": "In Figure 3(b), why do GCQL and GCCQL result in an increase in unsafe states? Do the trends depend on the specific metrics used? \n\nIn Figure 1, how to interpret the expansion in visited regions? Are there unsafe regions that should be avoided?", "limitations": "Yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. As far as I know, the OGSRL framework is novel in the safe RL literature. The framework is clearly defined and well-motivated by clinical concerns. \n\n2. The theoretical and empirical results corroborate to illustrate the clinical insights well. \n\n3. Experiments show comparisons across extensive baselines and their safe-guarded variants. \n\nWeaknesses:\n\n1. Experiments are only conducted on one dataset. Even though this is mainly a theoretical paper, given that it is grounded in medical applications, it would make the results stronger to include one or two more relevant datasets. \n\n2. The proposed framework is model-based, which may be irrelevant in problems with large state spaces. It would be helpful to discuss such limitations and extensions to model-free counterparts.", "was_revised": true, "final_justification": "The changes proposed in rebuttal will help strengthen the paper. I will maintain my original recommendation.", "rebuttal": "We appreciate the reviewer's valuable feedback and comments. \nWe will address the comments about the weakness first and then answer the question.\nAll the concerns are addressed in a point-to-point way as follows. \n\n**Only One Dataset [Weakness 1].**\nThank you for the suggestion. We agree that evaluating our framework on additional datasets strengthens the evidence for its generalizability. In response, we tested our approach on the Synthetic Acute Hypotension Dataset [1] from PhysioNet, which provides ideal complementary validation as it focuses on acute hypotension management—a different critical care condition from sepsis—while maintaining comparable data structure. This dataset contains 3,910 ICU stays with 187,680 hourly state-action pairs over 48 hours, featuring an 18-dimensional physiological feature space and 2-dimensional action space (administered fluid boluses and vasopressors). Our findings demonstrate strong consistency with the sepsis results. We will include a \"Validation of Generalizability\" subsection in the appendix of the camera-ready version to detail this evaluation.\nThe guardian mechanism shows consistent improvements across both datasets. For model-free approaches, GCQL achieved near-perfect clinician alignment with an MCR of 0.973 ± 0.002, an 18% improvement over CQL's 0.824 ± 0.004. More dramatically, GMB-CPO transformed MB-CPO's near-zero concordance (0.060 ± 0.008) to a clinically meaningful 0.700 ± 0.063. Regarding clinical safety, GMB-CPO achieved the highest AIR at 0.482 ± 0.071—a 17% improvement over MB-CPO (0.411 ± 0.036) and substantially higher than both CQL (0.281 ± 0.031) and GCQL (0.301 ± 0.046). This confirms that the guardian mechanism not only constrains policies to safe regions but also enables more clinically appropriate responses during patient deterioration. As for the cumulative rewards, our proposed OGSRL (GMB-CPO) achieved the highest mean value of 14.88 (median: 16.14), outperforming the model-based method without guardian (MB-CPO, mean 3.9, median 5.38), the model-free method (GCQL, mean 12.42, median 12.69), and the standard of care (SOC, mean 10.37, median 11.21). These findings are consistent with the MIMIC-III results presented in Figure 2 and suggest that OGSRL can deliver improved health outcomes. These cross-disease results validate OGSRL as a generalizable framework for safe offline RL in critical care, with consistent guardian benefits across different medical conditions.\n\n[1] Kuo, N., Finfer, S., Jorm, L., & Barbieri, S. (2022). Synthetic Acute Hypotension and Sepsis Datasets Based on MIMIC-III and Published as Part of the Health Gym Project (version 1.0.0). PhysioNet. \n\n**Model‑free Counterparts [Weakness 2].**\nThank you for raising this point. While our main contribution is a model-based framework, we do discuss its applicability to model-free methods in Section 3.1 and Appendix G.4. Specifically, we show how the guardian mechanism can be integrated with CQL and CCQL to constrain policy learning within the data support, and evaluate these variants (GCQL, GCCQL) in our experiments.\nWe will clarify this connection by adding a brief remark in the experimental section of the camera-ready version, which will make our presentation better. \nThanks for your advice.\n\n**Increase in Unsafe States (GCQL and GCCQL) [Question 1].**\nThank you for the insightful question. GCQL and GCCQL are built to improve cumulative reward (or equivalently reduce SOFA score) while suppressing OOD actions, but they are not formulated to handle explicit **safety constraints** as in constrained MDPs. In our setting, we enforce constraints on two physiological indicators: SpO₂ and urine output. These indicators are **not part of the reward function** and must be separately constrained to ensure safety.\nSpO₂ is moderately correlated with SOFA, so improving SOFA under GCQL/GCCQL can indirectly help maintain SpO₂ within a safe range. However, **urine output is not directly aligned with SOFA**, so methods that only optimize reward (e.g., GCQL/GCCQL) may lead to unsafe values for this indicator. In contrast, **our GCPO explicitly includes both indicators as constraints**, ensuring they remain within clinically acceptable bounds while still improving reward.\nThis result illustrates the importance of formulating medical treatment problems as **constrained MDPs** rather than unconstrained ones, especially when domain knowledge highlights specific physiological safety requirements.\n\n**Expansion in Visited Regions (Figure 1) [Question 2].**\nThank you for this helpful question. In Figure 1, the blue points represent next states under the standard-of-care (SOC) policy and indicate the **in-distribution (ID) region**—i.e., where the dataset provides coverage and the learned dynamics are reliable. This region should **not be interpreted as uniformly safe**, but rather as the support of the data.\nSafety is handled separately via explicit constraints (e.g., on SpO₂ and urine output), and these constraints are enforced **within** the ID region. Outside the ID region (Or inside the OOD region), model predictions become unreliable—even if a policy appears “safe” there, we cannot trust it. with the proposed Guardian, our theoretical results (e.g., Theorem 1, Corollary 1) provide high-probability safety guarantees since the proposed Guardian can constrain the policy inside the ID region.\nThus, the expansion should be interpreted as a measure of generalization beyond the dataset, not as an indication of safety."}, {"review_id": "akWs7QQs7E", "reviewer": "Reviewer_N8n5", "summary": "The manuscript tackles a classic challenge in offline RL: “don’t go off the rails.” This problem is critical for offline RL in healthcare because—once the data manifold is left—value estimates become unreliable and treatment suggestions can turn dangerous. The authors propose Offline Guarded Safe Reinforcement Learning (OGSRL), a model-based framework that adds two safety layers to policy learning:\n* An “OOD guardian.” A classifier is learned from data to carve out the state–action region actually supported by clinical trajectories. An additional cost term penalizes any policy that strays outside that set, so exploration happens only where the model is trustworthy.\n* Medical safety constraints. Domain rules (e.g., minimum SpO₂, urine output) are encoded as cumulative costs inside a CMDP.\n\nThe authors prove that any policy satisfying both constraints (i) stays in distribution with high probability and (ii) is near-optimal with respect to the best policy supported by the data.", "questions": "1. Lemma 2 implies that the polynomial degree $d$ must grow with dimension, making the SDP intractable for real EHR spaces. In practice, the PSoS fit is replaced by a kernel-density heuristic (App G.3). Do the formal guarantees still apply in this setting?  \n2. The guardian idea resembles “probabilistic shields.” Could the manuscript discuss this connection in greater depth?  \n3. Performance appears to depend heavily on guardian hyper-parameters ($\\alpha,\\ \\bar c_g$). A sensitivity or robustness analysis would strengthen the results.  \n4. Is the train/test split performed at the patient level (each ICU stay in only one fold) or at the time-window level?  \n5. All experiments use adult MIMIC-III sepsis data with two control variables (fluids, vasopressors) and 4-hour windows. Please discuss whether the method is specific to this dataset and task.  \n6. Off-Dynamics RL (e.g., “Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifiers”) also uses classifier-based shaping. A comparison would be valuable.  \n7. The theoretical coverage guarantee (Thm 1 + Cor 1) relies on Hoeffding’s inequality and therefore treats each state–action pair as IID, even though 4-hour ICU slices are highly correlated. How sensitive is the guardian’s safety bound to this autocorrelation? Has a block-bootstrap or mixing-time analysis been attempted to estimate an effective sample size and adjust the $\\delta$-confidence level? Would thinning trajectories (e.g., one sample every 24 h) materially change the guardian set or downstream performance?", "limitations": "see above", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. The manuscript is generally well-structured and easy to follow.\n2. Solid theory: finite-sample safety and performance bounds are derived for the guardian-augmented CMDP (Theorems 1–3), and the evaluation is thorough—seven baselines, guardian vs. no-guardian ablations, and multiple clinical metrics (MCR, AIR, mortality, physiological safety).\n3. The method extends prior conservative approaches (e.g., CQL) by jointly bounding states and actions, merging guardian concepts with CMDP safety in a principled way. The probabilistic in-support guarantee appears to be new to medical RL.\n\n**Weaknesses**\n1. In the experiment, the train and test sets are created via a purely random five-fold split. It is therefore unclear whether the policy encounters genuine OOD points during testing; a detailed train/test analysis would help.\n2. Appendix G.5 states that evaluation begins with real test-set states, after which the k-NN dynamics model (trained on the 60 % training fold) predicts subsequent states. Consequently, the simulated trajectories remain similar to the training distribution, so the OOD guardian is not fully stressed.\n3. The guardian and dynamics models are fitted only on the training fold, yet the held-out 40 % has a very similar distribution. Thus the reward gains and OOD-avoidance curves in Figure 2 and Table 1 are judged inside a comfort zone already labeled as safe. The manuscript does not show how the method behaves under a genuine distribution shift.", "was_revised": true, "final_justification": "My main concerns have been substantially addressed, so I remian my rating accordingly.", "rebuttal": "We sincerely appreciate the reviewer’s constructive comments and thoughtful questions. Your feedback has not only helped us identify ways to improve the clarity and quality of our manuscript but also encouraged deeper reflection on our research. Beyond your technical insights, we are also grateful for your patience and kind engagement throughout the review process.\n\nBelow, we respond point-by-point to your questions first, followed by a discussion of the identified weaknesses and limitations.\n\n**Formal Guarantee for KDE-base Guardian [Question 1].**\nThank you for the insightful question. While the KDE-based guardian used in practice is not an explicit solution to the PSoS optimization, it can be interpreted as approximating a feasible (but possibly suboptimal) sublevel set in the PSoS formulation. That is, there exists a higher-degree PSoS function whose sublevel set closely matches the KDE support and satisfies the PSoS constraint at a relaxed coverage threshold $\\tilde{\\alpha} < \\alpha$.\nAs a result, Theorem 1 still applies with a reduced confidence level corresponding to $\\tilde{\\alpha}$. This provides a valid—albeit looser—probabilistic guarantee for the KDE-based implementation.\n\n**Connection to Probabilistic Shield [Question 2].**\nThank you for the helpful suggestion. In Section 3.2, we briefly discuss the connection between our guardian and shield-based methods. Following your advice, we will expand this discussion in the camera-ready version to more explicitly relate our guardian to *probabilistic shields*—highlighting the shared goal of constraining undesired behaviors, but emphasizing that our method operates entirely offline and provides high-probability guarantees over state-action trajectories, rather than real-time action filtering.\n\n**Sensitivity or Robustness analysis [Question 3].**\nThank you for raising this important point. We agree that the guardian’s performance can be influenced by its hyperparameters ($\\gamma$, $\\beta$,$H$, $\\alpha_{c}$). In our implementation, these values were selected based on theoretical guidance (e.g., from Lemma 1 and Appendix B) and prior literature to ensure reasonable coverage and conservativeness.\nWhile we did not perform a full sensitivity analysis, we agree that it is a valuable direction for future work, especially to better understand trade-offs between conservativeness and performance.\n\n**Patient Level or Time Level [Question 4].**\nThank you for this question. The split is performed at the patient (ICU‑stay) level: every complete trajectory belongs to a single fold, so no patient’s data appear in both the training and the test partitions. In the camera-ready version, we will clarify the patient-level splitting methodology.\n\n**Method's generality [Question 5].**\nThank you for inviting us to clarify the scope of our approach.  OGSRL is not confined to the adult sepsis task with 4‑hour windows and two control variables. To demonstrate its broader applicability, we trained the OGSRL on the Synthetic Acute Hypotension cohort [1] (PhysioNet 2022), which represents a different disease and patient population. A detailed account of this cross‑cohort study appears in our response to weakness 1 raised by Reviewer oUug and we will include a \"Validation of Generalizability\" subsection in the appendix of the camera-ready version. The hypotension dataset includes 3,910 ICU stays with 187,680 hourly state-action pairs over 48 hours, featuring an 18-dimensional physiological feature space and 2-dimensional actions (fluid boluses and vasopressor administration). The guardian mechanism consistently improved performance on both datasets. For example, on the Acute Hypotension dataset, the model-free GCQL with guardian achieved near-perfect clinician alignment (MCR: 0.973 ± 0.002), an 18% improvement over CQL without guardian (0.824 ± 0.004). The guardian had an even greater impact in model-based methods: GMB-CPO increased MB-CPO’s concordance from 0.060 ± 0.008 to 0.700 ± 0.063. In terms of clinical safety, GMB-CPO achieved the highest AIR at 0.482 ± 0.071—a 17% improvement over MB-CPO (0.411 ± 0.036) and substantially higher than CQL (0.281 ± 0.031) or GCQL (0.301 ± 0.046). Cumulative reward analysis further supports these results: OGSRL (GMB-CPO) achieved the highest performance (mean: 14.88, median: 16.14), outperforming the model-based method without guardian (MB-CPO: mean 3.9, median 5.38), the model-free approach (GCQL: mean 12.42, median 12.69), and standard of care (SOC: mean 10.37, median 11.21). These findings are consistent with our MIMIC-III results (see Figure 2) and underscore OGSRL’s potential to enhance patient outcomes. These cross-disease validation results establish OGSRL as a generalizable framework for safe offline reinforcement learning in critical care, with the guardian mechanism providing consistent benefits across diverse medical conditions.\n\n[1] Kuo, N., Finfer, S., Jorm, L., & Barbieri, S. (2022). Synthetic Acute Hypotension and Sepsis Datasets Based on MIMIC-III and Published as Part of the Health Gym Project (version 1.0.0). PhysioNet.\n\n**Connection with Off-Dynamics RL [Question 6].**\nThank you for pointing this out. While our approach and Off-Dynamics RL both use classifiers to influence learning, the goals are different. Our guardian is designed to restrict policy optimization to the in-distribution region with formal safety guarantees, whereas Off-Dynamics RL typically uses classifiers for reward shaping or domain adaptation.\nTo clarify this distinction, we will add a brief remark in Section 3.3 of the camera-ready version. We agree that highlighting this difference can help prevent potential confusion, but note that our method is focused on safety rather than transfer.\n\n**Sensitivity to Correlation [Question 7].**\nThank you for this insightful question. We agree that ICU trajectories exhibit temporal correlation. However, the guardian’s support estimation (Theorem 1) is performed over individual state-action pairs, not full trajectories, and the dataset consists of many such pairs drawn from diverse patient episodes and treatment decisions. \nThis allows us to treat the collection of (s, a) pairs as approximately IID for the purposes of applying Hoeffding’s inequality, or we can randomly resample the points again for the guardian's support estimation over individual state-action pairs, which is a standard and accepted approach in support estimation literature. While this may lead to slightly conservative bounds in theory, the probabilistic guarantees remain meaningful in practice. Incorporating techniques like block bootstrapping or mixing-time analysis is an interesting direction for future work.\n\n**Train/Test Set Split [Weakness 1].**\nThank you for raising this important point. While the train/test split is random, it does result in different distributions: the **training set does not fully cover** the support of the test set. As a result, the policy—trained only on the training data—can encounter **genuinely out-of-distribution (OOD)** state-action pairs during evaluation.\nFurthermore, the learned policy generates its own trajectories during rollout, which may deviate from the behavior policy (e.g., clinician policy). This can further push the policy into OOD regions, even if the test set itself contains only in-distribution samples. As shown in Figure 1, methods without a guardian explore outside the data support, illustrating the practical relevance of OOD control.\nTherefore, even under a random split, both **data coverage gaps** and **policy rollout** contribute to OOD exposure—making the guardian mechanism essential for reliability.\n\n**Implementation of kNN [Weakness 2].**\nThank you for highlighting this issue. We acknowledge that we did not spell out a key implementation detail: we employ two separately trained k‑nearest‑neighbour dynamics models. k‑NN‑train, fitted on the 60 % training fold, is used during policy learning, while k‑NN‑eval, trained on the entire dataset, generates trajectories for evaluation. Since k‑NN‑eval has access to a wider range of state–action pairs, the roll‑outs it produces are not anchored to the policy’s training manifold, exposing the guardian to a more demanding out‑of‑distribution test. We will explain this setup explicitly and retain the names k‑NN‑train and k‑NN‑eval in Section G.5, “Application of k‑NN Models in Model‑based Methods and Policy Evaluation,” of the camera‑ready manuscript.\n\n**Five-Fold Split [Weakness 3].**\nThank you for this important observation. We should clarify that our five-fold split is performed at the patient level—entire patient trajectories are assigned exclusively to either training or testing, ensuring no information leakage. Patient-level heterogeneity ensures test patients have unique physiological characteristics and treatment responses never seen during training. We acknowledge that testing under more extreme distribution shifts (temporal, cross-institution, or severity-based splits) would further strengthen our evaluation. In the camera-ready version, we will clarify the patient-level splitting methodology and discuss the benefits of testing on more challenging distribution shifts."}, {"review_id": "f71h5rOcyy", "reviewer": "Reviewer_S2be", "summary": "This work focuses on applying offline RL algorithms to healthcare scenarios when learning from clinical behavior. In this context, a primary challenge is inappropriate generalization due to explorations out-of-distribution. This work proposes the Offline Guarded Safe Reinforcement Learning (OGSRL) to constrain optimization within regions where there is state-action support. In contrast to past work, which only focuses on constraining out-of-distribution actions, this work extend this to the full support of state-action pairs. This is combined with a safety cost constraint aimed at providing domain-specific safeguards that can encode medical knowledge. Theoretical guarantees on safety and near-optimality are provided, where policies using the learning algorithm can achieve performance close to the best possible policy supported by the data. Results show that OGSRL better handles out-of-distribution detection and generalization, and achieves better performance compared to clinical decisions.", "questions": "1. What is the tolerance to poor measurement of safety rules? \n2. How do simpler methods for guardian support (e.g., simple classifiers) perform as shielding mechanisms? \n3. How exactly is \"in-distribution\" defined when learning the guardian? Should there be sufficient examples of the state/action pair, or is only one example enough to be in-distribution?", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "**Strengths**\n\n1. The work does a good job of setting up the limitation of past work with respect to one constraining actions vs constraining state-action pairs in terms of long-term trajectories. \n2. The idea of a guardian model which can detect in-support state-action pairs is very interesting and a great way of solving the problem. \n3. The framework including cost-sensitivity is a good idea and well-suited for clinical application. This specific call-out in terms of the two definitions of safety (one for learning and one to encode medical knowledge) is appreciated. \n4. The results are compelling and who the learned state-actions are generally clustered in-distribution. The improvement on baselines is also major. \n5. The theoretical results are compelling and provide good evidence for the overall learning algorithm.  \n\n**Weaknesses**\n\n1. The explanation of how to learn the guardian using polynomial sum-of-squares is a bit confusing and the presentation in section 3.1 as a whole could be improved. \n2. The results are only validated on MIMIC-III. It would be useful to see how results transfer to other healthcare datasets. However, given the depth of results, this limitation is not quite severe.", "was_revised": true, "final_justification": "Author response helped alleviate many questions/concerns, and kept score accordingly.", "rebuttal": "We sincerely appreciate the reviewer’s encouraging comments. Your feedback and suggestions are highly valuable and have helped us improve the clarity and quality of our manuscript.\n\nBelow, we respond to your questions first, followed by point-by-point responses to the comments regarding weaknesses.\n\n**Tolerance to Poor Measurement of Safety Rules [Question 1].**\nThank you for this important question. Safety thresholds like SpO₂ and urine output are based on clinical guidelines but may vary in practice. Our framework remains robust to such variability, as the constraints can be conservatively defined. Since policy optimization is restricted to the in-distribution region—where model predictions are reliable—the risk from noisy or uncertain measurements is further mitigated.\nIn principle, robustness can be improved by tightening the safety thresholds, though this may reduce optimality. Investigating this trade-off in a data-driven and quantitative manner is an interesting direction for future work.\n\n**Simpler Guardian Support as Shielding Mechanisms [Question 2].**\nThank you for this meaningful question. In practice, we adopt a kernel-based guardian (Appendix G.3) for scalability, while the PSoS formulation provides theoretical safety guarantees.\nWhile simpler classifiers (e.g., logistic regression or Gaussian processes) can be used to construct shields and may support pointwise safety guarantees in online settings, the **focus and theoretical scope of our guardian are different**. Traditional shielding methods are designed to avoid unsafe *actions* during **online exploration**, and their guarantees are typically pointwise (e.g., avoid specific failures at each time step).\nIn contrast, our guardian is designed to constrain the policy to **in-distribution state-action regions**, where the learned dynamics model is reliable—ensuring **offline reliability** over entire trajectories with high probability and then contributes to the safety (Theorem 1). Moreover, our guardian applies a **penalty-based shaping mechanism** (In GCQL) or **constrained shaping mechanism** (in GMB-CPO) during policy optimization, discouraging unreliable generalization beyond the dataset.\nWe believe this distinction in purpose and guarantees is central, and we appreciate the opportunity to clarify it.\n\n\n**Definition of \"in-distribution\" [Question 3].**\nThank you for this important question. Conceptually, the in-distribution (ID) region refers to the set of state-action pairs where the data-collecting process (i.e., the behavior policy and system dynamics) has non-negligible support. This is the region where the learned models (e.g., reward, transition) are expected to be reliable. Importantly, the ID region typically covers only a subset of the full state-action space.\nIn practice, we approximate the ID region using either a polynomial sublevel set (PSoS) or kernel density estimation (KDE), based on a desired empirical coverage level α (see Appendix G.3). In both cases, a single occurrence of a state-action pair is not sufficient to be considered in-distribution; instead, local density is required—i.e., the point must lie in a region consistently populated by the dataset.\nFortunately, the clinician behavior policy provides sufficient coverage of such regions in practice. We do not require repeated sampling of specific state-action pairs or full trajectories. Even a single trajectory can contribute to learning a meaningful guardian. However, when the dataset is small, it is important to adopt more conservative thresholds to ensure both in-distribution validity and safety.\n\n**Presentation of Section 3.1 [Weakness 1].**\nThank you for the helpful feedback. We agree that the explanation of the polynomial sum-of-squares (PSoS) approach in Section 3.1 could be made clearer, especially for readers less familiar with this technique. While the current version provides all necessary definitions and guarantees, we will make minor adjustments to improve readability—clarifying the motivation, simplifying the notation where possible, and briefly reinforcing the connection to the practical KDE-based implementation.\n\n**Single Dataset Validation [Weakness 2].**\nThank you for this valuable suggestion. We have validated our approach on the Synthetic Acute Hypotension Dataset [1] from PhysioNet, which complements our sepsis analysis by examining a different critical care condition. This dataset contains 3,910 ICU stays with 187,680 hourly observations, featuring 18 physiological dimensions and 2-dimensional vasopressor and fluid administration actions.\nThe results strongly support our method's generalizability. GCQL achieved near-perfect clinician alignment (MCR: 0.973 ± 0.002), improving CQL by 18%. More notably, GMB-CPO transformed MB-CPO from near-zero concordance (0.060 ± 0.008) to clinically meaningful alignment (0.700 ± 0.063). For safety metrics, GMB-CPO achieved the highest AIR (0.482 ± 0.071)—17% better than MB-CPO and substantially outperforming both CQL and GCQL. Regarding cumulative rewards, our proposed OGSRL (GMB-CPO) achieved the highest mean value of 14.88 (median: 16.14), outperforming the model-based method without a guardian (MB-CPO: mean 3.9, median 5.38), the model-free method (GCQL: mean 12.42, median 12.69), and current standard of care protocols (SOC: mean 10.37, median 11.21). These results are consistent with the findings on the MIMIC-III dataset shown in Figure 2, indicating that OGSRL can deliver improved health outcomes.\nThese cross-disease results confirm OGSRL as a robust framework for safe offline RL in critical care. The guardian mechanism consistently improves both clinician alignment and safety across diverse medical conditions. We will include these findings in a \"Validation of Generalizability\" subsection in the appendix of the camera-ready version. \n\n[1] Kuo, N., Finfer, S., Jorm, L., & Barbieri, S. (2022). Synthetic Acute Hypotension and Sepsis Datasets Based on MIMIC-III and Published as Part of the Health Gym Project (version 1.0.0). PhysioNet."}, {"review_id": "UXaUViBAWe", "reviewer": "Reviewer_BWnV", "summary": "The paper proposes a safe offline reinforcement learning method that aims to avoid OOD states by learning a guardian function, which constrains the policy to visit only states within the support of the offline dataset with high probability. By enabling policy improvement within this safe region, the learned policy can outperform clinician decisions while maintaining safety.", "questions": "1.\tIn Figure 1, why do some methods without the guardian (e.g., MB-TRPO and MB-CPO) predominantly visit regions outside the support of the offline dataset? Is this behavior due to overestimated values or modeling error in underrepresented regions?\n2.\tHow sensitive is the performance with respect to the choice of the empirical coverage threshold $\\alpha_c$? I think this is not a major issue, but an ablation study on the choice of $\\alpha_c$ would illustrate the trade-off between conservativeness and performance.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**  \n\nThe sepsis treatment task effectively illustrates the practical application of the proposed method. In addition, the authors provide a detailed theoretical and algorithmic analysis, highlighting the method’s potential generalizability to a wide range of offline reinforcement learning scenarios.\n\n**Weaknesses**  \n\nWhile the paper is generally well-written, clarity in certain sections could be slightly improved. One example is the experiment setup in Section 4.1, which can be improved by separating the dataset description, reward&safety mechanism design, and OGSRL implementation\n\nThe paper would benefit from an empirical investigation of how the conservativeness of the in-support region, defined either by $\\alpha_c$ in (GCL) or the outlier threshold in KDE, affects policy performance and safety.", "was_revised": true, "final_justification": "The changes proposed in rebuttal helped clarify my questions/concerns. I will maintain my original recommendation.", "rebuttal": "We thank the reviewer for their careful reading and thoughtful feedback, which have helped us identify several ways to further clarify and strengthen our manuscript. \n\nBelow, we respond point‑by‑point to the identified weaknesses and questions. \n\n**OOD Visits by Non-Guardian Methods [Question 1].**\nThank you for this important observation. Non-guardian model-based methods such as MB-TRPO and MB-CPO are indeed more likely to visit OOD regions because they perform multi-step policy optimization using a learned transition model. During planning, model prediction errors can compound over longer rollout horizons, leading the policy to exploit model inaccuracies and visit unrealistic state-action sequences that extend beyond the empirical distribution of the offline dataset. Unlike model-free CQL—which constrains the policy to actions supported by the data—model-based approaches can generate state–action sequences that move further away from the empirical distribution, resulting in higher OOD visitation. We will clarify this explanation in the revised caption of Figure 1 and the results discussion.\n\n**Sensitivity to α Threshold [Question 2].**\nThank you for highlighting the importance of $\\alpha$ sensitivity. In the camera-ready version, we will include a dedicated appendix section titled “Guardian Hyper-parameter Sensitivity” to systematically discuss how varying $\\alpha$ affects both reward and OOD risk. Our new experiments show that choosing $\\alpha$ between 5% and 30% achieves consistently strong results, with relatively minor differences across this range. As $\\alpha$ increases, the guardian becomes more conservative, enforcing stricter adherence to the offline support and reducing the risk of OOD violations, but this can also limit the achievable reward. In contrast, decreasing $\\alpha$ makes the support region less conservative, allowing for higher reward but at the cost of more frequent OOD violations. We will document this trade-off for both GCQL and GMB-CPO in the appendix of the camera-ready version.\n\n**Presentation of Section 4.1 [Weakness 1].**\nThank you for this feedback regarding the clarity of our experimental setup. We agree that Section 4.1 would benefit from a more organized structure. In the camera-ready version, we will move the content from Appendix G.2 (Sepsis Treatment Implementation for RL) into Section 4.1 and restructure this subsection into three clear parts. In the camera-ready version, we will restructure this section into three distinct parts: one for the dataset description, one for the reward and safety design, and one for implementation details. This reorganization will improve readability and allow readers to more easily understand each component of our experimental design independently. \n\n**Conservativeness of Support Threshold [Weakness 2].**\nWe appreciate the suggestion to provide a more systematic analysis of the conservativeness of the in-support region. In the camera-ready version, we will add a dedicated appendix section titled “Guardian Hyper-parameter Sensitivity”, where we examine how varying the $\\alpha$ threshold for both GCL and the KDE guardian influences policy performance (reward) as well as key evaluation metrics, including OOD risk rate (MCR) and clinical safety (AIR). Our preliminary results indicate that setting $\\alpha$ between 5% and 30% consistently yields strong performance. Higher $\\alpha$ values (stricter support) reduce OOD rviolations but may limit the achievable reward, whereas lower $\\alpha$ values relax the safety constraints, potentially increasing reward at the cost of greater OOD risk. Full results and details will be provided in the appendix of the camera-ready version."}]

Author rebuttal:

Meta-review: This paper proposes Offline Guarded Safe Reinforcement Learning (OGSRL), a novel framework designed to ensure safety in offline RL by combining two key mechanisms: (1) a guardian model that restricts the policy to operate within the empirical support of the data (state–action pairs), and (2) clinical safety constraints modeled via a constrained MDP (CMDP). The authors provide theoretical guarantees on both safety and near-optimality and validate their approach on a sepsis treatment task using the MIMIC-III dataset.

The paper’s strengths include a well-motivated problem setting, a conceptually novel safety framework, and solid theoretical contributions. The idea of jointly constraining states and actions goes beyond previous work focused solely on actions, addressing a critical issue in offline RL. The experiments are thorough, with multiple baselines, ablation studies, and clinically relevant metrics. The application to healthcare makes the work particularly impactful.

However, there are some limitations. The experiments are limited to a single dataset, and the train/test distribution is closely aligned, which may not fully stress the model’s OOD handling capability. The guardian model is practically implemented using KDE rather than the theoretically proposed polynomial method, and no robustness or sensitivity analysis is provided for key hyperparameters.

During the rebuttal, the authors clarified the train/test split and addressed the gap between theory and implementation. 

Overall, this is a technically solid and timely contribution to safe offline RL with strong relevance to healthcare. While not yet suited for a spotlight due to empirical limitations, it is a clear accept for presentation, offering important insights and a valuable foundation for future research.
Decision: Accept (spotlight)