Offline Inverse Constrained Reinforcement Learning for Safe-Critical Decision Making in Healthcare

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 16:18:24

Abstract: Reinforcement Learning (RL) applied in healthcare can lead to unsafe medical decisions and treatment, such as excessive dosages or abrupt changes, often due to agents overlooking common-sense constraints. Consequently, Constrained Reinforcement Learning (CRL) is a natural choice for safe decisions. However, specifying the exact cost function is inherently difficult in healthcare. Recent Inverse Constrained Reinforcement Learning (ICRL) is a promising approach that infers constraints from expert demonstrations. ICRL algorithms model Markovian decisions in an interactive environment. These settings do not align with the practical requirement of a decision-making system in healthcare, where decisions rely on historical treatment recorded in an offline dataset. To tackle these issues, we propose the Constraint Transformer (CT). Specifically, 1) utilize causal attention mechanism to incorporate historical decisions and observations into the constraint modeling and employ a non-Markovian layer for weighted constraints to capture critical states, 2) generative world model to perform exploratory data augmentation, thereby enabling offline RL methods to generate unsafe decision sequences. In multiple medical scenarios, empirical results demonstrate that CT can capture unsafe states and achieve strategies that approximate lower mortality rates, reducing the occurrence probability of unsafe behaviors.

Main paper: 
 introduction: 
 In recent years, the doctor-to-patient ratio imbalance has drawn attention, with the U.S. having only 223.1 physicians per 100,000 people [1] . AI-assisted therapy emerges as a promising solution, offering timely diagnosis, personalized care, and reducing dependence on experienced physicians. Therefore, the development of an effective AI healthcare assistant is crucial.Table 1 : The proportion of unsafe behaviors occurrences in vaso suggested by physician and DDPG. The typical range for vaso is 0.1 ∼ 0.2µg/(kg•min), with doses exceeding 0.5 considered high [2] . A cutoff value of 0.75 is identified as a critical threshold associated with increased mortality [3] .Drug dosage (µg/(kg • min)) Physician DDPG vaso > 0.75 2.27% 7.44% ↑ vaso > 0.91.71% 7.40% ↑ ∆ vaso > 0.75 2.45% 21.00% ↑ ∆ vaso > 0.9 1.88% 20.62% ↑ ∆ vaso: The change in vaso between two-time points.Reinforcement learning (RL) offers a promising approach to develop AI assistants by addressing sequential decisionmaking tasks. However, this method can still lead to unsafe behaviors, such as administering excessive drug dosages, inappropriate adjustments of medical parameters, or abrupt changes in medication dosages. These behaviors, such as "too high" or "sudden change" can significantly endanger patients, potentially resulting in acute hypotension, hypertension, arrhythmias, and organ damage, with fatal consequences [4, 5, 6] . For example, in sepsis treatment, patients receiving vasopressors (vaso) at dosages exceeding 1µg/(kg • min) have a mortality rate of 90% [7] . Moreover, the "sudden change" in vaso can rapidly affect blood vessels, causing acute fluctuations in blood pressure and posing life-threatening risks to patients [8] . Our experiments demonstrate that the work Submitted to 38th Conference on Neural Information Processing Systems (NeurIPS 2024). Do not distribute. [9] applying the Deep Deterministic Policy Gradient (DDPG) algorithm in sepsis indeed exhibits "too high" and "sudden change"1 unsafe behaviors in vaso recommendations, as shown in Table 1 .This paper aims to achieve safe healthcare policy learning to mitigate unsafe behaviors. The most common method for learning safe policies is Constrained Reinforcement Learning (CRL) [10, 11] ,with the key to its success lying in the constraints representation. However, in healthcare, we can only design the cost function based on prior knowledge, which limits its application due to a lack of personalization, universality, and reliance on prior knowledge. For more details about issues, please refer to Appendix A. Therefore, Inverse Constrained Reinforcement Learning (ICRL) [12] emerges as a promising approach, as it can infer the constraints adhered to by experts from their demonstrations.However, directly applying ICRL in healthcare presents several challenges: 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Vaso ( g/kg/min) The distribution of vaso for patients with the same state. The physician makes different decisions due to referencing historical information, while the agent based on Markov decision-making can only make the same decision.1) The Markov decision is not compatible with medical decisions. ICRL algorithms model Markov decisions, where the next state depends only on the current state and not on the history [13, 14] . However, in healthcare, the historical states of patients are crucial for medical decisionmaking [15] , as demonstrated in the experiments shown in Figure 1 . Therefore, ICRL algorithms based on Markov assumption can not capture patient history, and ignore individual patient differences, thereby limiting effectiveness.2) Interactive environment is not available for healthcare or medical decisions. ICRL algorithms [12, 16] follow an online learning paradigm, allowing agents to explore and learn from interactive environments. However, unrestricted exploration in healthcare often entails unsafe behaviors that could breach constraints and result in substantial losses. Therefore, it is necessary to infer constraints using only offline datasets.In this paper, we introduce offline Constraint Transformer (CT), a novel ICRL framework that incorporates patients' historical information into constraint modeling and learns from offline data to infer constraints in healthcare. Specifically, 1) Inspired by the recent success of transformers in sequence modeling [17, 18, 19] , we incorporate historical decisions and observations into constraint modeling using a causal attention mechanism. To capture key events in trajectories, we introduce a non-Markovian transformer to generate constraints and importance weights, and then define constraints using weighted sums. CT takes trajectories as input, allowing for the observation of patients' historical information and evaluation of key states.2) To learn from an offline dataset, we introduce a model-based offline RL method that simultaneously learns a policy model and a generative world model via auto-regressive imitation of the actions and observations in medical decisions. The policy model employs a stochastic policy with entropy regularization to prevent it from overfitting and improve its robustness. Utilizing expert datasets, the generative world model uses an auto-regressive exploration generation paradigm to effectively discover a set of violating trajectories. Then, CT can infer constraints in healthcare through these unsafe trajectories and expert trajectories.In the medical scenarios of sepsis and mechanical ventilation, we conduct experimental evaluations of offline CT. Experimental evaluations demonstrate that offline CT can capture patients' unsafe states and assign higher penalties, thereby providing more interpretable constraints compared to previous works [9, 20, 21] . Compared to unconstrained and custom constraints, CT achieves strategies that closely approximate lower mortality rates with a higher probability (improving by 8.85% compared to DDPG). To investigate the avoidance of unsafe behaviors with offline CT, we evaluate the probabilities of "too high" and "sudden changes" occurring in the sepsis. The experimental results show that CRL with CT can reduce the probability of unsafe behaviors to zero. 
related works: 
 Reinforcement Learning in Healthcare. RL has made great progress in the realm of healthcare, such as sepsis treatment [9, 20, 21, 22] , mechanical ventilation [23, 24, 25] , sedation [26] and anesthesia [27, 28] . However, these works mentioned above have not addressed potential safety issues such as sudden changes or too high doses of medication. Therefore, the development of policies that are both safe and applicable across various healthcare domains is crucial.Inverse Constrained Reinforcement Learning. Previous works inferred constraint functions by determining the feasibility of actions under current states. In discrete state-action spaces, Chou et al.[29] and Park et al. [30] learned constraint sets to differentiate constrained state-action pairs. Scobee & Sastry [31] proposed inferring constraint sets based on the principle of maximum entropy, while some studies [32, 33] extended this approach to stochastic environments using maximum causal entropy [34] . In continuous domains, Malik et al. [12] , Gaurav et al. [16] , and Qiao et al. [35] used neural networks to approximate constraints. Some works [11, 29] applied Bayesian Monte Carlo and variational inference to infer the posterior distribution of constraints in high-dimensional state spaces.Xu et al. [36] modeled uncertainty perception constraints for arbitrary and epistemic uncertainties.However, these methods can only be applied online and lack historical dependency.Transformers for Reinforcement Learning. Transformer has produced exciting progress on RL sequential decision problems [17, 18, 37, 38] . These works no longer explicitly learn Q-functions or policy gradients, but focus on action sequence prediction models driven by target rewards. Chen et al. [18] and Janner et al. [37] perform auto-regressive modeling of trajectories to achieve policy learning in an offline environment. Furthermore, Zheng et al. [17] unify offline pretraining and online fine-tuning within the Transformer framework.  
problem formulation: 
 We model the medical environment with a Constrained Markov Decision Process (CMDP) M c [39] ,which can be defined by a tuple (S, A, P, R, C, γ, κ, ρ 0 ). Similar to studies [23, 40] , we extract data within 72 hours of patient admission, with each 4-hour interval constituting a window or time step.The state indicators of the patient at each time step are denoted as s ∈ S. The administered drug doses or instrument parameters of interest are considered as actions a ∈ A, while reward function R is used to describe the quality of the patient's condition and provided by experts based on prior work [9, 23] . At each time step t, an agent performs an action a t at a patient's state s t . This process generates the reward r t ∼ R(s t , a t ), the cost c t ∼ C and the next state s t+1 ∼ P (• | s t , a t ), where P defines the transition probabilities. γ denotes the discount factor. κ ∈ R + denotes the bound of cumulative costs. ρ 0 defines the initial state distribution. The goal of the CRL policy π is to maximize the reward return while limiting the cost in a threshold κ:EQUATION)where T is the length of the trajectory τ . CRL commonly assumes that constraint signals are directly observable. However, in healthcare, such signals are not easily obtainable. Therefore, Our objective is to infer reasonable constraints for CRL to achieve safe policy learning in healthcare.Safe-Critical Decision Making with Constraint Inference in Healthcare. Our general goal is for our policy to approximate the optimal policy, which refers to the strategy under which the patient's mortality rate is minimized (achieving a zero mortality rate is often difficult since there are patients who can not recover, regardless of all potential future treatment sequences [41] ). Decision-making with constraints can formulate safer strategies by discovering and avoiding unsafe states, thereby approaching the optimal policy. However, most offline RL algorithms rely on online evaluation, where the agent is evaluated in an interactive environment, whereas in medical scenarios, only offline evaluation can be utilized.In previous works [5, 9, 40, 42] , they qualitatively analyzed by comparing the differences (DIFF)between the drug dosage recommended by our policy π and the dosage administered by clinical physicians π, and its relationship with mortality rates, through graphical analysis. In the graph depicting the relationship between the DIFF and mortality rate, at the point when DIFF is zero, the lower the mortality rate of patients, the better the performance of the policy [40] . To provide a more accurate quantitative evaluation, we introduce the concept of the probability of approaching the optimal policy, defined as ω:EQUATIONWe randomly collect 2N patients (with an equal number of known survivors and non-survivors under doctor's policy π) from the offline dataset. We then calculate the DIFF and sort it in ascending order.The optimality of the policy can be evaluated through the following two points: 1) The higher the survival probability (i.e., ω) of the top N patients, the lower the mortality rate can be achieved by executing π; 2) The smaller the DIFF among the surviving patients in the top N , the greater the probability that π is optimal. 
method: 
 To infer constraints and achieve safe decision-making in healthcare, we introduce the Offline Constraint Transformer (Figure 2 ), a novel ICRL framework.Inverse Constrained Reinforcement Learning. ICRL aims to recover the cost function C * by leveraging a set of trajectories D e = {τ (i) e } N i sampled from an expert policy π e , where N denotes the number of the trajectories. ICRL is commonly based on the Maximum Entropy framework [31] ,and the likelihood function is articulated as [12] :EQUATIONHere, Z M = exp(βr(τ ))I M (τ )dτ is the normalizing term. The indicator I M C (τ (i) ) signifies the extent to which the trajectory τ (i) satisfies the constraints. It can be approximated using a neural network ζ θ (τ (i) ) parameterized with θ, defined asζ θ (τ (i) ) = T t=0 ζ θ (s i t , a i t ).Consequently, the cost function can be formulated as C θ = 1 -ζ θ . Substituting the neural network for the indicator, we can update θ through the gradient of the log-likelihood function:EQUATIONwhere M ζθ denotes the MDP obtained after augmenting M with the cost function C θ , using the executing policy π M ζθ . And τ are sampled from the policy. In practice, ICRL can be conceptualized as a bi-level optimization task [11] . We can 1) update this policy based on Equation 1, and 2) employ Equation 4 for constraint learning. Intuitively, the objective of Equation 4 is to distinguish between trajectories generated by expert policies and imitation policies that may violate the constraints.Specifically, task 1) involves updating the policy using advanced CRL methods. Significant progress has been made in some works such as BCQ-Lagrangian (BCQ-Lag), COpiDICE [43] , VOCE [44] ,and CDT [38] . Meanwhile, task 2) focuses on learning the constraint function, as shown in Figure 2 . Our research primarily improves the latter process due to two main challenges facing ICRL in healthcare: Challenge 1) pertains to the limitations of the Markov property, and Challenge 2)involves the issue of inferring constraints only from offline datasets. To address these challenges, we propose the offline CT as our solution.CRL: Maximizing rewards under constraints.Constraints: Offline Constraint Transformer. ICRL methods relying on the Markov property overlook patients' historical information, focusing only on the current state.However, both current and historical states, along with vital sign changes are crucial for a human doctor's decision-making process [15] . To emulate the observational approach of humans, we draw inspiration from the Decision Transformer (DT) [18] to incorporate historical information into constraints for a more comprehensive observation and judgment. We propose a constraint modeling approach based on a causal attention mechanism, as shown in Figure 3 . The structure comprises a causal Transformer for sequential modeling and a non-Markovian layer for weighted constraints learning.Sequential Modeling for Constraints Inference. For a trajectory segment of length T , 2T input embeddings are generated, with each position containing state s and action a embeddings. Additionally, these embeddings undergo linear and normalization layers before being fed into the causal Transformer, which produces output embeddings {d t } T t=1 determined by preceding input embeddings from (s 1 , a 1 , ..., s T , a T ). Here, d t depends only on the previous t states and actions.Modeling Non-Markovian for Weighted Constraints Learning. Although d t represents the cost function c t derived from observations over long trajectories, it doesn't pinpoint which previous key actions or states led to its increase. In healthcare, identifying key actions or states is vital for analyzing risky behaviors and status, and enhancing model interpretability. To address this, we draw inspiration from the design of the preference attention layer in [19] and introduce an additional attention layer.This layer is employed to define the cost weight for non-Markovians. It takes the output embeddings from the causality transformer as input and generates the corresponding cost and importance weights.The output of the attention layer is computed by weighting the values through the normalized dot product between the query and other keys:EQUATIONHere, the key k t ∈ R m , query q t ∈ R m , and value c t ∈ R m are derived from the t-th input d t through linear transformations, where m denotes the embedding dimension. Furthermore, for each time step t, since d t depends only on the previous state-action pairs {(s i , a i )} t i=1 and serves as the input embedding for the attention layer, c t is also associated solely with the preceding t time steps.The representation of the cost function as a weighted sum is defined asC (τ ) = T t=1 w t • c t . Then,we can also determine the constraint function values for each preceding subsequence. Introducing the newly defined cost function, we redefine Equation 4for CT as:EQUATIONwhere ϕ is the parameter of CT, D e and D v represent the expert data and the violating data. This formulation implies that the constraint should be minimized on the expert policy and maximized on the violating policy. We construct an expert and a violating dataset to evaluate Equation 6in offline.The expert data can be acquired from existing medical datasets or hospitals. Regarding the violating dataset, we introduce a generative model to establish it, as detailed in Section 4.2. The model processes a trajectory, τ e ∈ D e , as a sequence of tokens encompassing the return-to-go, states, and actions, defined as ( R1 , s 1 , a 1 , ..., RT , s T , a T ). Notably, the return-to-go Rt at timestep t is the sum of future rewards, calculated as Rt = T t ′ =t r t ′ . At each timestep t, it employs the tokens from the preceding K timesteps as its input, where K represents the context length.Thus, the input tokens for it at timestep t are denoted ash t = { R-K:t , s -K:t , a -K:t-1 }, where R-K:t = { RK , ..., Rt }, s -K:t = {s K , ..., s t } and a -K:t-1 = {a K , ..., a t-1 }.Policy Model. The input tokens are encoded through a linear layer for each modality. Subsequently, the encoded tokens pass through a casual transformer to predict future action tokens. We use a stochastic policy [38] to achieve policy learning. Additionally, we utilize a Shannon entropy regularizerH [π ϑ (• | h)]to prevent policy overfitting and enhance robustness. The optimization objective is to minimize the negative log-likelihood loss while maximizing the entropy with weight λ:EQUATIONwhere the policyπ ϑ (• | h t ) = N (µ ϑ (h t ) , Σ ϑ (h t ))adopts the stochastic Gaussian policy representation and ϑ is the parameter.Generative World Model. To predict states and rewards, we usex t = {h t ∪ a t } as input encodedby linear layers. The encoded tokens pass through the casual transformer to predict hidden tokens.Then we utilize two linear layers to fit the rewards and states. The optimization objective for the two linear layers ℓ with the parameters φ and µ can be defined as:EQUATIONGenerating Violating Data. In RL, excessively high rewards, surpassing those provided by domain experts, may incentivize agents to violate the constraints in order to maximize the total reward [11] .Therefore, we set a high initial target reward R1 to obtain violation data. We feed R1 and initial state s D v = {τ (i) v } N i=1 .Note that certain other generative models, such as Variational Auto-Encoder (VAE) [45] , Generative Adversarial Networks (GAN) [46, 47] , and Denoising Diffusion Probabilistic Models (DDPM) [48, 49] , may be better at generating data. We introduce the model-based offline RL primarily because it has been shown to generate violating data with exploration [38] and possess the ability to process time-series features efficiently.To train offline CT, we gather the medical expert dataset D e from the environment. Then  
experiment: 
 In this section, we first provide a brief overview of the task, as well as data extraction and preprocessing. Subsequently, in Section 5.1, we demonstrate that CT can describe constraints in healthcare and capture critical patient states. We emphasize its applicability to various CRL methods and its ability to approach the optimal policy for reducing mortality rates in Section 5.2. Finally, Section 5.3 discusses the realization of the objective of safe medical policies. Safe policy learning: Update π using the cost function C ϕ (τ ) based on Equation (1) 8: end for Output: π and C(τ ) Tasks. We primarily use the sepsis task that is commonly used in previous works [9, 20, 42, 22] , and supplement some experiments on the mechanical ventilator task [23, 50] . The detailed definition of the two tasks mentioned above can be found in Appendix B.1 and B.2.Data Extraction and Pre-processing. Our medical dataset is derived from the Medical Information Mart for Intensive Care III (MIMIC-III) database [51] . For each patient, we gather relevant physiological parameters, including demographics, lab values, vital signs, and intake/output events. Data is grouped into 4-hour windows, with each window representing a time step. In cases of multiple data points within a step, we record either the average or the sum. We eliminate variables with significant missing values and use the k-nearest neighbors method to fill in the rest. Notably, the training dataset consists of data from surviving patients, while the validation set includes survivors and non-survivors. In this section, we primarily assess the efficacy of the cost function learned by offline CT in sepsis, focusing particularly on its capability to evaluate patient mortality rates and capture critical events. As SOFA and lactate levels become increasingly unsafe, the cost increases. Mean BP and HR at lower values within the safe range incur a lower cost, but as they move into unsafe ranges, the cost increases, penalizing previous state-action pairs. The cost can differentiate between relatively safe and unsafe regions.To assess the capability of the cost function to capture key events, we analyze the relationship between physiological indicators and cost values. We focus on four key indicators in sepsis treatment:Sequential Organ Failure Assessment (SOFA) score [52] , lactate levels [53] , Mean Arterial Pressure (MeanBP) [54] , and Heart Rate (HR) [55] . The SOFA score and lactate levels are critical indicators for assessing sepsis severity, with higher values indicating greater patient risk. MeanBP and HR are essential physiological metrics, typically ranging from 70 to 100 mmHg and 60 to 100 beats, respectively. Deviations from these ranges can signify patient risk. As depicted in Figure 6 , the cost values effectively distinguish between high-risk and safe conditions, reflecting changes in patient status. Additional details on other parameters' relationship with cost are in Appendix B.4.Offline CT Improve the Performance of CRL?Baselines. We adopt the DDPG method as the baseline in sepsis research [9] , and the Double Deep Q-Learning (DDQN) and Conservative Q-Learning (CQL) methods as baselines in ventilator research [23] . Since there are no other offline inverse reinforcement learning works available for reference, we have included two additional settings: no cost and custom cost. In the case of no cost, the cost is set to zero, while the design of custom constraints is outlined in Appendix A. These settings help evaluate whether CT can infer effective constraints.Metrics. To assess effectiveness, we use ω to indicate the probability that the policy is optimal and analyze the relationship between DIFF and mortality rate through a graph. Recently, Kondrup et al. [23] use the Fitted Q Evaluation (FQE) [56] to evaluate the policy in healthcare. However, the value estimates of FQE depend solely on the dataset D and the actions chosen by the policy π used to train FQE. This reliance can lead to inaccurate estimates when evaluating unseen state-action pairs. Therefore, we do not adopt this method as an evaluation metric. Blue: Safe policy is closer to the optimal policy. ↑: higher is better.Results. We combine our method CT with common CRL algorithms (e.g., VOCE, COpiDICE, BCQ-Lag, and CDT), and compare them with In addition, corresponding experiments are conducted on the mechanical ventilator, as shown in Figure 8 . Compared to previous methods DDQN and CQL, under the CDT+CT approach, a noticeable trend is observed where the proportion of mortality rates increases with increasing differences. When there is a significant difference in DIFF, the results may be unreliable, possibly due to the limited data distribution in the tail. Table 3 : The proportion of "too high" and "sudden change" occurrences in drug dosage recommended by RL methods.Physician DDPG CDT No cost Custom cost CT vaso > 0.75 2.27% 7.44% 0.13% vaso > 0.9 1.71% 7.40% 0.09% 0% ↓ (max = 0.00) 0% ↓ (max = 0.11) ∆ vaso > 0.75 2.45% 21.00% 0.64% ∆ vaso > 0.9 1.88% 20.62% 0.48% 0% ↓ (max ∆ = 0.00) 0% ↓ (max ∆ = 0.10)We have confirmed the existence of two unsafe strategy issues, namely "too high" and "sudden change" in the treatment of sepsis, particularly in vaso in Section 1. To validate whether the CRL+CT approach could address these concerns, we employ the same statistical methods to evaluate our methodology, shown in Table 3 . To elucidate the efficacy of CT, we compare it with CDT+No-cost and CDT+Custom-cost approaches. We find that only the custom cost and CT methods successfully mitigated the risks associated with "too high" and "sudden change" behaviors. However, the custom cost approach opts to avoid administering drugs to mitigate these risks. Without these drugs, the patient's condition may not be alleviated, potentially leading to patient mortality. The CDT+CT approach can give a more appropriate drug dosage.Ablation Study. To investigate the impact of each component on the model's performance, we conducted experiments by sequentially removing each component from the CDT+CT model. The results are presented in the lower half of Table 2 . Both CT and its non-Markovian layer (attention layer) are indispensable and crucial components; removing either one results in a decrease in performance. Additionally, we observed that even a pure generative model outperforms DDPG in terms of performance. This is primarily because it inherently operates as a sequence-based reinforcement learning model, possessing exploration and consideration for long-term history. Therefore, this further underscores the effectiveness of sequence-based approaches in healthcare applications. 
conclusion: 
 In this paper, we propose offline CT, a novel ICRL algorithm designed to address safety issues in healthcare. This method utilizes a causal attention mechanism to observe patients' historical information, similar to the approach taken by actual doctors and employs non-Markovian importance weights to effectively capture critical states. To achieve offline learning, we introduce a model-based offline RL for exploratory data augmentation to discover unsafe decisions and train CT. Experiments in sepsis and mechanical ventilation demonstrate that our method avoids risky behaviors while achieving strategies that closely approximate the lowest mortality rates.Limitations. There are also several limitations of offline CT: (1) Lack of rigorous theoretical analysis:We did not precisely define the types of constraint sets, thereby conducting rigorous theoretical analysis on constraint sets remains challenging; (2) Need for more computational resources: Due to the Transformer architecture, more computational resources are required; (3) Fewer evaluation metrics: There is a lack of more medical-specific evaluation metrics in the experimental evaluation section; (4) Unrealistic assumptions of expert demonstrations: we assume that expert demonstrations are optimal in both constraint satisfaction and reward maximization. However, in reality, this assumption may not always hold. Therefore, researching a more effective approach to address the aforementioned issues holds promise for the field of secure medical reinforcement learning.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: We provided a detailed description of the experimental setup and metrics.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.• The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: We tested our method with multiple random seeds and calculated the standard error.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: We explain the required computational resources and related information in the appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer:[Yes] Justification: Our work has a positive impact on safe healthcare, promoting the expansion of artificial intelligence technology into the medical field.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.• If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our work does not pose security risks because it is based on publicly available datasets and models.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.• Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The code, data, and models we referenced are all cited, and we followed the licenses and terms of use throughout the process.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.• If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes]Justification: We will provide detailed data extraction code and model code as part of the submission files.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.• The paper should discuss whether and how consent was obtained from people whose asset is used.• At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: This paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.• According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. 
 
 Appendix: We base our design on prior knowledge that intravenous (IV) intake exceeding 2000mL/4h or vasopressor (Vaso) dosage surpassing 1g/(kg•min) is generally considered unsafe in sepsis treatment [6] . To design a reasonable constraint function, we refer to the constraint function designed by Liu et al. in the Bullet safety gym environments [38] . We define the cost function as shown in Equation 9.Thus, during the treatment of sepsis, if the agent exceeds the maximum dosage thresholds of the two medications, it incurs a cost due to constraint violation.EQUATIONwhere, s and a represent the patient's state and action, respectively. a IV max = 2000 indicates that the maximum fluid intake through IV is 2000mL, and a V aso max = 1 signifies that the maximum Vaso dosage is 1µg/(kg • min).We applied our custom constraint function in the CDT [38] method, and the results are shown in Background-Policy 2• Use a constraint function to express general unsafe behaviors. There is over-restriction and no action is taken ! Our definition is similar to [40] . We extract data from adult patients meeting the criteria for sepsis-3 criteria [57] and collect their data within the first 72 hours of admission. • Intake and Output Events: Fluid Output -4 hourly period, Total Fluid Output, Mechanical Ventilation Action Space. Regarding the treatment of sepsis, there are two main types of medications: intravenous fluids and vasopressors. We select the total amount of intravenous fluids for each time unit and the maximum dose of vasopressors as the two dimensions of the action space, defined as (sum(IV), max (Vaso)). Each dimension is a continuous value greater than 0.Reward Function. We refer to the reward function used in [9] , as shown in the following equation:EQUATION)Where λ 0 and λ 1 are hyperparameters set to -0.25 and -0.2, respectively. This reward function is designed based on the SOFA score, as it is a key indicator of the health status for sepsis patients and widely used in clinical settings. The formula describes a penalty when the SOFA score increases and a reward when the SOFA score decreases. We set 6 as the cutoff value because the mortality rate sharply increases when the SOFA score exceeds 6 [58] .The RL problem definition for Mechanical Ventilation Treatment is referenced from [23] .State Space.• Demographics/Static: Elixhauser, SIRS, Gender, Re-admission, GCS, SOFA, Age Reward Function. The primary objective of setting respiratory parameters is to ensure the patient's survival. We adopt the same reward function design as the work [23] , defined as Equation 11. This reward function first considers the terminal reward: if the patient dies, the reward r is set to -1; otherwise, it is +1 in the terminal state. Additionally, to provide more frequent rewards, intermediate rewards are considered. Intermediate rewards mainly focus on the Apache II score, which evaluates various parameters to describe the patient's health status. This reward function utilizes the increase or decrease in this score to reward the agent.EQUATIONIn Equation 11 Generating violating data. In addition, we evaluate the violating actions generated by the model, as shown in Figure 12 . When compared with expert strategies and penalty distributions, we find that the actions generated by the model mostly fall within the legal range. However, it occasionally produces behaviors that are inappropriate for the current state, constituting violating data. This indicates that our generative model can produce legally violating data. The length of a trajectory. Regarding the selection of trajectory length, we consider the relationship between the average prediction error, the error of the last point in the trajectory, and the trajectory length. We use the model-based offline RL to generate trajectories and compare them with expert data using the Euclidean distance to measure their differences. We evaluate the average error and the error of the last point in the trajectory, as shown in Figure 11 . We observe that with an increase in trajectory length, the average prediction error at each time step decreases, while the state error stabilizes. Taking into account the observation length and prediction accuracy, we ultimately choose to generate trajectories with lengths ranging from 10 to 15.To validate that the CT method captures key states, we conduct statistical analysis on the relationship between state values and penalty values. We collect penalty values under different state values for all patients, and the complete information is shown in Figure 13 . We find that the CT method successfully captures unsafe states and imposes higher penalties accordingly. The safe range of state values is shown in Table 5 .To validate the role of the attention layer in capturing states in CT, we conducted tests, and the experimental results are presented in Figure 14 and 13. We found that the attention layer plays a crucial role in state capture. For instance, in the case of an increase in the SOFA score, without the attention layer, this increase cannot be captured, while with the attention layer, it clearly captures the change. Thus, this indicates that SOFA, as a key diagnostic indicator of sepsis, with the help of the attention layer, CT can accurately capture its changes. To train the CRL+CT model, we use a total of 3 NVIDIA GeForce RTX 3090 GPUs, each with 24GB of memory. Training a CRL+CT model typically takes 5-6 hours. We employ 5 random seeds for validation. We use the Adam optimization algorithm to optimize all our networks, updating the learning rate using a decay factor parameterization at each iteration. The main hyperparameters are summarized in Table 6 and 7. 1.0 1.5 2.0 2.5 3.0 3.5 Justification: Our code can be made publicly available after the paper is published.Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details.• While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).• The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://nips.cc/public/guides/CodeSubmissionPolicy) for more details.• The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.• The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.• At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).• Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.Justification: Although our work is related to healthcare, we train and test our models on offline data, adhering to ethical standards.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.• The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).In sepsis, "too high" indicates that the dosage of the vaso medication exceeds the threshold. And "sudden change" indicates that the change in vaso medication dosage between two time points exceeds the threshold. 


Decision: The paper proposes to build on prior work on inverse constrained reinforcement learning to enable safe decision-making addressing two main challenges: i) using a causal transformer to enable non-markovian state-space and subsequently policy function, ii) using a generative world model that enables data-augmentation using offline healthcare data. The overall constrained MDP solution is empirically evaluated on healthcare data to generate safe policies, data-augmentation using the world model, and modeling constraints. 

Reviewers consider the problem formulation well grounded in healthcare and provided meaningful empirical evaluation setup. However there are several presentation and clarity issues that are affecting the quality of the contributions. Overall I agree with some lack of clarity in setup, notation, as well as some claims made such as those pointed out by all reviewers, especially by cbZJ that state definitions could be redefined based on what is being captured. Further the choice of metric and reliance on mortality were also concerns raised for the specific motivation. Overall, the authors have addressed many of the raised questions, like considering ablations with respect to learned behavior policy, and agreed to address the typos and other framework and writing issues. However, considering the claimed benefit, fully addressing all issues, including the need for sensitivity analysis, and off-policy evaluation of the proposed method's output, clarifying the choice of evaluation metrics will require significant changes and warrants another round of peer review. 

Therefore, I encourage the authors to continue to improve their work, and recommend a rejection for NeurIPS

Reviews: review: paper_summary: paper_summary: The authors consider healthcare applications of RL algorithms in which implicit constraint modeling is critical for safe recommendations. 
This is modeled as an RL policy optimization with constraints. However, the constraints are often unknown and need to be inferred from expert data trajectories in the healthcare applications. The authors propose a neural network estimator to combine with the constrained MDP formulation. They also identify that the naive way to represent states leads to non-Markov structure. To address these issues, the paper proposes a simple modification (based on prior work on the "preference attention layer") to a causal transformer based policy which attempts to model a parametric constraint function. Evaluations are based on healthcare benchmarks.
strength_weakness: strength_weakness: - It seems like the system (which is the patient) is not feasible to model as evolving according to an Markov process on the observed state at each time, but instead a POMDP with a high dimensional latent state. 

- Clarity and presentation can be improved. In Equation (4), what is $\hat{\tau}$, this was never defined for being such a key component of the procedure. Similar issues in Eq (8) related to clarity.
questions: questions: - In explaining the main Equation (4), the blurb in lines 162-164 needs some attention. Specifically, what does "MDP obtained after augmenting M with cost function $C_\theta$, using the executing policy ..." mean exactly?

- In Eq (8) what does $s_t, r_{t-1} \in x_t \sim D_e$ represent? It looks like $x_t$ is a representation generated by the transformer, and $s_t, r_t$ are from the data. Is there a problem in the notation?

- Is the transformer representations being trained from the gradients of both the policy model as well as the world model? If so, how do they interact? If not, consider adding stop gradients at the relevant places in Eq (7) and/or (8) to make this explicit.
limitations: limitations: Yes.
paper_summary: paper_summary: This paper introduces the Constraint Transformer (CT) framework to enhance safe decision-making in healthcare. The proposed CT model uses transformers to incorporate historical patient data into constraint modelling and employs a generative world model to create exploratory data for offline RL training. The authors supported their points by presenting experimental results in scenarios like sepsis treatment, showing that CT effectively reduces unsafe behaviours and approximates lower mortality rates, outperforming existing methods in both safety and interoperability.
strength_weakness: strength_weakness: Despite its strengths and novelty, this paper suffers from several critical technical flaws, primarily concerning the soundness of evaluation rather than the method itself:

1. **Definition of Metric**: The metric $\omega$ is defined by comparing drug dosages related to **mortality rate**, which I believe is a flawed definition, even though it has been used in previous papers. Mortality rate can be influenced by numerous factors, making it unsuitable as a reward for RL, which considers a limited number of drugs. It is challenging to convince clinicians that mortality can indicate the 'treatment quality' of vasopressor or mechanical ventilation. This suggests that the reward is not solely a function of the previous action and state but also many unconsidered features (hidden variables) in the datasets, such as adrenaline, dopamine, historical medical conditions, phenotypes, etc. [1] pointed out that doctors usually set a MAP target (e.g., 65) and administer vasopressors until the patient reaches this safe pressure; [2] suggests using the NEWS2 score as the reward supported by clinical evidence. None of these directly use mortality. While it is understandable that this paper is not a clinical study, and hence, it is not the authors' responsibility to identify clinically appropriate reward designs, I recommend referring to [1] and [2] for a reward design that makes more clinical sense.

2. **Definition of Optimal Policy**: This paper follows [3]'s definition of optimal policy. From my understanding, the clinician's policy $\hat{\pi}$ is approximated by a neural network. [2] pointed out that in the sepsis dataset, the behaviour policy can result in critical flaws in a very small number of states. Although the number of incorrect predictions is limited, they can still bias the off-policy evaluation results severely. I suspect this paper may encounter a similar issue. The authors should provide experiments and visualizations on the learning quality of the behaviour policy to justify their approach.

3. **Model-Based Off-Policy Evaluation**: The data imbalance in both the sepsis and ventilation datasets is significant. It is questionable whether the learned model can generalize well. The most acceptable way to validate the method remains using simulated data where all policies can be tested online. One possible testbed is the DTR-Bench[4] medical simulated environment.

The paper has a few other minor technical flaws compared to the above three.

[1] Jeter, Russell, et al. "Does the" Artificial Intelligence Clinician" learn optimal treatment strategies for sepsis in intensive care?." arXiv preprint arXiv:1902.03271 (2019).

[2] Luo, Zhiyao, et al. "Position: Reinforcement Learning in Dynamic Treatment Regimes Needs Critical Reexamination." Forty-first International Conference on Machine Learning.

[3] Aniruddh Raghu, Matthieu Komorowski, Leo Anthony Celi, Peter Szolovits, and Marzyeh Ghassemi. Continuous state-space models for optimal sepsis treatment: a deep reinforcement learning approach. In Machine Learning for Healthcare Conference, pages 147–163. PMLR, 2017

[4] Luo, Zhiyao, et al. "DTR-Bench: An in silico Environment and Benchmark Platform for Reinforcement Learning Based Dynamic Treatment Regime." arXiv preprint arXiv:2405.18610 (2024).
questions: questions: Attention Mechanism: The paper highlights the importance of the causal attention mechanism in the Constraint Transformer. Since the two experimental datasets are both short-term time series datasets, is transformer really necessary? Is it possible that an RNN can be better than a transformer?
limitations: limitations: There is no negative societal impact or limitation that needs clarification. 
In addition to the weakness I mentioned, this paper:
1. lacks off-policy evaluation results.
2. may summarise more related work in this 'dynamic treatment regime' field. A few examples are listed below:

[1] Kondrup, F., Jiralerspong, T., Lau, E., de Lara, N., Shkrob, J., Tran, M. D., Precup, D., and Basu, S. Towards safe mechanical ventilation treatment using deep offline reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 15696–
15702, 2023.

[2] Liu, Y., Logan, B., Liu, N., Xu, Z., Tang, J., and Wang, Y. Deep reinforcement learning for dynamic treatment regimes on medical registry data. In 2017 IEEE international conference on healthcare informatics (ICHI), pp. 380–385. IEEE, 2017.

[3] Nambiar, M., Ghosh, S., Ong, P., Chan, Y. E., Bee, Y. M., and Krishnaswamy, P. Deep offline reinforcement learning for real-world treatment optimization applications. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 4673–4684, 2023.

[4] Peng, X., Ding, Y., Wihl, D., Gottesman, O., Komorowski,M., Li-wei, H. L., Ross, A., Faisal, A., and Doshi-Velez,F. Improving sepsis treatment strategies by combining deep and kernel-based reinforcement learning. In AMIA Annual Symposium Proceedings, volume 2018, pp. 887. American Medical Informatics Association, 2018.
paper_summary: paper_summary: The paper uses the Inverse Constrained Reinforcement Learning (ICRL) framework to infer constraints in healthcare problems from expert demonstrations. It proposes the Constraint Transformer (CT) to address the dependence of decisions on historical data, which is generally ignored in ICRL methods with Markovian assumptions. It borrows the causal transformer from the previous decision transformer to incorporate history into constraint modeling. Additionally, a model-based offline RL model is trained to generate violating data. The CT demonstrates improved safety by effectively modeling constraints based on both violating and expert data.
strength_weakness: strength_weakness: -	The proposed method depends heavily on the generated violating data, which defines the objective function in the constraint transformer. How sensitive is the estimated policy to the generative world model? Figure 12 shows that the action distributions in the expert dataset and the violating dataset are different. The VASO action seldom takes a large value in the violating dataset. Will this distribution difference cause any trouble in the learning of the constraint?
-	Could the authors provide some details on how the DIFF between the estimated policy and the physicians' policy is calculated through graphical analysis? It would also be helpful if the authors could explain how Figure 7 is plotted. Is it calculated based on the dosage differences at each timestamp? In addition, what are the implications of the three DIFF evaluation metrics in Table 2? Since both IV and VASO are part of the action space, is the ACTION DIFF alone sufficient to evaluate the estimated policy?
questions: questions: See above.
limitations: limitations: The authors have addressed the limitations of their work.
paper_summary: paper_summary: This paper studies inverse RL with learnable constraints in the offline setting, focusing on practical applications in healthcare tasks. The main approach appears to be combining the decision transformer architecture in the offline RL literature with inverse constrained RL with max entropy framework. Experiments were conducted on two healthcare tasks: sepsis and mechanical ventilation.
strength_weakness: strength_weakness: There is significant issues with the writing: the text is incoherent throughout and really hindered my understanding of the paper.
questions: questions: 1. Unclear what exactly is the "gap" the paper addresses. It mentions several issues including Markov assumption and history dependence, personalization & individual differences, offline setting, decision transformers, and it's unclear what the focus is. Otherwise, the paper seems to be applying decision transformer architecture to the constrained IRL problem. 

1. Table 1: "too high" associated with increased mortality making it an "unsafe behavior" - this seems like confounding, patients are sick and that's why they receive high dose. Calling it "unsafe behavior" without conditioning on the patient state is inappropriate. 

2. Claims about Markovianity: if Markov assumption does not hold, it seems to suggest the state is not capturing enough information about the "environment state". 
    - L49: "The Markov decision is not compatible with medical decisions." and "historical states of patients are crucial for medical decision" why can't one incorporate history into states? 
    - Fig 1: if within the same state different treatments should be used due to individual differences, that also suggests the state definition is not capturing individual differences (L57 "ignore individual differences") and you should redefine the states. 
3. Organization & Flow:
    - page 4 Sec 4 Methods talks about ICRL - is this your proposed method or previous work? If it's previous work it should be stated clearly and probably does not belong in Methods. 
    - L116 - when talking about Sec 3 Problem Formulation of constrained MDP, the text talks about "we extract data within 72 hours of patient admission, with each 4-hour interval" - this seems like experimental details rather than the mathematical setup of your method. 
4. Writing and notation: writing is incoherent, many notations are not defined. 
    - L122 cost c_t \sim C, shouldn't cost depend on (s,a)? 
    - Eqn (2) in the definition of "probability of approaching the optimal policy", what does "top N" mean? Top N according to what? What is N? If N is dataset size, how do you "collect 2N from offline dataset" (L144)? 
    - L145 "calculate the DIFF and sort it in ascending order" Is the DIFF just one number? How do you sort it? What are you sorting? 
    - Eqn (3) what is R(tau), L157 is it ZM or ZMC, what is β
    - L162 "where M ζθ denotes the MDP obtained after augmenting M with the cost function Cθ" Do you mean ζθ is the same as Cθ? What is π_Mζθ (L163)?
    - L202 "doesn't" contractions should be avoided in academic writing
    - L206 "This layer is employed to define the cost weight for non-Markovians" What is "non-Markovians"? 
    - L207 What is "causality transformer"? In "generate the importance weights", importance weights has a specific meaning in RL. The authors probably meant the cost weights. 
    - L219 "We construct an expert and a violating dataset to evaluate Equation 6 in offline." This sentence doesn't make sense grammatically. 
    - L248 "generating violating data" have you defined violating data? "may incentivize": why does the procedure incentivize generation of "violating data", do you show it theoretically or empirically? 
    - L283 why "Notably, the training dataset consists of data from surviving patients" Please provide more justification for this. Training on only surviving patients would limit the dataset and state-actions available to the learning agent. 
5. Questions about experiments: 
    - L297 if cost values are positively correlated with mortality rates which is the reward, what's the point of having a separate cost function instead of just using the reward? 
    - L300 "This indicates that the attention layer plays a crucial role in assessing constraints." could you elaborate why? 
    - L321 "FQE cannot evaluate unseen (s,a)" are you saying your approach can evaluate unseen (s,a)? 
    - L331 "combined DIFF": vaso and IV have very different scales, how do you combine? 
    - L357 "corresponding experiments are conducted on the mechanical ventilator". What? 
    - It's unclear what Fig 8 is showing. Table 3 not explained, unclear if lower or higher is better, what is max ∆?
limitations: limitations: Limitations are discussed.

Comments: title: Acknowledgement of rebuttal
First reply to the rebuttal
comment: Thank you again for reviewing our work! We will continue to refine the paper. If you have any further questions, we would be happy to discuss them with you.
According to the current update of the authors results, I believe the score can be increased to 5. I'm happy to discuss further with AC and other reviewers
Thank you for the thoughtful comments on our work! Please allow us to address your concerns and answer the questions.

## Weekness 1

## Response: 

1. Thank you for your suggestions. We will add naive baselines to our experiments. 

2. This is indeed a very good approach to observe the differences in actions. In our study, the action space ranges from 0 to 24. Our experiments revealed that the agent rarely takes actions where both drug doses are at the maximum level (i.e., 24), while there is no significant reduction in other actions. However, the agent tends to choose higher VASO drug doses compared to doctors. We are considering whether the discrete action space has influenced this result, leading the model to perceive that danger only arises when both drug doses are high.

3. Sorry, we have not yet completed this part of the experiment, but we will continue to supplement this aspect in our subsequent research.

## Weakness 3

## Response: 

Our previous statement may have been unclear. "Too high" refers to a drug dosage that is lethal for all patients. This was just an example of a dangerous action, and there could also be the type of dangerous action you mentioned, where the total dosage is too high. However, our model does not restrict the categories of "dangerous actions." In our paper, we tested "too high" and "sudden change." However, we have not tested whether the model can restrict other types of "dangerous actions." We plan to test this in other continuous action space environments within the DTR Bench in the future.

By "not overestimating," we mean that in the sepsis environment of the DTR Bench, the actions taken are "yes" or "no." This action space may make it difficult to determine whether a drug dosage is high or low. It is not an issue of overestimation based on value or actor-critic methods.

## Question

## Response: 

Thank you for your response. I understand your concerns. We apologize that we have not yet completed the LSTM ablation experiments. To improve our work, we will conduct additional experiments in the future to enhance our study.

## Other questions

## Response: 

1. Regarding the cost function defined by humans, we designed one based on expert opinions in the paper. Designing a cost function by humans can be challenging, particularly in determining hyperparameters. With the help of LLMs, we consider that there may be some latent state indicators that are currently unknown in the medical field. Therefore, this prior knowledge may be difficult to provide to large models.

2. The optimality of behavioral strategies is indeed hard to define. Since a zero-mortality action strategy does not exist—because some patients will die regardless of treatment—we assume that a strategy with a lower mortality rate is more optimal. For the baseline selection, we chose the expert strategy (i.e., the physician's strategy data with a patient mortality rate of 0) and the zero drug dosage strategy.

Thank you for your response. We will continue to supplement our work with the following experiments: other environment OPE tests, DTR Bench online testing, and LSTM ablation experiments, to further strengthen our study.
While I thank the authors for the responses, I am maintaining my assessment of the submission given how substantial the clarifications and revisions are. I hope the authors can incorporate these to strengthen the paper.
Thank you for your thoughtful and detailed response to my comments. I have no further questions at this time.
Thank you for the clarifications. I believe my original review scores are fair and stick to them.
Thanks to the authors for providing their views and justification.

Regarding weakness 1:
I agree with your explanation to reward design and I appreciate the added cost function for NEWS and MAP. However, I still have 2 concerns: 
1. You did not compare naive baselines (random, zero-drug, etc.) in your added OPE result; according to [1], including naive baselines are critical. You may also add naive baselines to other experiments. 
2. The OPE result is generally good. RMSE_VASO is significantly larger than offline RL algos while RMSE_IV is not. You might visualise the policy and check if the difference is caused by a reduced use of VASO, therefore preventing high dose.
3. No OPE result on ventilation dataset

## Regarding weakness 3:
Yes, I agree with the authors that 'dangerous' actions (according to your definition) are actual values that do not need a model for prediction. However, I think the authors have some confusion about the concept of 'dangerous actions' and 'dangerous states'. 

Technically, 'dangerous actions' don't mean a high dose, a high dose doesn't always lead to dangerous states. It does not seem right to define a high dose to be dangerous, as some high doses might be necessary. Even if the agent proposed high doses while the doctors did not, it does not mean the agent is acting less dangerously or less optimally, as we do not have counterfactuals without simulation environments. Combined with the sad fact that all your evaluations are offline on imbalanced datasets, it is tough to convince me that your approach is 'safer'. 

Let me give you a very simple example to show that low doses can be dangerous: the recommended dose of insulin for Type 1 diabetic patients is 0.5 units/day, roughly. Policy 1 gives 0.25 units at noon and night and 0 elsewhere; policy 2 gives 0.01 units every 1 min for 24 hours. Accumulatively, the policy 2 will kill the patient because the summed dosage is too high. However, we did not observe any single 'high' dosage in policy 2. This example shows that we cannot simply say high-dose action is dangerous. There are a lot of complicated PK/PD effects we might consider further beyond high doses. I recommended using online testing because the simulation environment will consider the PK/PD for you, such that a higher reward means a 'safer' performance. If you disagree with this please justify further. 

"DTR-Bench currently cannot perform online evaluations of medication dosages, as it only provides a discrete action space"
No, DTR-Bench has four environments, three of which have continuous action spaces and the sepsis one has a discrete action space. The sepsis environment's states are human-readable (because they are designed in that way). By checking the state index, you can tell how many vital signs are abnormal. The authors should read more literature on the dynamic treatment regime field. 

Also, I am confused by the authors' statement that 'there is no overestimation'. From my understanding, all value-based or actor-critic methods have overestimation issues. The authors may explain further.

In conclusion, the justification for weakness 3 is weak.

## Regarding the question:
The authors might answer my previous questions directly: "Since the two experimental datasets are both short-term time series datasets, is a transformer really necessary? Is it possible that an RNN can be better than a transformer?"
I asked this because many exciting architectures or models in other fields do not work in healthcare. There are numerous reasons, but one is that healthcare data is imbalanced and insufficient for large models to train. Some works in healthcare show that transformers do not improve performance in small-scale healthcare tasks; some even decrease due to over-parameterization and/or overfitting. I would invite the authors to eliminate my concern through ablation studies. For example, what if you replace the transformer with a shallow encoder-decoder LSTM?  

Apart from the weaknesses I mentioned before, I have a few minor questions that were not presented. The authors might answer if time allows:
1. Is it possible that a human-defined cost function can surpass some baselines or your method? For example, with the help of LLMs, a non-healthcare expert can know the normal range of all clinical variables and define a heuristic cost function very easily. Since this knowledge is prior, it should intuitively outperform most baselines and possibly your method. If so, is the motivation of your paper solid?
2. Do you have any assumptions for the optimality of the behavioural policy? Will that affect the choice of your baselines?

The current rebuttal are insufficient to allow me change the score. I look forward to further discussion if anything mentioned above is not agreeable or wrong.


References:

[1] Luo, Zhiyao, et al. "Position: Reinforcement Learning in Dynamic Treatment Regimes Needs Critical Reexamination." Forty-first International Conference on Machine Learning.