Title: Offline Inverse Constrained Reinforcement Learning for Safe-Critical Decision Making in Healthcare
Venue: NeurIPS 2024
Accepted: no
Keywords: Reinforcement Learning

Inverse Constrained Reinforcement Learning

Healthcare
Abstract: Reinforcement Learning (RL) applied in healthcare can lead to unsafe medical decisions and treatment, such as excessive dosages or abrupt changes, often due to agents overlooking common-sense constraints. Consequently, Constrained Reinforcement Learning (CRL) is a natural choice for safe decisions. However, specifying the exact cost function is inherently difficult in healthcare. Recent Inverse Constrained Reinforcement Learning (ICRL) is a promising approach that infers constraints from expert demonstrations. ICRL algorithms model Markovian decisions in an interactive environment. These settings do not align with the practical requirement of a decision-making system in healthcare, where decisions rely on historical treatment recorded in an offline dataset. To tackle these issues, we propose the Constraint Transformer (CT). Specifically, 1) utilize causal attention mechanism to incorporate historical decisions and observations into the constraint modeling and employ a non-Markovian layer for weighted constraints to capture critical states, 2) generative world model to perform exploratory data augmentation, thereby enabling offline RL methods to generate unsafe decision sequences. In multiple medical scenarios, empirical results demonstrate that CT can capture unsafe states and achieve strategies that approximate lower mortality rates, reducing the occurrence probability of unsafe behaviors.

1

Abstract

Reinforcement Learning (RL) applied in healthcare can lead to unsafe medical decisions and treatment, such as excessive dosages or abrupt changes, often due to agents overlooking common-sense constraints. Consequently, Constrained Reinforcement Learning (CRL) is a natural choice for safe decisions. However, specifying the exact cost function is inherently difficult in healthcare. Recent Inverse Constrained Reinforcement Learning (ICRL) is a promising approach that infers constraints from expert demonstrations. ICRL algorithms model Markovian decisions in an interactive environment. These settings do not align with the practical requirement of a decision-making system in healthcare, where decisions rely on historical treatment recorded in an offline dataset. To tackle these issues, we propose the Constraint Transformer (CT). Specifically, 1) utilize causal attention mechanism to incorporate historical decisions and observations into the constraint modeling and employ a non-Markovian layer for weighted constraints to capture critical states, 2) generative world model to perform exploratory data augmentation, thereby enabling offline RL methods to generate unsafe decision sequences. In multiple medical scenarios, empirical results demonstrate that CT can capture unsafe states and achieve strategies that approximate lower mortality rates, reducing the occurrence probability of unsafe behaviors.

1 Introduction

In recent years, the doctor-to-patient ratio imbalance has drawn attention, with the U.S. having only 223.1 physicians per 100,000 people [1]. AI-assisted therapy emerges as a promising solution, offering timely diagnosis, personalized care, and reducing dependence on experienced physicians. Therefore, the development of an effective AI healthcare assistant is crucial.

Reinforcement learning (RL) offers a promising approach to develop AI assistants by addressing sequential decisionmaking tasks. However, this method can still lead to unsafe behaviors, such as administering excessive drug dosages, inappropriate adjustments of medical parameters, or abrupt changes in medication dosages. These behaviors, such as 'too high' or 'sudden change' can significantly endanger patients, potentially resulting in acute hypotension, hypertension, arrhythmias, and organ damage, with fatal consequences [4, 5, 6]. For example, in sepsis treatment, patients receiving vasopressors (vaso) at dosages exceeding 1 µg/ ( kg · min ) have a mortality rate of 90% [7]. Moreover, the 'sudden change' in vaso can rapidly affect blood vessels, causing acute fluctuations in blood

Table 1: The proportion of unsafe behaviors occurrences in vaso suggested by physician and DDPG. The typical range for vaso is 0 . 1 ∼ 0 . 2 µg/ ( kg · min ) , with doses exceeding 0 . 5 considered high [2]. A cutoff value of 0 . 75 is identified as a critical threshold associated with increased mortality [3].

∆ vaso: The change in vaso between two-time points.

pressure and posing life-threatening risks to patients [8]. Our experiments demonstrate that the work

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

[9] applying the Deep Deterministic Policy Gradient (DDPG) algorithm in sepsis indeed exhibits 'too high' and 'sudden change' 1 unsafe behaviors in vaso recommendations, as shown in Table 1.

This paper aims to achieve safe healthcare policy learning to mitigate unsafe behaviors. The most common method for learning safe policies is Constrained Reinforcement Learning (CRL) [10, 11], with the key to its success lying in the constraints representation. However, in healthcare, we can only design the cost function based on prior knowledge, which limits its application due to a lack of personalization, universality, and reliance on prior knowledge. For more details about issues, please refer to Appendix A. Therefore, Inverse Constrained Reinforcement Learning (ICRL) [12] emerges as a promising approach, as it can infer the constraints adhered to by experts from their demonstrations. However, directly applying ICRL in healthcare presents several challenges:

1) The Markov decision is not compatible with medical decisions. ICRL algorithms model Markov decisions, where the next state depends only on the current state and not on the history [13, 14]. However, in healthcare, the historical states of patients are crucial for medical decisionmaking [15], as demonstrated in the experiments shown in Figure 1. Therefore, ICRL algorithms based on Markov assumption can not capture patient history, and ignore individual patient differences, thereby limiting effectiveness. 2) Interactive environment is not available for healthcare or medical decisions. ICRL algorithms [12, 16] follow an online learning paradigm, allowing agents to explore and learn from interactive environments. However, unrestricted exploration in healthcare often entails unsafe behaviors that could breach constraints and result

0.00.10.20.30.40.50.60.70.80.9

g/kg/min)

0

50

100

150

200

250

Count

Physician

Agent

in substantial losses. Therefore, it is necessary to infer constraints using only offline datasets.

In this paper, we introduce offline Constraint Transformer (CT), a novel ICRL framework that incorporates patients' historical information into constraint modeling and learns from offline data to infer constraints in healthcare. Specifically,

1) Inspired by the recent success of transformers in sequence modeling [17, 18, 19], we incorporate historical decisions and observations into constraint modeling using a causal attention mechanism. To capture key events in trajectories, we introduce a non-Markovian transformer to generate constraints and importance weights, and then define constraints using weighted sums. CT takes trajectories as input, allowing for the observation of patients' historical information and evaluation of key states.

2) To learn from an offline dataset, we introduce a model-based offline RL method that simultaneously learns a policy model and a generative world model via auto-regressive imitation of the actions and observations in medical decisions. The policy model employs a stochastic policy with entropy regularization to prevent it from overfitting and improve its robustness. Utilizing expert datasets, the generative world model uses an auto-regressive exploration generation paradigm to effectively discover a set of violating trajectories. Then, CT can infer constraints in healthcare through these unsafe trajectories and expert trajectories.

In the medical scenarios of sepsis and mechanical ventilation, we conduct experimental evaluations of offline CT. Experimental evaluations demonstrate that offline CT can capture patients' unsafe states and assign higher penalties, thereby providing more interpretable constraints compared to previous works [9, 20, 21]. Compared to unconstrained and custom constraints, CT achieves strategies that closely approximate lower mortality rates with a higher probability (improving by 8 . 85% compared to DDPG). To investigate the avoidance of unsafe behaviors with offline CT, we evaluate the probabilities of 'too high' and 'sudden changes' occurring in the sepsis. The experimental results show that CRL with CT can reduce the probability of unsafe behaviors to zero.

2 Related Works

Reinforcement Learning in Healthcare. RL has made great progress in the realm of healthcare, such 89 as sepsis treatment [9, 20, 21, 22], mechanical ventilation [23, 24, 25], sedation [26] and anesthesia 90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

[27, 28]. However, these works mentioned above have not addressed potential safety issues such as sudden changes or too high doses of medication. Therefore, the development of policies that are both safe and applicable across various healthcare domains is crucial.

Inverse Constrained Reinforcement Learning. Previous works inferred constraint functions by determining the feasibility of actions under current states. In discrete state-action spaces, Chou et al. [29] and Park et al. [30] learned constraint sets to differentiate constrained state-action pairs. Scobee &Sastry [31] proposed inferring constraint sets based on the principle of maximum entropy, while some studies [32, 33] extended this approach to stochastic environments using maximum causal entropy [34]. In continuous domains, Malik et al. [12], Gaurav et al. [16], and Qiao et al. [35] used neural networks to approximate constraints. Some works [11, 29] applied Bayesian Monte Carlo and variational inference to infer the posterior distribution of constraints in high-dimensional state spaces. Xu et al. [36] modeled uncertainty perception constraints for arbitrary and epistemic uncertainties. However, these methods can only be applied online and lack historical dependency.

Transformers for Reinforcement Learning. Transformer has produced exciting progress on RL sequential decision problems [17, 18, 37, 38]. These works no longer explicitly learn Q-functions or policy gradients, but focus on action sequence prediction models driven by target rewards. Chen et al. [18] and Janner et al. [37] perform auto-regressive modeling of trajectories to achieve policy learning in an offline environment. Furthermore, Zheng et al. [17] unify offline pretraining and online fine-tuning within the Transformer framework. Liu et al. [38] and Kim et al. [19] integrate the transformer architecture into constraint learning and preference learning. The transformer architecture, with its sequence modeling capability and independence from the Markov assumption, can capture temporal dependencies in medical decision-making. Thus, it is well-suited for trajectory learning and personalized learning in medical settings.

3 Problem Formulation

We model the medical environment with a Constrained Markov Decision Process (CMDP) M c [39], which can be defined by a tuple ( S , A , P , R , C , γ, κ, ρ 0 ) . Similar to studies [23, 40], we extract data within 72 hours of patient admission, with each 4 -hour interval constituting a window or time step. The state indicators of the patient at each time step are denoted as s ∈ S . The administered drug doses or instrument parameters of interest are considered as actions a ∈ A , while reward function R is used to describe the quality of the patient's condition and provided by experts based on prior work [9, 23]. At each time step t , an agent performs an action a t at a patient's state s t . This process generates the reward r t ∼ R ( s t , a t ) , the cost c t ∼ C and the next state s t +1 ∼ P ( · | s t , a t ) , where P defines the transition probabilities. γ denotes the discount factor. κ ∈ R + denotes the bound of cumulative costs. ρ 0 defines the initial state distribution. The goal of the CRL policy π is to maximize the reward return while limiting the cost in a threshold κ :

where T is the length of the trajectory τ . CRL commonly assumes that constraint signals are directly observable. However, in healthcare, such signals are not easily obtainable. Therefore, Our objective is to infer reasonable constraints for CRL to achieve safe policy learning in healthcare.

Safe-Critical Decision Making with Constraint Inference in Healthcare. Our general goal is for our policy to approximate the optimal policy, which refers to the strategy under which the patient's mortality rate is minimized (achieving a zero mortality rate is often difficult since there are patients who can not recover, regardless of all potential future treatment sequences [41]). Decision-making with constraints can formulate safer strategies by discovering and avoiding unsafe states, thereby approaching the optimal policy.

However, most offline RL algorithms rely on online evaluation, where the agent is evaluated in an interactive environment, whereas in medical scenarios, only offline evaluation can be utilized. In previous works [5, 9, 40, 42], they qualitatively analyzed by comparing the differences (DIFF) between the drug dosage recommended by our policy π and the dosage administered by clinical physicians ˆ π , and its relationship with mortality rates, through graphical analysis. In the graph depicting the relationship between the DIFF and mortality rate, at the point when DIFF is zero, the lower the mortality rate of patients, the better the performance of the policy [40]. To provide a more accurate quantitative evaluation, we introduce the concept of the probability of approaching the optimal policy, defined as ω :

N

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

We randomly collect 2 N patients (with an equal number of known survivors and non-survivors under doctor's policy ˆ π ) from the offline dataset. We then calculate the DIFF and sort it in ascending order. The optimality of the policy can be evaluated through the following two points: 1) The higher the survival probability (i.e., ω ) of the top N patients, the lower the mortality rate can be achieved by executing π ; 2) The smaller the DIFF among the surviving patients in the top N , the greater the probability that π is optimal.

4 Method

To infer constraints and achieve safe decision-making in healthcare, we introduce the Offline Constraint Transformer (Figure 2), a novel ICRL framework.

Inverse Constrained Reinforcement Learning. ICRL aims to recover the cost function C ∗ by leveraging a set of trajectories D e = { τ ( i ) e } N i sampled from an expert policy π e , where N denotes the number of the trajectories. ICRL is commonly based on the Maximum Entropy framework [31], and the likelihood function is articulated as [12]:

Here, Z M = ∫ exp( βr ( τ )) I M ( τ ) dτ is the normalizing term. The indicator I M C ( τ ( i ) ) signifies the extent to which the trajectory τ ( i ) satisfies the constraints. It can be approximated using a neural network ζ θ ( τ ( i ) ) parameterized with θ , defined as ζ θ ( τ ( i ) ) = ∏ T t =0 ζ θ ( s i t , a i t ) . Consequently, the cost function can be formulated as C θ = 1 -ζ θ . Substituting the neural network for the indicator, we can update θ through the gradient of the log-likelihood function:

where M ˆ ζ θ denotes the MDP obtained after augmenting M with the cost function C θ , using the executing policy π M ˆ ζ θ . And ˆ τ are sampled from the policy. In practice, ICRL can be conceptualized as a bi-level optimization task [11]. We can 1) update this policy based on Equation 1, and 2) employ Equation 4 for constraint learning. Intuitively, the objective of Equation 4 is to distinguish between trajectories generated by expert policies and imitation policies that may violate the constraints.

Specifically, task 1) involves updating the policy using advanced CRL methods. Significant progress has been made in some works such as BCQ-Lagrangian (BCQ-Lag), COpiDICE [43], VOCE [44], and CDT [38]. Meanwhile, task 2) focuses on learning the constraint function, as shown in Figure 2. Our research primarily improves the latter process due to two main challenges facing ICRL in healthcare: Challenge 1) pertains to the limitations of the Markov property, and Challenge 2) involves the issue of inferring constraints only from offline datasets. To address these challenges, we propose the offline CT as our solution.

CRL:

Maximizing rewards under constraints.

Constraints:

Offline Constraint Transformer.

2

Safe Agent

Casual Transformer

Attention Layer

Constraint Transformer (Section 4.1)

Input:

Violating Data

Expert Data

Expert Data

Medical Expert Data

Traditional CRL Method (e.g., BCQ-Lag,

COpiDICE, VOCE and CDT)

Safe Policy

States

→

Best action

Model-based Offline RL (Section 4.2)

Input:

Expert Data

Violating Data

Offline Transformer-based Constraints

Safe-Critical Decision Making with Constraints (Section 4.3)

Casual Transformer

1

Offline Constraint Transformer. To address the first challenge, we delve into the inherent issues of 174 applying the Markov property to healthcare and draw inspiration from the successes of Transformer 175 in decision-making, redefining the representation of the constraints. To realize the offline training, we 176 consider the essence of ICRL updates, proposing a model-based RL to generate unsafe behaviors 177 used to train CT. We outline three parts: establishing the constraint representation model (Section 178 4.1), creating an offline RL for violating data (Section 4.2), and learning safe policies (Section 4.3). 179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

Casual Transformer

Non-Markovian  Layer

Linear

MatMul

Scale

MatMul

4.1 Constraint Transformer

ICRL methods relying on the Markov property overlook patients' historical information, focusing only on the current state. However, both current and historical states, along with vital sign changes are crucial for a human doctor's decision-making process [15]. To emulate the observational approach of humans, we draw inspiration from the Decision Transformer (DT) [18] to incorporate historical information into constraints for a more comprehensive observation and judgment. We propose a constraint modeling approach based on a causal attention mechanism, as shown in Figure 3. The structure comprises a causal Transformer for

sequential modeling and a non-Markovian layer for weighted constraints learning.

Sequential Modeling for Constraints Inference. For a trajectory segment of length T , 2 T input embeddings are generated, with each position containing state s and action a embeddings. Additionally, these embeddings undergo linear and normalization layers before being fed into the causal Transformer, which produces output embeddings { d t } T t =1 determined by preceding input embeddings from ( s 1 , a 1 , ..., s T , a T ) . Here, d t depends only on the previous t states and actions.

Modeling Non-Markovian for Weighted Constraints Learning. Although d t represents the cost function c t derived from observations over long trajectories, it doesn't pinpoint which previous key actions or states led to its increase. In healthcare, identifying key actions or states is vital for analyzing risky behaviors and status, and enhancing model interpretability. To address this, we draw inspiration from the design of the preference attention layer in [19] and introduce an additional attention layer. This layer is employed to define the cost weight for non-Markovians. It takes the output embeddings from the causality transformer as input and generates the corresponding cost and importance weights. The output of the attention layer is computed by weighting the values through the normalized dot product between the query and other keys:

Here, the key k t ∈ R m , query q t ∈ R m , and value c t ∈ R m are derived from the t -th input d t through linear transformations, where m denotes the embedding dimension. Furthermore, for each time step t , since d t depends only on the previous state-action pairs { ( s i , a i ) } t i =1 and serves as the input embedding for the attention layer, c t is also associated solely with the preceding t time steps. The representation of the cost function as a weighted sum is defined as C ( τ ) = ∑ T t =1 w t · c t . Then, we can also determine the constraint function values for each preceding subsequence. Introducing the newly defined cost function, we redefine Equation 4 for CT as:

where ϕ is the parameter of CT, D e and D v represent the expert data and the violating data. This formulation implies that the constraint should be minimized on the expert policy and maximized on the violating policy. We construct an expert and a violating dataset to evaluate Equation 6 in offline. The expert data can be acquired from existing medical datasets or hospitals. Regarding the violating dataset, we introduce a generative model to establish it, as detailed in Section 4.2.

4.2 Model-based Offline RL 222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

To train CT offline, we introduce a modelbased offline RL method (Figure 4) to generate violating data that refers to unsafe behavioral data and can be represented as τ v = ( s 1 , a 1 , r 1 , s 2 , ... ) ∈ D v . The model simultaneously learns a policy model and a

Casual Transformer

generative world model via auto-regressive imitation of the actions and observations in healthcare. The model processes a trajectory, τ e ∈ D e , as a sequence of tokens encompassing the return-to-go, states, and actions, defined as ( ˆ R 1 , s 1 , a 1 , ..., ˆ R T , s T , a T ) . Notably, the return-to-go ˆ R t at timestep t is the sum of future rewards, calculated as ˆ R t = ∑ T t ′ = t r t ′ . At each timestep t , it employs the tokens from the preceding K timesteps as its input, where K represents the context length. Thus, the input tokens for it at timestep t are denoted as h t = { ˆ R -K : t , s -K : t , a -K : t -1 } , where ˆ R -K : t = { ˆ R K , ..., ˆ R t } , s -K : t = { s K , ..., s t } and a -K : t -1 = { a K , ..., a t -1 } .

Policy Model. The input tokens are encoded through a linear layer for each modality. Subsequently, the encoded tokens pass through a casual transformer to predict future action tokens. We use a stochastic policy [38] to achieve policy learning. Additionally, we utilize a Shannon entropy regularizer H [ π ϑ ( · | h )] to prevent policy overfitting and enhance robustness. The optimization objective is to minimize the negative log-likelihood loss while maximizing the entropy with weight λ :

where the policy π ϑ ( · | h t ) = N ( µ ϑ ( h t ) , Σ ϑ ( h t )) adopts the stochastic Gaussian policy representation and ϑ is the parameter.

Generative World Model. To predict states and rewards, we use x t = { h t ∪ a t } as input encoded by linear layers. The encoded tokens pass through the casual transformer to predict hidden tokens. Then we utilize two linear layers to fit the rewards and states. The optimization objective for the two linear layers ℓ with the parameters φ and µ can be defined as:

Generating Violating Data. In RL, excessively high rewards, surpassing those provided by domain experts, may incentivize agents to violate the constraints in order to maximize the total reward [11]. Therefore, we set a high initial target reward ˆ R 1 to obtain violation data. We feed ˆ R 1 and initial state s ( i ) 1 into the model-based offline RL to generate τ ( i ) v in an auto-regressive manner, as depicted in model-based offline RL of Figure 2, where ˜ a , ˜ r and ˜ s are predicted by the model. The target reward ˆ R decreases incrementally and can be represented as ˆ R t +1 = ˆ R t -˜ r t . Considering the average error in trajectory prediction, we generate trajectories with the length K = 10 , as detailed in Appendix B.3. Repeating N initial states, we can get violating data D v = { τ ( i ) v } N i =1 .

Note that certain other generative models, such as Variational Auto-Encoder (V AE) [45], Generative Adversarial Networks (GAN) [46, 47], and Denoising Diffusion Probabilistic Models (DDPM) [48, 49], may be better at generating data. We introduce the model-based offline RL primarily because it has been shown to generate violating data with exploration [38] and possess the ability to process time-series features efficiently.

4.3 Safe-Critical Decision Making with Constraints.

To train offline CT, we gather the medical expert dataset D e from the environment. Then, we employ gradient descent to train the model-based offline RL, guided by Equation 7 and Equation 8, continuing until the model converges. Using this RL model, we automatically generate violating data denoted as D v . Subsequently, CT is optimized based on Equation 6 to get the cost function C , leveraging samples from both D e and D v . To learn a safe policy, we train the policy π using C until it converges based on Equation 1. The detailed training procedure is presented in Algorithm 1.

5 Experiment

In this section, we first provide a brief overview of the task, as well as data extraction and preprocessing. Subsequently, in Section 5.1, we demonstrate that CT can describe constraints in healthcare and capture critical patient states. We emphasize its applicability to various CRL methods and its ability to approach the optimal policy for reducing mortality rates in Section 5.2. Finally, Section 5.3 discusses the realization of the objective of safe medical policies.

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

Algorithm 1 Safe Policy Learning with Offline CT

Input: Expert trajectories D e , context length K , target reward ˆ R 1 , samples N , episode length T

1: Train model-based offline RL M : Update ϑ , φ and µ using the Equation (7) and Equation (8)

2: for t = 1,...,T do

3: Sample initial states S 1 from D e

4: Generate the violating dataset: D v ←M . generate_data( S 1 , ˆ R 1 , K )

5: Sample set of trajectories { τ ( i ) e } N i =1 and { τ ( i ) v } N i =1 from D e and D v

6: Train offline CT: Use { τ ( i ) e } N i =1 and { τ ( i ) v } N i =1 to update ϕ based on Equation (6)

7: Safe policy learning: Update π using the cost function C ϕ ( τ ) based on Equation (1)

8: end for

Output:

π and C ( τ )

Tasks. We primarily use the sepsis task that is commonly used in previous works [9, 20, 42, 22], and supplement some experiments on the mechanical ventilator task [23, 50]. The detailed definition of the two tasks mentioned above can be found in Appendix B.1 and B.2.

Data Extraction and Pre-processing. Our medical dataset is derived from the Medical Information Mart for Intensive Care III (MIMIC-III) database [51]. For each patient, we gather relevant physiological parameters, including demographics, lab values, vital signs, and intake/output events. Data is grouped into 4-hour windows, with each window representing a time step. In cases of multiple data points within a step, we record either the average or the sum. We eliminate variables with significant missing values and use the k -nearest neighbors method to fill in the rest. Notably, the training dataset consists of data from surviving patients, while the validation set includes survivors and non-survivors.

Model-based Offline RL Evaluation. To ensure the rigor of the experiments, we evaluate the validity of the model-based offline RL, as detailed in Appendix B.3.

5.1 Can Offline CT Learn Effective Constraints?

In this section, we primarily assess the efficacy of the cost function learned by offline CT in sepsis, focusing particularly on its capability to evaluate patient mortality rates and capture critical events. First, we employ the cost function to compute cost values for the validation dataset. Subsequently, we statistically analyze the relationship between these cost values and mortality rates. As shown in Figure 5, there is an increase in patient mortality rates with rising cost values. It's noteworthy that such increases in mortality rates are often attributed to suboptimal medical decisions. Therefore, these

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Expected Cost

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Proportion Mortality

Mortality and Expected Cost

CT

CT (No attention layer)

tween cost and mortality. experimental findings affirm that the cost values effectively reflect the quality of medical decision-making. To observe the impact of the attention layer (non-Markovian layer), we conduct experiments by removing the attention layer from CT. The results reveal that the penalty values do not correlate proportionally with mortality rates. This indicates that the attention layer plays a crucial role in assessing constraints.

unsafe

unsafe

Safe

Safe

To assess the capability of the cost function to capture key events, we analyze the relationship 301 between physiological indicators and cost values. We focus on four key indicators in sepsis treatment: 302 Sequential Organ Failure Assessment (SOFA) score [52], lactate levels [53], Mean Arterial Pressure 303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

(MeanBP) [54], and Heart Rate (HR) [55]. The SOFA score and lactate levels are critical indicators for assessing sepsis severity, with higher values indicating greater patient risk. MeanBP and HR are essential physiological metrics, typically ranging from 70 to 100 mmHg and 60 to 100 beats, respectively. Deviations from these ranges can signify patient risk. As depicted in Figure 6, the cost values effectively distinguish between high-risk and safe conditions, reflecting changes in patient status. Additional details on other parameters' relationship with cost are in Appendix B.4.

5.2 Can Offline CT Improve the Performance of CRL?

Baselines. We adopt the DDPG method as the baseline in sepsis research [9], and the Double Deep Q-Learning (DDQN) and Conservative Q-Learning (CQL) methods as baselines in ventilator research [23]. Since there are no other offline inverse reinforcement learning works available for reference, we have included two additional settings: no cost and custom cost. In the case of no cost, the cost is set to zero, while the design of custom constraints is outlined in Appendix A. These settings help evaluate whether CT can infer effective constraints.

Metrics. To assess effectiveness, we use ω to indicate the probability that the policy is optimal and analyze the relationship between DIFF and mortality rate through a graph. Recently, Kondrup et al. [23] use the Fitted Q Evaluation (FQE) [56] to evaluate the policy in healthcare. However, the value estimates of FQE depend solely on the dataset D and the actions chosen by the policy π used to train FQE. This reliance can lead to inaccurate estimates when evaluating unseen state-action pairs. Therefore, we do not adopt this method as an evaluation metric.

Results. We combine our method CT with common CRL algorithms (e.g., VOCE, COpiDICE, BCQ-Lag, and CDT), and compare them with both no-cost and custom cost settings. Each CRL model is trained using no cost, custom cost, and CT separately, with other parameters set the same during training. For evaluation metrics, we use IV difference (IV DIFF), vaso difference (VASO DIFF), and combined [IV, VASO] difference (ACTION DIFF) as the metrics to be ranked. We measure the mean and variance of ω % in 10 sets of random seeds, and the results are shown in Table 2. From the results, we can conclude: (1) In different CRL methods, CT consistently makes the strategy closer to the one with lower mortality rates, with a probability 8 . 85% higher than DDPG. (2) We find that CDT+CT achieves better results on all three metrics. CDT is also a transformer-based method, which indicates that transformer-based architecture indeed exhibits more outstanding performance in healthcare.

Figure 7 illustrates the relationship between IV and VASO DIFF with mortality rates under the DDPG and CDT+CT methods in sepsis. In VASO DIFF, when the gap is zero, the mortality rate under CDT+CT is lower than that under DDPG, indicating that following the former strategy could lead to a lower mortality rate. Similarly, in IV DIFF, the same trend is observed. Notably, for the IV strategy, the lowest mortality rate for DDPG does not occur at the point where the difference is zero, indicating a significant estimation bias.

ω

%

COST

IV DIFF

↑

VASO DIFF

↑

↑

DDPG

-

50.95

±

1.34

51.45

±

0.75

51.15

±

1.15

No cost

47.45

±

0.52

46.35

±

1.82

51.00

±

0.86

Custom cost

46.45

±

0.46

52.00

±

0.98

49.40

±

1.04

VOCE

CT

53.33

±

0.94

59.04

±

1.13

56.15

±

1.08

No cost

48.30

±

0.91

60.10

±

0.6

51.25

±

0.70

Custom cost

53.05

±

1.35

55.20

±

0.24

53.90

±

1.04

CopiDICE

CT

51.95

±

0.41

60.85

±

1.08

54.60

±

0.60

No cost

47.50

±

1.32

51.05

±

0.61

49.35

±

1.08

Custom cost

51.54

±

0.16

56.23

±

1.43

53.69

±

1.62

BCQ-Lag

CT

52.45

±

1.01

55.34

±

1.20

54.39

±

0.86

No cost

56.50

±

0.81

62.45

±

1.20

58.90

±

1.34

Custom cost

54.70

±

1.12

59.85

±

1.51

57.80

±

1.00

CDT

CT

57.15

±

1.67

65.20

±

1.22

60.00

±

1.49

CDT

Without CT

56.50

±

0.81

62.45

±

1.20

58.90

±

1.34

CDT

No attention layer 55.25

±

1.46

64.00

±

1.54

57.90

±

0.78

Generative

Model

-

55.49

±

2.55

56.60

±

1.33

57.00

±

2.06

Blue:

Safe policy is closer to the optimal policy.

↑

:

higher is better.

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0.8

0.10

0.15

0.20

0.25

0.30

0.35

0.40

Mortality

800

400

0

400

800

CDT+CT

DDPG

VASO DIFF

IV DIFF

In addition, corresponding experiments are conducted on the mechanical ventilator, as shown in Figure 8. Compared to previous methods DDQN and CQL, under the CDT+CT approach, a noticeable trend is observed where the proportion of mortality rates increases with increasing differences. When

there is a significant difference in DIFF, the results may be unreliable, possibly due to the limited 360 data distribution in the tail.

6

5

4

3

2

1 0

1

2

3

4

5

6

PEEP (DDQN)

0.3

0.4

0.5

Mortality

6

5

4

3

2

1 0

1

2

3

4

5

6

PEEP (CQL)

0.30

0.35

0.40

0.45

6

5

4

3

2

1 0

1

2

3

4

5

6

PEEP (CDT)

0.4

0.6

0.8

1.0

6

5

4

3

2

1 0

1

2

3

4

5

6

PEEP (CDT+CT)

0.3

0.4

0.5

0.6

6

5

4

3

2

1 0

1

2

3

4

5

6

FiO2 (DDQN)

0.300

0.325

0.350

0.375

Mortality

6

5

4

3

2

1 0

1

2

3

4

5

6

FiO2 (CQL)

0.30

0.35

6

5

4

3

2

1 0

1

2

3

4

5

6

FiO2 (CDT)

0.30

0.32

0.34

6

5

4

3

2

1 0

1

2

3

4

5

6

FiO2 (CDT+CT)

0.30

0.35

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

5.3 Can CRL with Offline CT Learn Safe Policies?

We have confirmed the existence of two unsafe strategy issues, namely 'too high' and 'sudden change' in the treatment of sepsis, particularly in vaso in Section 1. To validate whether the CRL+CT approach could address these concerns, we employ the same statistical methods to

evaluate our methodology, shown in Table 3. To elucidate the efficacy of CT, we compare it with CDT+No-cost and CDT+Custom-cost approaches. We find that only the custom cost and CT methods successfully mitigated the risks associated with 'too high' and 'sudden change' behaviors. However, the custom cost approach opts to avoid administering drugs to mitigate these risks. Without these drugs, the patient's condition may not be alleviated, potentially leading to patient mortality. The CDT+CT approach can give a more appropriate drug dosage.

Ablation Study. To investigate the impact of each component on the model's performance, we conducted experiments by sequentially removing each component from the CDT+CT model. The results are presented in the lower half of Table 2. Both CT and its non-Markovian layer (attention layer) are indispensable and crucial components; removing either one results in a decrease in performance. Additionally, we observed that even a pure generative model outperforms DDPG in terms of performance. This is primarily because it inherently operates as a sequence-based reinforcement learning model, possessing exploration and consideration for long-term history. Therefore, this further underscores the effectiveness of sequence-based approaches in healthcare applications.

6 Conclusion

In this paper, we propose offline CT, a novel ICRL algorithm designed to address safety issues in healthcare. This method utilizes a causal attention mechanism to observe patients' historical information, similar to the approach taken by actual doctors and employs non-Markovian importance weights to effectively capture critical states. To achieve offline learning, we introduce a model-based offline RL for exploratory data augmentation to discover unsafe decisions and train CT. Experiments in sepsis and mechanical ventilation demonstrate that our method avoids risky behaviors while achieving strategies that closely approximate the lowest mortality rates.

Limitations. There are also several limitations of offline CT: (1) Lack of rigorous theoretical analysis: We did not precisely define the types of constraint sets, thereby conducting rigorous theoretical analysis on constraint sets remains challenging; (2) Need for more computational resources: Due to the Transformer architecture, more computational resources are required; (3) Fewer evaluation metrics: There is a lack of more medical-specific evaluation metrics in the experimental evaluation section; (4) Unrealistic assumptions of expert demonstrations: we assume that expert demonstrations are optimal in both constraint satisfaction and reward maximization. However, in reality, this assumption may not always hold. Therefore, researching a more effective approach to address the aforementioned issues holds promise for the field of secure medical reinforcement learning.

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

References

Stephen Petterson, Robert McNellis, Kathleen Klink, David Meyers, and Andrew Bazemore. The state of primary care in the united states: A chartbook of facts and statistics. Washington, DC: Robert Graham Center , 2018.

Estevão Bassi, Marcelo Park, Luciano Cesar Pontes Azevedo, et al. Therapeutic strategies for high-dose vasopressor-dependent shock. Critical care research and practice , 2013, 2013.

Thomas Auchet, Marie-Alix Regnier, Nicolas Girerd, and Bruno Levy. Outcome of patients with septic shock and high-dose vasopressor therapy. Annals of Intensive Care , 7:1-9, 2017.

Davide Tommaso Andreis and Mervyn Singer. Catecholamines for inflammatory shock: a jekyll-and-hyde conundrum. Intensive care medicine , 42:1387-1397, 2016.

Yan Jia, John Burden, Tom Lawton, and Ibrahim Habli. Safe reinforcement learning for sepsis treatment. In 2020 IEEE International conference on healthcare informatics (ICHI) , pages 1-7. IEEE, 2020.

Rui Shi, Olfa Hamzaoui, Nello De Vita, Xavier Monnet, and Jean-Louis Teboul. Vasopressors in septic shock: which, when, and how much? Annals of Translational Medicine , 8(12), 2020.

Claude Martin, Sophie Medam, Francois Antonini, Julie Alingrin, Malik Haddam, Emmanuelle Hammad, Bertrand Meyssignac, Coralie Vigne, Laurent Zieleskiewicz, and Marc Leone. Norepinephrine: not too much, too long. Shock , 44(4):305-309, 2015.

Kristin Lavigne Fadale, Denise Tucker, Jennifer Dungan, and Valerie Sabol. Improving nurses' vasopressor titration skills and self-efficacy via simulation-based learning. Clinical Simulation in Nursing , 10(6):e291-e299, 2014.

Yong Huang, Rui Cao, and Amir Rahmani. Reinforcement learning for sepsis treatment: A continuous action space solution. In Machine Learning for Healthcare Conference , pages 631-647. PMLR, 2022.

Yongshuai Liu, Avishai Halev, and Xin Liu. Policy learning with constraints in model-free reinforcement learning: A survey. In The 30th International Joint Conference on Artificial Intelligence (IJCAI) , 2021.

Guiliang Liu, Yudong Luo, Ashish Gaurav, Kasra Rezaee, and Pascal Poupart. Benchmarking constraint inference in inverse reinforcement learning. arXiv preprint arXiv:2206.09670 , 2022.

Shehryar Malik, Usman Anwar, Alireza Aghasi, and Ali Ahmed. Inverse constrained reinforcement learning. In International conference on machine learning , pages 7390-7399. PMLR, 2021.

Masaaki Kijima. Markov processes for stochastic modeling . Springer, 2013.

Zhiyue Zhang, Hongyuan Mei, and Yanxun Xu. Continuous-time decision transformer for healthcare applications. In International Conference on Artificial Intelligence and Statistics , pages 6245-6262. PMLR, 2023.

Catherine Plaisant, Brett Milash, Anne Rose, Seth Widoff, and Ben Shneiderman. Lifelines: 439 visualizing personal histories. In Proceedings of the SIGCHI conference on Human factors in 440 computing systems , pages 221-227, 1996. 441

442

443

Ashish Gaurav, Kasra Rezaee, Guiliang Liu, and Pascal Poupart. Learning soft constraints from constrained expert demonstrations. arXiv preprint arXiv:2206.01311 , 2022.

444

445

446

447

448

449

Qinqing Zheng, Amy Zhang, and Aditya Grover. Online decision transformer. In international conference on machine learning , pages 27042-27059. PMLR, 2022.

Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems , 34:15084-15097, 2021.

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

497

498

Changyeon Kim, Jongjin Park, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. Preference transformer: Modeling human preferences using transformers for rl. arXiv preprint arXiv:2303.00957 , 2023.

Aniruddh Raghu, Matthieu Komorowski, Imran Ahmed, Leo Celi, Peter Szolovits, and Marzyeh Ghassemi. Deep reinforcement learning for sepsis treatment. arXiv preprint arXiv:1711.09602 , 2017.

Xuefeng Peng, Yi Ding, David Wihl, Omer Gottesman, Matthieu Komorowski, H Lehman Liwei, Andrew Ross, Aldo Faisal, and Finale Doshi-Velez. Improving sepsis treatment strategies by combining deep and kernel-based reinforcement learning. In AMIA Annual Symposium Proceedings , volume 2018, page 887. American Medical Informatics Association, 2018.

Thanh Cong Do, Hyung Jeong Yang, Seok Bong Yoo, and In-Jae Oh. Combining reinforcement learning with supervised learning for sepsis treatment. In The 9th International Conference on Smart Media and Applications , pages 219-223, 2020.

Flemming Kondrup, Thomas Jiralerspong, Elaine Lau, Nathan de Lara, Jacob Shkrob, My Duc Tran, Doina Precup, and Sumana Basu. Towards safe mechanical ventilation treatment using deep offline reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 15696-15702, 2023.

Wei Gong, Linxiao Cao, Yifei Zhu, Fang Zuo, Xin He, and Haoquan Zhou. Federated inverse reinforcement learning for smart icus with differential privacy. IEEE Internet of Things Journal , 2023.

Chao Yu, Guoqi Ren, and Yinzhao Dong. Supervised-actor-critic reinforcement learning for intelligent mechanical ventilation and sedative dosing in intensive care units. BMC medical informatics and decision making , 20:1-8, 2020.

Niloufar Eghbali, Tuka Alhanai, and Mohammad M Ghassemi. Patient-specific sedation management via deep reinforcement learning. Frontiers in Digital Health , 3:608893, 2021.

Giulia Calvi, Eleonora Manzoni, and Mirco Rampazzo. Reinforcement q-learning for closedloop hypnosis depth control in anesthesia. In 2022 30th Mediterranean Conference on Control and Automation (MED) , pages 164-169. IEEE, 2022.

Gabriel Schamberg, Marcus Badgeley, Benyamin Meschede-Krasa, Ohyoon Kwon, and Emery N Brown. Continuous action deep reinforcement learning for propofol dosing during general anesthesia. Artificial Intelligence in Medicine , 123:102227, 2022.

Glen Chou, Dmitry Berenson, and Necmiye Ozay. Learning constraints from demonstrations. In Algorithmic Foundations of Robotics XIII: Proceedings of the 13th Workshop on the Algorithmic Foundations of Robotics 13 , pages 228-245. Springer, 2020.

Daehyung Park, Michael Noseworthy, Rohan Paul, Subhro Roy, and Nicholas Roy. Inferring task goals and constraints using bayesian nonparametric inverse reinforcement learning. In Conference on robot learning , pages 1005-1014. PMLR, 2020.

Dexter RR Scobee and S Shankar Sastry. Maximum likelihood constraint inference for inverse reinforcement learning. arXiv preprint arXiv:1909.05477 , 2019.

David L McPherson, Kaylene C Stocking, and S Shankar Sastry. Maximum likelihood constraint inference from stochastic demonstrations. In 2021 IEEE Conference on Control Technology and Applications (CCTA) , pages 1208-1213. IEEE, 2021.

Mattijs Baert, Pietro Mazzaglia, Sam Leroux, and Pieter Simoens. Maximum causal entropy inverse constrained reinforcement learning. arXiv preprint arXiv:2305.02857 , 2023.

Brian D Ziebart, J Andrew Bagnell, and Anind K Dey. Modeling interaction via the principle of maximum causal entropy. 2010.

Guanren Qiao, Guiliang Liu, Pascal Poupart, and Zhiqiang Xu. Multi-modal inverse constrained reinforcement learning from a mixture of demonstrations. Advances in Neural Information Processing Systems , 36, 2024.

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

532

533

534

535

536

537

538

539

540

541

542

543

544

545

546

547

Sheng Xu and Guiliang Liu. Uncertainty-aware constraint inference in inverse constrained reinforcement learning. In The Twelfth International Conference on Learning Representations , 2023.

Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems , 34:12731286, 2021.

Zuxin Liu, Zijian Guo, Yihang Yao, Zhepeng Cen, Wenhao Yu, Tingnan Zhang, and Ding Zhao. Constrained decision transformer for offline safe reinforcement learning. arXiv preprint arXiv:2302.07351 , 2023.

Eitan Altman. Constrained markov decision processes with total cost criteria: Lagrangian approach and dual linear program. Mathematical methods of operations research , 48:387-417, 1998.

Aniruddh Raghu, Matthieu Komorowski, Leo Anthony Celi, Peter Szolovits, and Marzyeh Ghassemi. Continuous state-space models for optimal sepsis treatment: a deep reinforcement learning approach. In Machine Learning for Healthcare Conference , pages 147-163. PMLR, 2017.

Mehdi Fatemi, Taylor W Killian, Jayakumar Subramanian, and Marzyeh Ghassemi. Medical dead-ends and learning to identify high-risk states and treatments. Advances in Neural Information Processing Systems , 34:4856-4870, 2021.

Matthieu Komorowski, Leo A Celi, Omar Badawi, Anthony C Gordon, and A Aldo Faisal. The artificial intelligence clinician learns optimal treatment strategies for sepsis in intensive care. Nature medicine , 24(11):1716-1720, 2018.

Jongmin Lee, Cosmin Paduraru, Daniel J Mankowitz, Nicolas Heess, Doina Precup, Kee-Eung Kim, and Arthur Guez. Coptidice: Offline constrained reinforcement learning via stationary distribution correction estimation. arXiv preprint arXiv:2204.08957 , 2022.

Jiayi Guan, Guang Chen, Jiaming Ji, Long Yang, Zhijun Li, et al. Voce: Variational optimization with conservative estimation for offline safe reinforcement learning. Advances in Neural Information Processing Systems , 36, 2024.

Yongju Kim, Hyung Keun Park, Jaimyun Jung, Peyman Asghari-Rad, Seungchul Lee, Jin You Kim, Hwan Gyo Jung, and Hyoung Seop Kim. Exploration of optimal microstructure and mechanical properties in continuous microstructure space using a variational autoencoder. Materials & Design , 202:109544, 2021.

Tim Hsu, William K Epting, Hokon Kim, Harry W Abernathy, Gregory A Hackett, Anthony D Rollett, Paul A Salvador, and Elizabeth A Holm. Microstructure generation via generative adversarial network for heterogeneous, topologically complex 3d materials. Jom , 73:90-102, 2021.

Akshay Iyer, Biswadip Dey, Arindam Dasgupta, Wei Chen, and Amit Chakraborty. A conditional generative model for predicting material microstructures from processing methods. arXiv preprint arXiv:1910.02133 , 2019.

Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, and Mubarak Shah. Diffusion models in vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023.

Christian Düreth, Paul Seibert, Dennis Rücker, Stephanie Handford, Markus Kästner, and Maik Gude. Conditional diffusion-based microstructure reconstruction. Materials Today Communications , 35:105608, 2023.

Arne Peine, Ahmed Hallawa, Johannes Bickenbach, Guido Dartmann, Lejla Begic Fazlic, Anke Schmeink, Gerd Ascheid, Christoph Thiemermann, Andreas Schuppert, Ryan Kindle, et al. Development and validation of a reinforcement learning algorithm to dynamically optimize mechanical ventilation in critical care. NPJ digital medicine , 4(1):32, 2021.

548

549

550

551

552

553

554

555

556

557

558

559

560

561

562

563

564

565

566

567

568

569

570

571

Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, a freely accessible critical care database. Scientific data , 3(1):1-9, 2016.

Yonglin Li, Chunjiang Yan, Ziyan Gan, Xiaotu Xi, Zhanpeng Tan, Jun Li, and Guowei Li. Prognostic values of sofa score, qsofa score, and lods score for patients with sepsis. Annals of palliative medicine , 9(3):1037044-1031044, 2020.

Seung Mok Ryoo, JungBok Lee, Yoon-Seon Lee, Jae Ho Lee, Kyoung Soo Lim, Jin Won Huh, Sang-Bum Hong, Chae-Man Lim, Younsuck Koh, and Won Young Kim. Lactate level versus lactate clearance for predicting mortality in patients with septic shock defined by sepsis-3. Critical care medicine , 46(6):e489-e495, 2018.

Nishant Raj Pandey, Yu-yao Bian, and Song-tao Shou. Significance of blood pressure variability in patients with sepsis. World journal of emergency medicine , 5(1):42, 2014.

Marta Carrara, Bernardo Bollen Pinto, Giuseppe Baselli, Karim Bendjelid, and Manuela Ferrario. Baroreflex sensitivity and blood pressure variability can help in understanding the different response to therapy during acute phase of septic shock. Shock , 50(1):78-86, 2018.

Hoang Le, Cameron Voloshin, and Yisong Yue. Batch policy learning under constraints. In International Conference on Machine Learning , pages 3703-3712. PMLR, 2019.

Mervyn Singer, Clifford S Deutschman, Christopher Warren Seymour, Manu Shankar-Hari, Djillali Annane, Michael Bauer, Rinaldo Bellomo, Gordon R Bernard, Jean-Daniel Chiche, Craig M Coopersmith, et al. The third international consensus definitions for sepsis and septic shock (sepsis-3). Jama , 315(8):801-810, 2016.

Flavio Lopes Ferreira, Daliana Peres Bota, Annette Bross, Christian Mélot, and Jean-Louis Vincent. Serial evaluation of the sofa score to predict outcome in critically ill patients. Jama , 286(14):1754-1758, 2001.

572

573

574

575

576

577

578

579

580

581

582

583

584

585

A Design and Analysis of the Custom Constraint Function

We base our design on prior knowledge that intravenous (IV) intake exceeding 2000 mL/ 4 h or vasopressor (Vaso) dosage surpassing 1 g/ ( kg · min ) is generally considered unsafe in sepsis treatment [6]. To design a reasonable constraint function, we refer to the constraint function designed by Liu et al. in the Bullet safety gym environments[38]. We define the cost function as shown in Equation 9. Thus, during the treatment of sepsis, if the agent exceeds the maximum dosage thresholds of the two medications, it incurs a cost due to constraint violation.

where, s and a represent the patient's state and action, respectively. a IV max = 2000 indicates that the maximum fluid intake through IV is 2000 mL , and a V aso max = 1 signifies that the maximum Vaso dosage is 1 µg/ ( kg · min ) .

We applied our custom constraint function in the CDT [38] method, and the results are shown in Figure 9. Compared to the Vaso dosage recommended by doctors, our strategy exhibits excessive suppression of the Vaso. The maximum dosage of Vaso is 0 . 0011 µg/ ( kg · min ) , which is minimal and insufficient to provide the patient with effective therapeutic effects.

Therefore, Equation 9 is not suitable. The primary issues may include uniform constraint strength 586 for excessive drug dosages, for instance, the cost for IV exceeding 2000 mL and IV exceeding 587 3000 mL is the same at 1 ; lack of generalization, where the constraint cost does not vary with the 588 patient's tolerance. If a patient has an intolerance to V ASO, the maximum value for V ASO maybe 0 , 589 which cannot be captured by the self-imposed constraint function. Moreover, it lacks generalization, 590 requiring redesign of the constraint function when addressing other unsafe medical issues; and it's 591 essential to ensure the correctness of the underlying medical knowledge premises. Background-Policy 2 · Use a constraint function to express general unsafe behaviors .

Our Policy IV

Our Policy Vaso

400

5000

350

4000

300

Vasomax

=0.0011

250

3000

200

150

2000

100

1000

50

0

0

250

500

7501000 1250 1500 1750 2000

0

0.0

0.2

0.4

90

0.8

1.0

Physician's Policy IV

Physician's Policy Vaso

1600

1400

4000

1200

1000

3000

800-

2000

600

400

1000

200

0

0

250

500

7501000 1250 1500 1750 2000

0-

0°0

0.2

0.4

0.6

0.8

1.0

There is over-restriction and no action is taken !

B Experiment Supplement

B.1 Sepsis Problem Define

Our definition is similar to [40]. We extract data from adult patients meeting the criteria for sepsis-3 criteria [57] and collect their data within the first 72 hours of admission.

State Space. We use a 4-hour window and select 48 patient indicators as the state for a one-time unit of the patient. The state indicators include Demographics/Static, Lab Values, Vital Signs, and Intake and Output Events, detailed as follows [40]:

Demographics/Static: Shock Index, Elixhauser, SIRS, Gender, Re-admission, GCS - Glasgow Coma Scale, SOFA - Sequential Organ Failure Assessment, Age

592

593

594

595

596

597

598

599

600

601

602

603

604

605

606

607

608

609

610

611

612

613

614

615

616

617

618

619

620

621

622

623

624

625

626

627

628

629

630

631

632

633

634

635

636

637

638

639

640

641

642

Lab Values Albumin: Arterial pH, Calcium, Glucose, Hemoglobin, Magnesium, PTT Partial Thromboplastin Time, Potassium, SGPT - Serum Glutamic-Pyruvic Transaminase, Arterial Blood Gas, BUN Blood Urea Nitrogen, Chloride, Bicarbonate, INR - International Normalized Ratio, Sodium, Arterial Lactate, CO2, Creatinine, Ionised Calcium, PT - Prothrombin Time, Platelets Count, SGOT Serum Glutamic-Oxaloacetic Transaminase, Total bilirubin, White Blood Cell Count

Vital Signs: Diastolic Blood Pressure, Systolic Blood Pressure, Mean Blood Pressure, PaCO2, PaO2, FiO2, PaO/FiO2 ratio, Respiratory Rate, Temperature (Celsius), Weight (kg), Heart Rate, SpO2

Intake and Output Events: Fluid Output 4 hourly period, Total Fluid Output, Mechanical Ventilation

Action Space. Regarding the treatment of sepsis, there are two main types of medications: intravenous fluids and vasopressors. We select the total amount of intravenous fluids for each time unit and the maximum dose of vasopressors as the two dimensions of the action space, defined as (sum(IV) , max (Vaso)). Each dimension is a continuous value greater than 0 .

Reward Function. We refer to the reward function used in [9], as shown in the following equation:

Where λ 0 and λ 1 are hyperparameters set to -0 . 25 and -0 . 2 , respectively. This reward function is designed based on the SOFA score, as it is a key indicator of the health status for sepsis patients and widely used in clinical settings. The formula describes a penalty when the SOFA score increases and a reward when the SOFA score decreases. We set 6 as the cutoff value because the mortality rate sharply increases when the SOFA score exceeds 6 [58].

B.2 Mechanical Ventilation Treatment Problem Define

The RL problem definition for Mechanical Ventilation Treatment is referenced from [23].

State Space.

Demographics/Static: Elixhauser, SIRS, Gender, Re-admission, GCS, SOFA, Age

Lab Values Albumin: Arterial pH, Glucose, Hemoglobin, Magnesium, PTT, BUN Blood Urea Nitrogen, Chloride, Bicarbonate, INR, Sodium, Arterial Lactate, CO2, Creatinine, Ionised Calcium, PT, Platelets Count, White Blood Cell Count, Hb

Vital Signs: Diastolic Blood Pressure, Systolic Blood Pressure, Mean Blood Pressure, Temperature, Weight (kg), Heart Rate, SpO2

Intake and Output Events: Urine output, vasopressors, intravenous fluids, cumulative fluid balance

Action Space. The action space mainly consists of Positive End Expiratory Pressure (PEEP) and Fraction of Inspired Oxygen (FiO2), which are crucial parameters in ventilator settings. Here, we consider a discrete space configuration, with each parameter divided into 7 intervals. Therefore, our action space is 7 × 7 , depicted as 4.

Reward Function. The primary objective of setting respiratory parameters is to ensure the patient's survival. We adopt the same reward function design as the work [23], defined as Equation 11. This reward function first considers the terminal reward: if the patient dies, the reward r is set to -1 ; otherwise, it is +1 in the terminal state. Additionally, to provide more frequent rewards, intermediate rewards are considered. Intermediate rewards mainly focus on the Apache II score, which evaluates

various parameters to describe the patient's health status. This reward function utilizes the increase or 643 decrease in this score to reward the agent. 644

In Equation 11, T represents the length of the patient's trajectory, m indicates whether the patient ultimately dies, A denotes the Apache II score, and max A and min A respectively denote the maximum and minimum values.

B.3 The Evaluation of Model-based Offline RL

Generating data within a reasonable range. To validate model-based offline RL, we first check whether the values it produces fall within the legal range. The results are depicted in Figure 10. After analyzing the generated data, we find that the majority of state values have a probability of over 99% of being within the legal range. A few values related to gender and re-admission range between 60% and 70% . This could be due to these two indicators having limited correlation with other metrics, making them more challenging for the model to assess.

>= 95%

80%-95%

60%-70%

< 60%

Ranges

0

10

20

30

40

50

Number of Features

44

GCS

mechvent

gender,re_admission

0

Proportion of Features in Different Ranges

0

2

4

6

8

10

12

14

16

18

Trajectory Length

0.2

0.4

0.6

0.8

1.0

Average Prediction Error

action trajectory

state trajectory

action last point

state last point

Generating violating data. In addition, we evaluate the violating actions generated by the model, as shown in Figure 12. When compared with expert strategies and penalty distributions, we find that the actions generated by the model mostly fall within the legal range. However, it occasionally produces behaviors that are inappropriate for the current state, constituting violating data. This indicates that our generative model can produce legally violating data.

Expert Dataset

Violating Dataset

0.5

0.5

1.0

0.4

0.4

0.8

Action Dimension vaso

Action Dimension vaso

0.3

0.3

0.6

Cost (c)

0.2

0.2

0.4

0.1

0.1

0.2

0.0

0.0

0.0

0

100

200

300

400

500

600

700

800

0

100

200

300

400

500

600

700

800

Action Dimension iv

Action Dimension iv

645

646

647

648

649

650

651

652

653

654

655

656

657

658

659

The length of a trajectory. Regarding the selection of trajectory length, we consider the relationship 660 between the average prediction error, the error of the last point in the trajectory, and the trajectory 661

length. We use the model-based offline RL to generate trajectories and compare them with expert 662 data using the Euclidean distance to measure their differences. We evaluate the average error and 663 the error of the last point in the trajectory, as shown in Figure 11. We observe that with an increase 664 in trajectory length, the average prediction error at each time step decreases, while the state error 665 stabilizes. Taking into account the observation length and prediction accuracy, we ultimately choose 666 to generate trajectories with lengths ranging from 10 to 15 . 667

668

669

670

671

672

673

674

675

676

677

678

679

680

B.4 The Evaluation of Cost function in Sepsis

To validate that the CT method captures key states, we conduct statistical analysis on the relationship between state values and penalty values. We collect penalty values under different state values for all patients, and the complete information is shown in Figure 13. We find that the CT method successfully captures unsafe states and imposes higher penalties accordingly. The safe range of state values is shown in Table 5.

To validate the role of the attention layer in capturing states in CT, we conducted tests, and the experimental results are presented in Figure 14 and 13. We found that the attention layer plays a crucial role in state capture. For instance, in the case of an increase in the SOFA score, without the attention layer, this increase cannot be captured, while with the attention layer, it clearly captures the change. Thus, this indicates that SOFA, as a key diagnostic indicator of sepsis, with the help of the attention layer, CT can accurately capture its changes.

↑ indicates higher values are more normal, while ↓ indicates lower values are more normal.

The maximum value for GCS is 15 . The minimum value for SIRS, SOFA, and Shock_Index is 0 .

B.5 Experimental Settings

To train the CRL+CT model, we use a total of 3 NVIDIA GeForce RTX 3090 GPUs, each with 681 24GB of memory. Training a CRL+CT model typically takes 5-6 hours. We employ 5 random seeds 682 for validation. We use the Adam optimization algorithm to optimize all our networks, updating the 683 learning rate using a decay factor parameterization at each iteration. The main hyperparameters are 684 summarized in Table 6 and 7. 685

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

Albumin values

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

Cost

5

4

3

2

1

0

1

2

3

4

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

Cost

0

1

2

3

4

5

6

7

8

9

Arterial_lactate values

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

Cost

7.0

7.1

7.2

7.3

7.4

7.5

Arterial_pH values

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

Cost

0

5

10

15

20

25

BUN values

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

Cost

0

5

10

15

20

25

30

35

40

45

50

55

CO2_mEqL values

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

Cost

3

4

5

6

7

8

9

10

11

12

13

14

Calcium values

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

Cost

80

85

90

95

100

105

110

115

Chloride values

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

Cost

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

Creatinine values

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

Cost

50

60

70

80

90

100

110

120

DiaBP values

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

Cost

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

FiO2_1 values

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

Cost

2

3

4

5

6

7

8

9

10 11 12 13 14

GCS values

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

Cost

5

10

15

20

25

30

35

40

45

HCO3 values

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

Cost

0

40 80 120160200240280320360400440480

Glucose values

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

Cost

50

60

70

80

90

100 110 120 130

HR values

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

Cost

0

2

4

6

8

10

12

14

16

18

Hb values

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

Cost

0.5 0.7 0.9 1.1 1.3 1.5 1.7 1.9 2.1 2.3 2.5 2.7 2.9

INR values

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

Cost

50

60

70

80

90

100 110 120 130 140

MeanBP values

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

Cost

6

16

26

36

46

56

66

76

86

96 106116

PT values

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

Cost

5

15 25 35 45 55 65 75 85 95105 115 125 135 145

PTT values

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

Cost

25

225 425 625 8251025 1225 1425 1625 1825

PaO2_FiO2 values

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

Cost

5

105 205 305 405 505 605 705 805 905

Platelets_count values

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

Cost

2

3

4

5

6

7

Potassium values

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

Cost

2

7

12

17

22

27

32

37

42

47

RR values

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

Cost

0

20

40

60

80 100 120 140 160 180

SGOT values

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

Cost

0

20

40

60

80 100 120 140 160 180

SGPT values

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

Cost

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

SIRS values

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

Cost

0

1

2

3

4

5

6

7

8

9 1011121314

SOFA values

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

Cost

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Shock_Index values

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

Cost

100

110

120

130

140

150

Sodium values

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

Cost

85

87

89

91

93

95

97

99

101

SpO2 values

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

Cost

50 60 70 80 90100 110 120 130 140 150 160 170 180 190

SysBP values

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

Cost

35.035.536.036.537.037.538.038.539.039.5

Temp_C values

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

Cost

0

10

20

30

40

50

60

70

80

90

WBC_count values

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

Cost

20

30

40

50

60

70

paCO2 values

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

Cost

50 60 70 80 90100 110 120 130 140 150 160 170

paO2 values

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

Cost

0

1

2

3

4

5

6

7

8

9 101112131415

SOFA values

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

Cost

50

60

70

80

90

100 110 120 130 140 150

MeanBP values

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

0.55

0.60

0.65

0.70

0.75

0.80

Cost

50

60

70

80

90

100 110 120 130 140 150

HR values

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

0.55

0.60

0.65

0.70

0.75

0.80

Cost

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

Lactate values

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

Cost

CT (No attention layer)

CT

686

687

688

689

690

691

692

693

694

695

696

697

698

699

700

701

702

703

704

705

706

707

708

709

710

711

712

713

714

715

716

717

718

719

720

721

722

723

724

725

726

727

728

729

730

731

732

733

734

735

736

737

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: In the abstract and introduction, we delineate the main motivations and contributions of this paper and its application in the field of safe reinforcement learning in healthcare.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: In the final section, this paper discusses the limitations of the method.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We have documented the relevant theories and assumptions in the paper or supplementary materials.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Our approach is reproducible, and our code can be made publicly available after the paper is published, including the relevant data processing procedures.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Our code can be made publicly available after the paper is published.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provided a detailed description of the experimental setup and metrics.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We tested our method with multiple random seeds and calculated the standard error.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We explain the required computational resources and related information in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Although our work is related to healthcare, we train and test our models on offline data, adhering to ethical standards.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer:[Yes]

Justification: Our work has a positive impact on safe healthcare, promoting the expansion of artificial intelligence technology into the medical field.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our work does not pose security risks because it is based on publicly available datasets and models.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The code, data, and models we referenced are all cited, and we followed the licenses and terms of use throughout the process.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We will provide detailed data extraction code and model code as part of the submission files.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

998 999 1000 1001 1002

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies inverse RL with learnable constraints in the offline setting, focusing on practical applications in healthcare tasks. The main approach appears to be combining the decision transformer architecture in the offline RL literature with inverse constrained RL with max entropy framework. Experiments were conducted on two healthcare tasks: sepsis and mechanical ventilation.\n\nSTRENGTHS:\nStudies an interesting and important problem.\n\nWEAKNESSES:\nThere is significant issues with the writing: the text is incoherent throughout and really hindered my understanding of the paper.\n\nQUESTIONS:\n1. Unclear what exactly is the \"gap\" the paper addresses. It mentions several issues including Markov assumption and history dependence, personalization & individual differences, offline setting, decision transformers, and it's unclear what the focus is. Otherwise, the paper seems to be applying decision transformer architecture to the constrained IRL problem. \n\n1. Table 1: \"too high\" associated with increased mortality making it an \"unsafe behavior\" - this seems like confounding, patients are sick and that's why they receive high dose. Calling it \"unsafe behavior\" without conditioning on the patient state is inappropriate. \n\n2. Claims about Markovianity: if Markov assumption does not hold, it seems to suggest the state is not capturing enough information about the \"environment state\". \n    - L49: \"The Markov decision is not compatible with medical decisions.\" and \"historical states of patients are crucial for medical decision\" why can't one incorporate history into states? \n    - Fig 1: if within the same state different treatments should be used due to individual differences, that also suggests the state definition is not capturing individual differences (L57 \"ignore individual differences\") and you should redefine the states. \n3. Organization & Flow:\n    - page 4 Sec 4 Methods talks about ICRL - is this your proposed method or previous work? If it's previous work it should be stated clearly and probably does not belong in Methods. \n    - L116 - when talking about Sec 3 Problem Formulation of constrained MDP, the text talks about \"we extract data within 72 hours of patient admission, with each 4-hour interval\" - this seems like experimental details rather than the mathematical setup of your method. \n4. Writing and notation: writing is incoherent, many notations are not defined. \n    - L122 cost c_t \\sim C, shouldn't cost depend on (s,a)? \n    - Eqn (2) in the definition of \"probability of approaching the optimal policy\", what does \"top N\" mean? Top N according to what? What is N? If N is dataset size, how do you \"collect 2N from offline dataset\" (L144)? \n    - L145 \"calculate the DIFF and sort it in ascending order\" Is the DIFF just one number? How do you sort it? What are you sorting? \n    - Eqn (3) what is R(tau), L157 is it ZM or ZMC, what is β\n    - L162 \"where M ζθ denotes the MDP obtained after augmenting M with the cost function Cθ\" Do you mean ζθ is the same as Cθ? What is π_Mζθ (L163)?\n    - L202 \"doesn't\" contractions should be avoided in academic writing\n    - L206 \"This layer is employed to define the cost weight for non-Markovians\" What is \"non-Markovians\"? \n    - L207 What is \"causality transformer\"? In \"generate the importance weights\", importance weights has a specific meaning in RL. The authors probably meant the cost weights. \n    - L219 \"We construct an expert and a violating dataset to evaluate Equation 6 in offline.\" This sentence doesn't make sense grammatically. \n    - L248 \"generating violating data\" have you defined violating data? \"may incentivize\": why does the procedure incentivize generation of \"violating data\", do you show it theoretically or empirically? \n    - L283 why \"Notably, the training dataset consists of data from surviving patients\" Please provide more justification for this. Training on only surviving patients would limit the dataset and state-actions available to the learning agent. \n5. Questions about experiments: \n    - L297 if cost values are positively correlated with mortality rates which is the reward, what's the point of having a separate cost function instead of just using the reward? \n    - L300 \"This indicates that the attention layer plays a crucial role in assessing constraints.\" could you elaborate why? \n    - L321 \"FQE cannot evaluate unseen (s,a)\" are you saying your approach can evaluate unseen (s,a)? \n    - L331 \"combined DIFF\": vaso and IV have very different scales, how do you combine? \n    - L357 \"corresponding experiments are conducted on the mechanical ventilator\". What? \n    - It's unclear what Fig 8 is showing. Table 3 not explained, unclear if lower or higher is better, what is max ∆?\n\nLIMITATIONS:\nLimitations are discussed.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper uses the Inverse Constrained Reinforcement Learning (ICRL) framework to infer constraints in healthcare problems from expert demonstrations. It proposes the Constraint Transformer (CT) to address the dependence of decisions on historical data, which is generally ignored in ICRL methods with Markovian assumptions. It borrows the causal transformer from the previous decision transformer to incorporate history into constraint modeling. Additionally, a model-based offline RL model is trained to generate violating data. The CT demonstrates improved safety by effectively modeling constraints based on both violating and expert data.\n\nSTRENGTHS:\n-\tThe paper addresses a gap in existing ICRL applications by integrating historical data into the decision-making process.\n-\tThe paper augments the violating data in the offline training dataset with a generative world model.\n-\tThe proposed method has been thoroughly evaluated in three aspects: effective constraints, improved sepsis strategies, and safe policies.\n\nWEAKNESSES:\n-\tThe proposed method depends heavily on the generated violating data, which defines the objective function in the constraint transformer. How sensitive is the estimated policy to the generative world model? Figure 12 shows that the action distributions in the expert dataset and the violating dataset are different. The VASO action seldom takes a large value in the violating dataset. Will this distribution difference cause any trouble in the learning of the constraint?\n-\tCould the authors provide some details on how the DIFF between the estimated policy and the physicians' policy is calculated through graphical analysis? It would also be helpful if the authors could explain how Figure 7 is plotted. Is it calculated based on the dosage differences at each timestamp? In addition, what are the implications of the three DIFF evaluation metrics in Table 2? Since both IV and VASO are part of the action space, is the ACTION DIFF alone sufficient to evaluate the estimated policy?\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nThe authors have addressed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the Constraint Transformer (CT) framework to enhance safe decision-making in healthcare. The proposed CT model uses transformers to incorporate historical patient data into constraint modelling and employs a generative world model to create exploratory data for offline RL training. The authors supported their points by presenting experimental results in scenarios like sepsis treatment, showing that CT effectively reduces unsafe behaviours and approximates lower mortality rates, outperforming existing methods in both safety and interoperability.\n\nSTRENGTHS:\nThis paper shows its strengths in the following aspects:\n- The paper addresses the novel angle of ensuring safety in offline reinforcement learning (RL) for healthcare, a critical and previously underexplored issue.\n- It incorporates Inverse Constrained Reinforcement Learning (ICRL) into offline reinforcement learning (RL) for healthcare, introducing a novel approach to inferring constraints from expert demonstrations in a non-interactive environment.\n- The implementation of a causal transformer to learn the constraint function is interesting, allowing the integration of historical patient data and capturing critical states more effectively.\n- Extensive results on 2 datasets are presented. The proposed Constraint Transformer (CT) framework is shown to reduce unsafe behaviours and approximates lower mortality rates.\n\nWEAKNESSES:\nDespite its strengths and novelty, this paper suffers from several critical technical flaws, primarily concerning the soundness of evaluation rather than the method itself:\n\n1. **Definition of Metric**: The metric $\\omega$ is defined by comparing drug dosages related to **mortality rate**, which I believe is a flawed definition, even though it has been used in previous papers. Mortality rate can be influenced by numerous factors, making it unsuitable as a reward for RL, which considers a limited number of drugs. It is challenging to convince clinicians that mortality can indicate the 'treatment quality' of vasopressor or mechanical ventilation. This suggests that the reward is not solely a function of the previous action and state but also many unconsidered features (hidden variables) in the datasets, such as adrenaline, dopamine, historical medical conditions, phenotypes, etc. [1] pointed out that doctors usually set a MAP target (e.g., 65) and administer vasopressors until the patient reaches this safe pressure; [2] suggests using the NEWS2 score as the reward supported by clinical evidence. None of these directly use mortality. While it is understandable that this paper is not a clinical study, and hence, it is not the authors' responsibility to identify clinically appropriate reward designs, I recommend referring to [1] and [2] for a reward design that makes more clinical sense.\n\n2. **Definition of Optimal Policy**: This paper follows [3]'s definition of optimal policy. From my understanding, the clinician's policy $\\hat{\\pi}$ is approximated by a neural network. [2] pointed out that in the sepsis dataset, the behaviour policy can result in critical flaws in a very small number of states. Although the number of incorrect predictions is limited, they can still bias the off-policy evaluation results severely. I suspect this paper may encounter a similar issue. The authors should provide experiments and visualizations on the learning quality of the behaviour policy to justify their approach.\n\n3. **Model-Based Off-Policy Evaluation**: The data imbalance in both the sepsis and ventilation datasets is significant. It is questionable whether the learned model can generalize well. The most acceptable way to validate the method remains using simulated data where all policies can be tested online. One possible testbed is the DTR-Bench[4] medical simulated environment.\n\nThe paper has a few other minor technical flaws compared to the above three.\n\n[1] Jeter, Russell, et al. \"Does the\" Artificial Intelligence Clinician\" learn optimal treatment strategies for sepsis in intensive care?.\" arXiv preprint arXiv:1902.03271 (2019).\n\n[2] Luo, Zhiyao, et al. \"Position: Reinforcement Learning in Dynamic Treatment Regimes Needs Critical Reexamination.\" Forty-first International Conference on Machine Learning.\n\n[3] Aniruddh Raghu, Matthieu Komorowski, Leo Anthony Celi, Peter Szolovits, and Marzyeh Ghassemi. Continuous state-space models for optimal sepsis treatment: a deep reinforcement learning approach. In Machine Learning for Healthcare Conference, pages 147–163. PMLR, 2017\n\n[4] Luo, Zhiyao, et al. \"DTR-Bench: An in silico Environment and Benchmark Platform for Reinforcement Learning Based Dynamic Treatment Regime.\" arXiv preprint arXiv:2405.18610 (2024).\n\nQUESTIONS:\nAttention Mechanism: The paper highlights the importance of the causal attention mechanism in the Constraint Transformer. Since the two experimental datasets are both short-term time series datasets, is transformer really necessary? Is it possible that an RNN can be better than a transformer?\n\nLIMITATIONS:\nThere is no negative societal impact or limitation that needs clarification. \nIn addition to the weakness I mentioned, this paper:\n1. lacks off-policy evaluation results.\n2. may summarise more related work in this 'dynamic treatment regime' field. A few examples are listed below:\n\n[1] Kondrup, F., Jiralerspong, T., Lau, E., de Lara, N., Shkrob, J., Tran, M. D., Precup, D., and Basu, S. Towards safe mechanical ventilation treatment using deep offline reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 15696–\n15702, 2023.\n\n[2] Liu, Y., Logan, B., Liu, N., Xu, Z., Tang, J., and Wang, Y. Deep reinforcement learning for dynamic treatment regimes on medical registry data. In 2017 IEEE international conference on healthcare informatics (ICHI), pp. 380–385. IEEE, 2017.\n\n[3] Nambiar, M., Ghosh, S., Ong, P., Chan, Y. E., Bee, Y. M., and Krishnaswamy, P. Deep offline reinforcement learning for real-world treatment optimization applications. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 4673–4684, 2023.\n\n[4] Peng, X., Ding, Y., Wihl, D., Gottesman, O., Komorowski,M., Li-wei, H. L., Ross, A., Faisal, A., and Doshi-Velez,F. Improving sepsis treatment strategies by combining deep and kernel-based reinforcement learning. In AMIA Annual Symposium Proceedings, volume 2018, pp. 887. American Medical Informatics Association, 2018.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors consider healthcare applications of RL algorithms in which implicit constraint modeling is critical for safe recommendations. \nThis is modeled as an RL policy optimization with constraints. However, the constraints are often unknown and need to be inferred from expert data trajectories in the healthcare applications. The authors propose a neural network estimator to combine with the constrained MDP formulation. They also identify that the naive way to represent states leads to non-Markov structure. To address these issues, the paper proposes a simple modification (based on prior work on the \"preference attention layer\") to a causal transformer based policy which attempts to model a parametric constraint function. Evaluations are based on healthcare benchmarks.\n\nSTRENGTHS:\n- The paper considers a practically motivated approach grounded in real data for a healthcare application where data could be challenging to obtain.\n- Proposes simple addition to the final layer of a causal transformer to parametrize constraints on trajectories learned from expert data.\n- Evaluations are conducted on real healthcare domain benchmarks and extensive ablations are included to ensure that the architecture change is meaningful in obtaining the improvements.\n\nWEAKNESSES:\n- It seems like the system (which is the patient) is not feasible to model as evolving according to an Markov process on the observed state at each time, but instead a POMDP with a high dimensional latent state. \n\n- Clarity and presentation can be improved. In Equation (4), what is $\\hat{\\tau}$, this was never defined for being such a key component of the procedure. Similar issues in Eq (8) related to clarity.\n\nQUESTIONS:\n- In explaining the main Equation (4), the blurb in lines 162-164 needs some attention. Specifically, what does \"MDP obtained after augmenting M with cost function $C_\\theta$, using the executing policy ...\" mean exactly?\n\n- In Eq (8) what does $s_t, r_{t-1} \\in x_t \\sim D_e$ represent? It looks like $x_t$ is a representation generated by the transformer, and $s_t, r_t$ are from the data. Is there a problem in the notation?\n\n- Is the transformer representations being trained from the gradients of both the policy model as well as the world model? If so, how do they interact? If not, consider adding stop gradients at the relevant places in Eq (7) and/or (8) to make this explicit.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}