Title: Shielding Regular Safety Properties in Reinforcement Learning
Venue: NeurIPS 2024
Accepted: no
Keywords: Safe Reinforcement Learning

Model Checking

Shielding
Abstract: To deploy reinforcement learning (RL) systems in real-world scenarios we need to consider requirements such as safety and constraint compliance, rather than blindly maximizing for reward. In this paper we study RL with regular safety properties. We present a constrained problem based on the satisfaction of regular safety properties with high probability and we compare our setup to the some common constrained Markov decision processes (CMDP) settings. We also present a meta-algorithm with provable safety-guarantees, that can be used to shield the agent from violating the regular safety property during training and deployment. We demonstrate the effectiveness and scalability of our framework by evaluating our meta-algorithm in both the tabular and deep RL setting.

Abstract

To deploy reinforcement learning (RL) systems in real-world scenarios we need to consider requirements such as safety and constraint compliance, rather than blindly maximizing for reward. In this paper we study RL with regular safety properties. We present a constrained problem based on the satisfaction of regular safety properties with high probability and we compare our setup to the some common constrained Markov decision processes (CMDP) settings. We also present a meta-algorithm with provable safety-guarantees, that can be used to shield the agent from violating the regular safety property during training and deployment. We demonstrate the effectiveness and scalability of our framework by evaluating our meta-algorithm in both the tabular and deep RL setting.

1 Introduction

The field of safe reinforcement learning (RL) [6, 28] has gained increasing interest, as practitioners begin to understand the challenges of applying RL in the real world [26]. There exist several distinct paradigms in the literature, including constrained optimization [2, 20, 49, 58, 62, 74], logical constraint satisfaction [17, 24, 3638, 66], safety-critical control [15, 19, 53], all of which are unified by prioritizing safety- and risk-awareness during the decision making process.

Regular safety properties [9] are interesting because for all but the simplest properties the correspond29 ing cost function is non-Markovian. Our problem setup consists of the standard RL objective with 30 regular safety properties as constraints, we note that there has been a significant body of work that 31 combines temporal logic constraints with RL [17, 24, 36-38, 66], although many of these do not 32 explicitly separate reward and safety in the same way that we do. 33

34

Our approach relies on shielding [3], which is a safe exploration strategy that ensures the satisfaction

of temporal logic constraints by deploying the learned policy in conjunction with a reactive system 35

that overrides any unsafe actions. Most shielding approaches typically make highly restrictive 36 assumptions, such as full knowledge of the environment dynamics [3], or access to a simulator [29], 37 although there has been recent work to deal with these restrictions [30, 39, 73]. In this paper, we 38 opt for the most permissive setting, where the dynamics of the environment are unknown, runtime 39 verification of the agent is realized by finite horizon model checking with a learned approximation of 40 the environment dynamics. However, in principle our framework is flexible enough to accommodate 41 more standard model checking procedures as long as certain assumptions are met. 42

43

44

45

Our approach can be summarised as an online shielding approach (see Fig. 1), that dynamically identifies unsafe actions during training and deployment, and deploys a safe 'backup policy' when necessary. We summarise the main contributions of our paper as follows:

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

(1) We state a constrained RL problem based on the satisfaction of regular safety properties with high probability, and we identify the conditions whereby our setup generalizes several CMDP settings, including expected and probabilistic cumulative cost constraints.

(2) We present several model checking algorithms that can verify the finite-horizon satisfaction probability of regular safety properties, this includes statistical model checking procedures that can be used if either the transition probabilities are unavailable or if the state space is too large.

(3) We develop a set of sample complexity results for the statistical model checking procedures introduced in point (2), which are then used to develop a shielding meta-algorithm with provable safety guarantees, even in the most permissive setting (i.e., no access to the transition probabilities).

(4) We empirically demonstrate the effectiveness of our framework on a variety of regular safety properties in both a tabular and deep RL settings.

2 Related Work

Safety Paradigms in Reinforcement Learning. There exist many safety paradigms in RL, the most popular being constrained MDPs. For CMDPs several constrained optimization algorithms have been developed, most are gradient-based methods built upon Lagrange relaxations of the constrained problem [20, 49, 58, 62] or projection-based local policy search [2, 74]. Model-based approaches to CMDP [7, 11, 41, 64] have also gathered recent interest as they enjoy better sample complexity than their model-free counterparts, which can be imperative for safe learning [44].

Linear Temporal Logic (LTL) constraints [17, 24, 36-38, 66] for RL have been developed as an alternative to CMDPs to specify stricter and more expressive constraints. The LTL formula is typically treated as the entire task specification, although some works have aimed to separate LTL satisfaction and reward into two distinct objectives [66]. The typical procedure in this setting is to identify end components of the MDP that satisfy the LTL constraint and construct a corresponding reward function such that the optimal policy satisfies the LTL constraint with maximal probability. Formal PAC-style guarantees have been developed for this setting [27, 36, 66, 71] although they typically rely on non-trivial assumptions. We note that LTL constraints can capture regular safety properties, although we explicitly separate reward and safety, making the work in this paper distinct from previous work.

More rigorous safety-guarantees can be obtained by using safety filters [3], control barrier functions (CBF) [5], and model predictive safety certification (MPSC) [67, 68]. To achieve zero-violation training these methods typically assume that the dynamics of the system are known and thus they are typically restricted to low-dimensional systems. While these methods come from safety-critical control, they are closely related to safe reinforcement learning [15].

Learning Over Regular Structures. RL and regular properties have been studied in conjunction before, perhaps most famously as 'Reward Machines' [42, 43] - a type of finite state automaton that specifies a different reward function at each automaton state. Reward machines do not explicitly deal with safety, rather non-Markovian reward functions that depend on histories distinguished by regular languages. Several methods have been developed to exploit the structure of these automata and dramatically speed up learning [42, 43, 55, 61], e.g., counter factual experiences .

Regular decision processes (RDP) [13] are a specific class non-Markovian DPs [8] that have also been studied in several works [13, 22, 51, 59, 65]. Most of these works are theoretical and slightly out-of-scope for this paper, as the RDP setting does not explicitly handle safety and encompasses both non-Markovian rewards and transition probabilities.

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

Shielding. From formal methods, shielding for safe RL [3] forces hard constraints on policies, using a reactive system that 'shields' the agent from taking unsafe actions. Synthesising a correct-byconstruction reactive 'shield' typically requires access to the environment dynamics and can be computationally demanding when the state or action space is large. Several recent works have aimed to scale the concept of shielding to more general settings, relaxing the prerequisite assumptions for shielding, by either only assuming access to a 'black box' model for planning [29], or learning a world model from scratch [30, 39, 73]. Other notable works that can be viewed as shielding include, MASE [69] - a safe exploration algorithm with access to an 'emergency reset button', and Recovery-RL [63] - which has access to a 'recovery policy' that is activated when the probability of reaching an unsafe state is too high. A simple form of shielding with LTL specifications has also been considered [37, 54], but experimentally these methods have only been tested in quite simple settings.

3 Preliminaries

For a finite set S , let Pow ( S ) denote the power set of S . Also, let Dist ( S ) denote the set of distributions over S , where a distribution µ : S → [0, 1] is a function such that ∑ s ∈S µ ( s ) = 1 . Let S ∗ and S ω denote the set of finite and infinite sequences over S respectively. The set of all finite and infinite sequences is denoted S ∞ = S ∗ ∪ S ω . We denote as | ρ | the length of a sequence ρ ∈ S ∞ , where | ρ | = ∞ if ρ ∈ S ω . We also denote as ρ [ i ] the i +1 -th element of a sequence, when i < | ρ | , and we denote as ρ ↓ = ρ [ | ρ | -1] the last element of a sequence, when ρ ∈ S ∗ . A sequence ρ 1 is a prefix of ρ 2 , denoted ρ 1 ⪯ ρ 2 , if | ρ 1 | ≤ | ρ 2 | and ρ 1 [ i ] = ρ 2 [ i ] for all 0 ≤ i ≤ | ρ 1 | . A sequence ρ 1 is a proper prefix of ρ 2 , denoted ρ 1 ≺ ρ 2 , if ρ 1 ⪯ ρ 2 and ρ 1 = ρ 2 .

̸

Labelled MDPs and Markov Chains. An MDP is a tuple M = ( S , A , P , P 0 , R , AP , L ) , where S and A are finite sets of states and actions resp.; P : S × A → Dist ( S ) is the transition function ; P 0 ∈ Dist ( S ) is the initial state distribution ; R : S × A → [0, 1] is the reward function ; AP is a set of atomic propositions , where Σ = Pow ( AP ) is the alphabet over AP ; and L : S → Σ is a labelling function , where L ( s ) denotes the set of atoms that hold in a given state s ∈ S . A memory-less (stochastic) policy is a function π : S → Dist ( A ) and its value function , denoted V π : S → R is defined as the expected reward from a given state under policy π , i.e., V π ( s ) = E π [ ∑ T t =0 R ( s t , a t ) | s 0 = s ] , where T is a fixed episode length. Furthermore, denote as M π = ( S , P π , P 0 , AP , L ) the Markov chain induced by a fixed policy π , where the transition function is such that P π ( s ′ | s ) = ∑ a ∈ A P ( s ′ | s , a ) π ( a | s ) . A path ρ ∈ S ∞ through M π is a finite (or infinite) sequence of states. Using standard results from measure theory it can be shown that the set of all paths { ρ ∈ S ω | ρ pref ⪯ ρ } with a common prefix ρ pref is measurable [9].

Probabilistic CTL. (PCTL) [9] is a branching-time temporal logic for specifying properties of stochastic systems. A well-formed PCTL property can be constructed with the following grammar,

where a ∈ AP , ▷ ◁ ∈ { < , > , ≤ , ≥} is a binary comparison operator, and p ∈ [0, 1] is a probability. Negation ¬ and conjunction ∧ are the familiar logical operators from propositional logic, and next X , until U and bounded until U ≤ n are the temporal operators from CTL [9]. We make the distinction here between state formula Φ and path formula φ . The satisfaction relation for state formula Φ is defined in the standard way for Boolean connectives. For probabilistic quantification we say that s | = P ▷ ◁ p [ φ ] iff Pr( s | = φ ) := Pr( ρ ∈ S ω | ρ [0] = s , ρ | = φ ) ▷ ◁ p . Let Pr M ( s | = φ ) be the probability w.r.t. the Markov chain M . For path formula φ the satisfaction relation is as follows,

From the standard operators of propositional logic we may derive disjunction ∨ , implication → and coimplication ↔ . We also note that the common temporal operators 'eventually' ♢ and 'always' □ , and their bounded counterparts ♢ ≤ n and □ ≤ n can be derived in a familiar way, i.e., ♢ Φ ::= true U Φ , □ Φ ::= ¬ ♢ ¬ Φ , resp. ♢ ≤ n Φ ::= true U ≤ n Φ , □ ≤ n Φ ::= ¬ ♢ ≤ n ¬ Φ .

Regular Safety Property. A linear time property P safe ⊆ Σ ω over the alphabet Σ is a safety property if for all words w ∈ Σ ω \ P safe , there exists a finite prefix w pref of w such that P safe ∩ { w ′ ∈ Σ ω |

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

w pref ⪯ w ′ } = ∅ . Any such sequence w pref is called a bad prefix for P safe , a bad prefix w pref is called minimal iff there does not exist w ′′ ≺ w pref such that w ′′ is a bad prefix for P safe . Let BadPref ( P safe ) and MinBadPref ( P safe ) denote the set of of bad and minimal bad prefixes resp.

A safety property P safe ∈ Σ ω is regular if the set BadPref ( P safe ) constitutes a regular language. That is, there exists some deterministic finite automata (DFA) that accepts the bad prefixes for P safe [9], that is, a path ρ ∈ S ω is 'unsafe' if the trace trace ( ρ ) = L ( ρ [0]), L ( ρ [1]), . . . ∈ Σ ω is accepted by the corresponding DFA.

Definition 3.1 (DFA) . A deterministic finite automata is a tuple D = ( Q , Σ, ∆, Q 0 , F ) , where Q is a finite set of states, Σ is a finite alphabet, ∆ : Q× Σ →Q is the transition function, Q 0 is the initial state, and F ⊆ Q is the set of accepting states. The extended transition function ∆ ∗ is the total function ∆ ∗ : Q× Σ ∗ →Q defined recursively as ∆ ∗ ( q , w ) = ∆(∆ ∗ ( q , w \ w ↓ ), w ↓ ) . The language accepted by DFA D is denoted L ( D ) = { w ∈ Σ ∗ | ∆ ∗ ( Q 0 , w ) ∈ F} .

Furthermore, we denote as P H safe ⊆ Σ ω the corresponding finite-horizon safety property for H ∈ Z + , where for all words w ∈ Σ ω \ P H safe there exists w pref ⪯ w such that | w pref | ≤ H and w pref ∈ BadPref ( P safe ) . We model check regular safety properties by synchronizing the DFA and Markov chain in a standard way - by computing the product Markov chain.

Definition 3.2 (Product Markov Chain) . Let M = ( S , P , P 0 , AP , L ) be a Markov chain and D = ( Q , Σ, ∆, Q 0 , F ) be a DFA. The product Markov chain is M⊗D = ( S × Q , P ′ , P ′ 0 , { accept } , L ′ ) , where L ′ ( ⟨ s , q ⟩ ) = { accept } if q ∈ F and L ′ ( ⟨ s , q ⟩ ) = ∅ o/w, P ′ 0 ( ⟨ s , q ⟩ ) = P 0 ( s ) if q = ∆( Q 0 , L ( s )) and 0 o/w, and P ′ ( ⟨ s ′ , q ′ ⟩|⟨ s , q ⟩ ) = P ( s ′ | s ) if q ′ = ∆( q , L ( s ′ )) and 0 o/w.

To compute the satisfaction probability of P safe for a given state s ∈ S we consider the set of paths ρ ∈ S ω from s and the corresponding trace in the DFA. We provide the following definition.

Definition 3.3 (Satisfaction probability for P safe ) . Let M = ( S , P , P 0 , AP , L ) be a Markov chain and let D = ( Q , Σ, ∆, Q 0 , F ) be the DFA such that L ( D ) = BadPref ( P safe ) . For a path ρ ∈ S ω in the Markov chain, let trace ( ρ ) = L ( ρ [0]), L ( ρ [1]), . . . ∈ Σ ω be the corresponding word over Σ = Pow ( AP ) . From a given state s ∈ S the satisfaction probability for P safe is defined as follows,

Perhaps more importantly, we note that this satisfaction probability can be written as the following reachability probability in the product Markov chain,

where q s = ∆( Q 0 , L ( s )) and ♢ accept is a PCTL path formula that reads, 'eventually accept ' [9].

For the corresponding finite-horizon safety property P H safe we state the following result.

Proposition 3.4 (Satisfaction probability for P H safe ) . Let M and D be the MDP and DFA in Defn. 3.3. For a path ρ ∈ S ω in the Markov chain, let trace H ( ρ ) = L ( ρ [0]), L ( ρ [1]) . . . , L ( ρ [ H ]) be the corresponding finite word over Σ = Pow ( AP ) . For a given state s ∈ S the finite horizon satisfaction probability for P safe is defined as follows,

where H ∈ Z + is some fixed model checking horizon. Similar to before, we show that the finite horizon satisfaction probability can be written as the following bounded reachability probability,

where q s = ∆( Q 0 , L ( s )) is as before and ♢ ≤ H accept is the corresponding step-bounded PCTL path formula that reads, 'eventually accept in H timesteps'.

The unbounded reachability probability can be computed by solving a system of linear equations, the 173 bounded reachability probability can be computed with O ( H ) matrix multiplications, in both cases 174 the time complexity of the procedure is a polynomial in the size of the product Markov chain [9]. 175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

4 Problem Setup

In this paper, we are interested in the quantitative model checking of regular safety properties for a fixed finite horizon H and in the context of episodic RL, i.e., where the length of the episode T is fixed. In particular, at every timestep we constrain the (step-bounded) reachability probability Pr( ⟨ s , q ⟩ ̸| = ♢ ≤ H accept ) in the product Markov chain M π ⊗D . We assume that H is chosen so as to avoid any irrecoverable states [35, 64], i.e., those that lead to a violation of the safety property no matter the sequence of actions taken, the precise details of this notion are presented in Section 6. We specify the following constrained problem,

Problem 4.1 (Step-wise bounded regular safety property constraint) . Let P safe be a regular safety property, D be the DFA such that L ( D ) = BadPref ( P safe ) and M be the MDP;

where all probability is taken under the product Markov Chain M π ⊗D , p 1 ∈ [0, 1] is a probability threshold, H is the model checking horizon and T is the fixed episode length.

The hyperparameter p 1 is be directly used to trade-off safety and exploration in a semantically meaningful way; p 1 prescribes the probability of satisfying the finite-horizon safety property P H safe at each timestep. In particular, if p 1 is sufficiently small then we can guarantee (with high-probability) that the regular safety property P safe is satisfied for the entire episode length T .

Proposition 4.2. Let P T safe denote the (episodic) regular safety property for a fixed episode length T . Then satisfying Pr ( ⟨ s t , q t ⟩ | = ♢ ≤ H accept ) ≤ p 1 for all t ∈ [0, T ] guarantees that Pr( s 0 | = P T safe ) ≥ 1 -p 1 · ⌈ T/H ⌉ , where s 0 ∼ P 0 is the initial state.

Comparison to CMDP. In the remainder of this section, we compare our problem setup to various CMDP settings [4], with the aim of unifying different perspectives from safe RL. The purpose of this is to show that our proposed method for solving Problem 4.1 can also be used to satisfy other more common CMDP constraints. First, we define the following cost function that prescribes a scalar cost C > 0 when the regular safety property P safe is violated and 0 otherwise.

Definition 4.3 (Cost function) . Let P safe be a regular safety property and let D be the DFA such that L ( D ) = BadPref ( P safe ) , modified such that for all q ∈ F , q →Q 0 . The cost function is then defined as,

where C > 0 is some generic scalar cost and L ′ is the labelling function defined in Def. 3.2.

Resetting the DFA. Rather than reset the environment, the DFA is reset once it reaches an accepting state, so as to measure the rate of constraint satisfaction over a fixed episode length T . This can easily be realized by replacing any outgoing transitions from the accepting states with transitions back to the initial state, i.e., for all q ∈ F , q →Q 0 .

Non-Markovian costs. The cost function is Markov on the product states ⟨ s , q ⟩ ∈ S × Q . However, in most cases the cost function is non-Markovian in the original state space S , since the automaton state q ∈ Q could depend on some arbitrary history of states. Thus our problem setup generalizes the standard CMDP framework with non-Markovian safety constraints.

Invariant properties. Invariant properties P inv (Φ) , also written □ Φ ('always Φ '), where Φ is a propositional state formula, are the simplest type of safety properties where the cost function is still Markov in the original state space. In this case we are operating in the standard CMDP framework, we also note that checking invariant properties with a fixed model checking horizon has been studied in previous works, as bounded safety [29, 30] and safety for a finite horizon [45].

The most common type of CMDP constraints are expected cumulative (cost) constraints , which constrain the expected cost below a given threshold.

Problem 4.4 (Expected cumulative constraint [4, 58]) .

where d 1 ∈ R + is the cost threshold and T is the fixed episode length.

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

Probabilistic cumulative (cost) constraints , are a stricter class of constraints that constrain the cumulative cost with high probability, rather than in expectation.

Problem 4.5 (Probabilistic cumulative constraint [18, 56]) .

where d 2 ∈ R + is the cost threshold, δ 2 is a tolerance parameter, and T is the fixed episode length.

We also consider instantaneous constraints , which bound the cost 'almost surely' at each timestep t ∈ [0, T ] . These are an even stricter type of constraint for highly safety-critical applications.

Problem 4.6 (Instantaneous constraint [23, 60, 69]) .

where d 3 ∈ R + is the cost threshold and T is the fixed episode length.

In particular, these problems define a constrained set of feasible policies Π . We make the distinction here between a feasible policy and a solution to the problem, the former being any policy satisfying the constraints of the problem and the later being the optimal policy within the feasible set Π .

Theorem 4.7. A feasible policy for Problem 4.1 is also a feasible policy for Problems 4.4, 4.5 and 4.6 under specific parameter settings for p 1 , d 1 , d 2 and δ 2 , and d 3 .

In Appendix G we provide a full set of statements that outline the relationships between the constrained problems presented in this section. The significance of these results is that they demonstrate by solving Problem 4.1 with our proposed method we can obtain feasible policies for Problems 4.4, 4.5 and 4.6, although for most of these problems there is no direct relationship between our problem setup, in particular we can say little about whether the optimal policy for one problem is necessarily optimal for another. Nevertheless, we find it interesting to explore the relationships between our setup and other perhaps more common constrained RL problems.

5 Model checking

In this section we outline several procedures for checking the finite-horizon satisfaction probability of regular safety properties and we summarise the settings in which they can be used.

Assumption 5.1. We are given access to the 'true' transition probabilities P .

Assumption 5.2. We are given access to a 'black box' model that perfectly simulates the 'true' transition probabilities P .

Assumption 5.3. We are given access to an approximate dynamic model ̂ P ≈ P , where the total variation (TV) distance D TV ( P π ( · | s ), ̂ P π ( · | s )) ≤ ϵ/H , for all s ∈ S . 1

Exact model checking. Under Assumption 5.1 we can precisely compute the (finite horizon) satisfaction probability of P safe , in the Markov chain M π induced by the fixed policy π in time O ( poly ( size ( M π ⊗D )) · H ) [9], where D is the DFA such that L ( D ) = BadPref ( P safe ) and H is the model checking horizon. H should not be too large and so the complexity of exact model checking ultimately depends on the size of the product M π ⊗D , and so if the size of either the MDP or DFA is too large then exact model checking may be infeasible.

Monte-Carlo model checking. To address the limitations of exact model checking, we can drop Assumption 5.1. Rather, under Assumption 5.2, we can sample sufficiently many paths from a 'black box' model of the environment dynamics and estimate the reachability probability Pr( ⟨ s , q ⟩ | = ♢ ≤ H accept ) in the product Markov chain M π ⊗D , by computing the proportion of accepting paths. Using statistical bounds, such as Hoeffding's inequality [40] or Bernstein-type bounds [52], we can bound the error of this estimate, with high probability.

Proposition 5.4. Let ϵ > 0 , δ > 0 , s ∈ S and H ≥ 1 be given. Under Assumption 5.2, we can obtain an ϵ -approximate estimate for the probability Pr( ⟨ s , q ⟩ | = ♢ ≤ H accept ) with probability at least 1 -δ , by sampling m ≥ 1 2 ϵ 2 log ( 2 δ ) paths from the 'black box' model.

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

We note that the time complexity of these statistical methods does not depend in the size of the product MDP or DFA, since the product states ⟨ s , q ⟩ ∈ S × Q can be computed on-the-fly , rather the time complexity depends on the horizon H , the desired level of accuracy ϵ , failure probability δ .

Model checking with approximate models. In most realistic cases neither the 'true' transition probabilities nor a perfect 'black box' model is available to us before-hand. Under Assumption 5.3 we can model check with an 'approximate' model of the MDP dynamics, which can either be constructed ahead of time (offline) or learned from experience, with maximum likelihood (or similar). We can then either exact model check in with the 'approximate' probabilities, or if the MDP is too large, we can leverage statistical model checking by sampling paths from the 'approximate' model.

Proposition 5.5. Let ϵ > 0 , δ > 0 , s ∈ S and H ≥ 1 be given. Under Assumption 5.3 we can make the following two statements:

We can obtain an ϵ -approximate estimate for Pr( ⟨ s , q ⟩ | = ♢ ≤ H accept ) with probability 1 by exact model checking with the transition probabilities of ̂ P π in time O ( poly ( size ( M π ⊗D )) · H ) .

We can obtain an ϵ -approximate estimate for Pr( ⟨ s , q ⟩ | = ♢ ≤ H accept ) with probability at least 1 -δ , by sampling m ≥ 2 ϵ 2 log ( 2 δ ) paths from the 'approximate' dynamics model ̂ P π .

6 Shielding the policy

At a high-level, the shielding meta-algorithm works by switching between the 'task policy' trained with RL to maximize rewards and a 'backup policy', which typically constitutes a low-reward, possibly rule-based policy that is guaranteed to be safe. In some cases this 'backup policy' may be available to us before training, although in most realistic cases it will need to be learned. In our case we switch from the 'task policy' to the 'backup policy' when the reachability probability Pr( ⟨ s , q ⟩ | = ♢ ≤ H accept ) exceeds the probability threshold p 1 . To check this we can use any of the model checking procedures presented earlier. The 'backup policy' is used when the reachability probability exceeds p 1 . Intuitively if the 'backup policy' is guaranteed to be safe, then our system should satisfy the constraints of Problem 4.1, independent of the 'task policy'.

Backup policy. In general we assume no knowledge of the safety dynamics before training and so the 'backup policy' needs to be learned. In particular, we can use the cost function defined in Defn. 4.3 and train the 'backup policy' with RL to minimize the expected discounted cost

( E π [ ∑ T t =0 γ t C ( s t , q t )] ). Importantly, we note that the cost function is defined on the product state space S × Q and so the 'backup policy' must also operate on this state space, possibly leading to slower convergence. However, we can eliminate this issue entirely by training the 'backup policy' with counterfactual experiences [42, 43] - a method originally used for reward machines that generates additional synthetic data for the policy, by simulating experience from each automaton state.

Meta Algorithm. We now present the structure of the shielding meta-algorithm (see Algorithm 1). The precise realization of this algorithm can vary depending on problem setting, tabular, deep RL, etc., however the main structure of the algorithm remains the same. In particular, during interaction with the environment we shield the agent by checking that the reachability probability Pr( ⟨ s , q ⟩ | = ♢ ≤ H accept ) does not exceed threshold p 1 . Then, with the new accumulated experience we update the 'task policy' denoted π task and the 'backup policy' denoted π safe with RL, and if need be

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

we update our (approximate) dynamics model accordingly. In principle, the underlying RL algorithm used to train either 'task policy' or 'backup policy' can differ, and the dynamics model can be a simple maximum likelihood estimate or something more complex, e.g., Gaussian Process model [25, 70], ensemble of parametric neural networks [21, 44] or a world model [32, 33].

Global Safety Guarantees. In the tabular setting we can guarantee the safety of the system described in Algorithm 1 under various assumptions, even when doing Monte-Carlo model checking on an 'approximate' model of the environment dynamics. First, we provide the following definitions.

Definition 6.1 (Non-critical state) . A product state ⟨ s , q ⟩ ∈ S × Q is said to be non-critical for a given model checking horizon H if for all policies π we have Pr( ⟨ s , q ⟩ | = ♢ ≤ H accept ) = 0 .

Definition 6.2 (Irrecoverable) . A critical state ⟨ s , q ⟩ ∈ S × Q is said to be irrecoverable with probability p 1 if for all policies π we have Pr( ⟨ s , q ⟩ | = ♢ accept ) ≥ p 1 . In other words, for any sequence of actions a 0 , a 1 , . . . the minimum probability Pr min ( ⟨ s , q ⟩ | = ♢ accept ) of reaching an accepting state is p 1 , where Pr min ( ⟨ s , q ⟩ | = ♢ accept ) = inf π Pr M π ⊗D ( ⟨ s , q ⟩ | = ♢ accept )

The safety-guarantees for Algorithm 1 rely on the following assumptions.

Assumption 6.3. We assume H is sufficiently large so that it is not possible to transition from any non-critical state to an irrecoverable state. Furthermore we assume that there exists some H ∗ < H such that if Pr min ( ⟨ s , q ⟩ | = ♢ accept ) = p 1 then Pr min ( ⟨ s , q ⟩ | = ♢ ≤ H ∗ accept ) = p 1 .

Assumption 6.4. The initial state ⟨ s 0 , L ( s 0 ) ⟩ is non-critical and for any state ⟨ s , q ⟩ ∈ S × Q that is not irrecoverable, the 'backup policy' π safe is satisfies Pr M π C ⊗D ( ⟨ s , q ⟩ | = ♢ ≤ H accept ) ≤ p 1

Theorem 6.5. Under Assumption 6.3 and 6.4, and provided that every state action pair ( s , a ) ∈ S×A has been visited at least O ( H 2 |S| 2 ϵ 2 log ( |A||S| 2 δ )) times. Then with probability 1 -δ the system satisfies the constraints of Problem 4.1, independent of the 'task policy'.

The theory is quite conservative here due to the strong dependence on | S | , in practice we can replace the outer | S | 2 by the maximum number of successor states from any given state. With regards to our assumptions, both are not overly restrictive. Assumption 6.3 essentially states that any irrecoverable states, will reach the accepting state with some probability > g within a fixed horizon H ∗ . Similar statements have been considered in prior work [35, 64]. Assumption 6.4 states that the 'backup policy' satisfies Pr( ⟨ s , q ⟩ | = ♢ ≤ H accept ) ≤ p 1 if possible, we would expect this to be the case when training the 'backup policy' with RL to minimize cost. The analysis for Theorem 6.5 then follows by showing that the system can be recovered to a non-critical state after entering a critical but not irrecoverable state.

7 Empirical Evaluation

We implemented two separate realizations of Algorithm 1, the first adapted to tabular environments which implements both exact or statistical model checking over the learned transition probabilities, the second is adapted to (visual) deep RL, making use of world models [32, 33], specifically DreamerV3 [34], to learn a latent dynamics model for model checking and policy optimization.

Tabular RL. We conduct experiments on a simple 'colour' gridworld environment, with regular safety properties of increasing difficulty. In short, the goal is to navigate from a starting state to a goal position as frequently as possible, while respecting a given regular safety property during training. The environment is stochastic - with some probability p the agent's action is ignored and another action is chosen uniformly instead. For smaller p val-

ues the environment becomes more deterministic and the safety property typically becomes easier to satisfy with higher probability, we refer the reader to Appendix D.1 for more details. Table 1 outlines the three safety properties used for our environments. We use PCTL-like notation to describe the safety properties, although strictly speaking (2) and (3) are actually PCTL ∗ path formula. Regardless of this slight technical detail, properties (1)-(3) are valid regular safety properties, as we can come up with a DFA that accepts the bad prefixes for them.

We compare our approach to Q-learning (without any penalties), and Q-learning on the product state space, with penalties provided by the cost function (Defn. 4.3) and trained with counterfactual

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

experiences [43]. In all cases, by separating reward and safety into two distinct policies, we are able to effectively trade-off the two objectives. Q-learning simply finds the best policy ignoring the costs, and Q-learning with penalties is able to find a safe policy, but struggles to meaningfully balance both objectives (see Fig. 2). Hyperparameter settings for all experiments are detailed in Appendix E. In addition, we provide an extensive series of ablation studies in Appendix F for these experiments. For example, we show that we don't loose much by using Monte Carlo model checking as opposed to exact model checking with the 'true' probabilities. We also show that tuning the cost coefficient C offers no meaningful way to trade-off reward and the probability of constraint satisfaction.

Deep RL. We deploy our version of Algorithm 1 built on DreamerV3 [34] on Atari Seaquest, provided as part of the Arcade Learning Environment (ALE)[10, 50]. We experiment with two different regular safety properties: (1) ( □ ¬ surface → □ ( surface → diver )) ∧ ( □ ¬ out-of-oxygen ) ∧ ( □ ¬ hit ) and (2) □ diver ∧ ¬ surface → ♢ ≤ 30 surface . We compare our approach to the base DreamerV3 algorithm and a version of DreamerV3 that implements the augmented Lagrangian penalty framework, similarly to [7, 41], for additional details see Appendix B.1.

Again our approach is able to effectively tradeoff both objectives, while (base) DreamerV3 ignores the cost, the Lagrangian approach appears to learn a safe policy that is not always efficient in terms of reward (see Fig. 3). We refer the reader to Appendix D.2 for more details of the environment and an extended discussion.

Separating Reward and Safety. The separation of reward and safety objectives into two distinct policies has been demonstrated as an effective strategy towards safety-aware decision making [3, 30, 46, 63], in many cases the safety objective is simpler and can be more quickly learnt [46]. In our experiments it is clear that when the system enters a critical state, the 'backup policy' is able to efficiently guide the system back to a non-critical state where the task policy can continue collecting reward. However, there is evidence that the complete separation of policies is not always appropriate [31] and penalties or a slight coupling of the policies is required to stop the 'task' and 'backup policy' fighting

for control of the system. Furthermore, by separating reward and safety, we typically loose any asymptotic convergence guarantees, similar to the situation faced for hierarchical RL [61], although there has been recent work to develop convergence guarantees for shielding [75].

8 Conclusion

In this paper we propose a shielding meta-algorithm for the runtime verification of regular safety 412 properties, given as a probabilistic constraint on the system. We provide a thorough theoretical 413 examination of the problem and develop probabilistic safety guarantees for the meta-algorithm, 414 which hold under reasonable assumptions. Empirically, we demonstrate that shielding is able to 415 effectively balance both reward and safety, in both the tabular and deep RL setting. A more thorough 416 theoretical and empirical examinations of the conditions for when shielding is appropriate would be 417 an interesting direction for future work. 418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

References

Pieter Abbeel and Andrew Y Ng. 2005. Exploration and apprenticeship learning in reinforcement learning. In Proceedings of the 22nd international conference on Machine learning . 1-8.

Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. 2017. Constrained policy optimization. In International conference on machine learning . PMLR, 22-31.

Mohammed Alshiekh, Roderick Bloem, Rüdiger Ehlers, Bettina Könighofer, Scott Niekum, and Ufuk Topcu. 2018. Safe reinforcement learning via shielding. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 32.

Eitan Altman. 1999. Constrained Markov decision processes: stochastic modeling . Routledge.

Aaron D Ames, Samuel Coogan, Magnus Egerstedt, Gennaro Notomista, Koushil Sreenath, and Paulo Tabuada. 2019. Control barrier functions: Theory and applications. In 2019 18th European control conference (ECC) . IEEE, 3420-3431.

Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. 2016. Concrete problems in AI safety. arXiv preprint arXiv:1606.06565 (2016).

Yarden As, Ilnura Usmanova, Sebastian Curi, and Andreas Krause. 2022. Constrained policy optimization via bayesian world models. arXiv preprint arXiv:2201.09802 (2022).

Fahiem Bacchus, Craig Boutilier, and Adam Grove. 1996. Rewarding behaviors. In Proceedings of the National Conference on Artificial Intelligence . 1160-1167.

Christel Baier and Joost-Pieter Katoen. 2008. Principles of model checking . MIT press.

M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. 2013. The Arcade Learning Environment: An Evaluation Platform for General Agents. Journal of Artificial Intelligence Research 47 (jun 2013), 253-279.

Felix Berkenkamp, Matteo Turchetta, Angela Schoellig, and Andreas Krause. 2017. Safe model-based reinforcement learning with stability guarantees. Advances in neural information processing systems 30 (2017).

James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. 2018. JAX: composable transformations of Python+NumPy programs . http: //github.com/google/jax

Ronen I Brafman, Giuseppe De Giacomo, et al. 2019. Regular Decision Processes: A Model for Non-Markovian Domains.. In IJCAI . 5516-5522.

Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. 2016. Openai gym. arXiv preprint arXiv:1606.01540 (2016).

Lukas Brunke, Melissa Greeff, Adam W Hall, Zhaocong Yuan, Siqi Zhou, Jacopo Panerati, and Angela P Schoellig. 2022. Safe learning in robotics: From learning-based control to safe reinforcement learning. Annual Review of Control, Robotics, and Autonomous Systems 5 (2022), 411-444.

Emma Brunskill, Bethany R Leffler, Lihong Li, Michael L Littman, and Nicholas Roy. 2009. Provably efficient learning with typed parametric models. (2009).

Mingyu Cai, Shaoping Xiao, Zhijun Li, and Zhen Kan. 2021. Optimal probabilistic motion planning with potential infeasible LTL constraints. IEEE transactions on automatic control 68, 1 (2021), 301-316.

Weiqin Chen, Dharmashankar Subramanian, and Santiago Paternain. 2024. Probabilistic constraint for safety-critical reinforcement learning. IEEE Trans. Automat. Control (2024).

Richard Cheng, Gábor Orosz, Richard M Murray, and Joel W Burdick. 2019. End-to-end safe 463 reinforcement learning through barrier functions for safety-critical continuous control tasks. In 464 Proceedings of the AAAI conference on artificial intelligence , Vol. 33. 3387-3395. 465

Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. 2018. Risk466 constrained reinforcement learning with percentile risk criteria. Journal of Machine Learning 467 Research 18, 167 (2018), 1-51. 468

469

470

471

Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. 2018. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. Advances in neural information processing systems 31 (2018).

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

Roberto Cipollone, Anders Jonsson, Alessandro Ronca, and Mohammad Sadegh Talebi. 2024. Provably Efficient Offline Reinforcement Learning in Regular Decision Processes. Advances in Neural Information Processing Systems 36 (2024).

Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerik, Todd Hester, Cosmin Paduraru, and Yuval Tassa. 2018. Safe exploration in continuous action spaces. arXiv preprint arXiv:1801.08757 (2018).

Giuseppe De Giacomo, Luca Iocchi, Marco Favorito, and Fabio Patrizi. 2020. Restraining bolts for reinforcement learning agents. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 34. 13659-13662.

Marc Deisenroth and Carl E Rasmussen. 2011. PILCO: A model-based and data-efficient approach to policy search. In Proceedings of the 28th International Conference on machine learning (ICML-11) . 465-472.

Gabriel Dulac-Arnold, Daniel Mankowitz, and Todd Hester. 2019. Challenges of real-world reinforcement learning. arXiv preprint arXiv:1904.12901 (2019).

Jie Fu and Ufuk Topcu. 2014. Probably approximately correct MDP learning and control with temporal logic constraints. arXiv preprint arXiv:1404.7073 (2014).

Javier Garcıa and Fernando Fernández. 2015. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research 16, 1 (2015), 1437-1480.

MGiacobbe, Mohammadhosein Hasanbeig, Daniel Kroening, and Hjalmar Wijk. 2021. Shielding atari games with bounded prescience. In Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS .

Alexander W Goodall and Francesco Belardinelli. 2023. Approximate Model-Based Shielding for Safe Reinforcement Learning. arXiv preprint arXiv:2308.00707 (2023).

Alexander W Goodall and Francesco Belardinelli. 2024. Leveraging Approximate Model-based Shielding for Probabilistic Safety Guarantees in Continuous Environments. arXiv preprint arXiv:2402.00816 (2024).

David Ha and Jürgen Schmidhuber. 2018. Recurrent World Models Facilitate Policy Evolution. In Advances in Neural Information Processing Systems , S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Eds.), Vol. 31. Curran Associates, Inc. https://proceedings.neurips.cc/paper_files/paper/2018/file/ 2de5d16682c3c35007e4e92982f1a2ba-Paper.pdf

Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. 2019. Learning latent dynamics for planning from pixels. In International conference on machine learning . PMLR, 2555-2565.

Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. 2023. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104 (2023).

Alexander Hans, Daniel Schneegass, Anton Schäfer, and Steffen Udluft. 2008. Safe Exploration for Reinforcement Learning. 143-148.

Mohammadhosein Hasanbeig, Alessandro Abate, and Daniel Kroening. 2018. Logicallyconstrained reinforcement learning. arXiv preprint arXiv:1801.08099 (2018).

Mohammadhosein Hasanbeig, Alessandro Abate, and Daniel Kroening. 2020. Cautious rein512 forcement learning with logical constraints. arXiv preprint arXiv:2002.12156 (2020). 513

514

515

516

517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

532

533

534

535

536

537

538

539

540

541

542

543

544

545

546

547

548

549

550

551

552

553

554

555

556

557

558

559

560

561

Mohammadhosein Hasanbeig, Daniel Kroening, and Alessandro Abate. 2020. Deep reinforcement learning with temporal logics. In Formal Modeling and Analysis of Timed Systems: 18th International Conference, FORMATS 2020, Vienna, Austria, September 1-3, 2020, Proceedings 18 . Springer, 1-22.

Chloe He, Borja G León, and Francesco Belardinelli. [n.d.]. Do androids dream of electric fences? Safety-aware reinforcement learning with latent shielding. CEUR Workshop Proceedings. https://ceur-ws.org/Vol-3087/paper_50.pdf

Wassily Hoeffding. 1963. Probability Inequalities for Sums of Bounded Random Variables. J. Amer. Statist. Assoc. 58, 301 (1963), 13-30.

Weidong Huang, Jiaming Ji, Borong Zhang, Chunhe Xia, and Yaodong Yang. 2023. Safe DreamerV3: Safe Reinforcement Learning with World Models. arXiv preprint arXiv:2307.07176 (2023).

Rodrigo Toro Icarte, Toryn Klassen, Richard Valenzano, and Sheila McIlraith. 2018. Using reward machines for high-level task specification and decomposition in reinforcement learning. In International Conference on Machine Learning . PMLR, 2107-2116.

Rodrigo Toro Icarte, Toryn Q Klassen, Richard Valenzano, and Sheila A McIlraith. 2022. Reward machines: Exploiting reward function structure in reinforcement learning. Journal of Artificial Intelligence Research 73 (2022), 173-208.

Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. 2019. When to trust your model: Model-based policy optimization. Advances in neural information processing systems 32 (2019).

Nils Jansen, Bettina Könighofer, Sebastian Junges, Alex Serban, and Roderick Bloem. 2020. Safe reinforcement learning using probabilistic shields. In 31st International Conference on Concurrency Theory (CONCUR 2020) . Schloss-Dagstuhl-Leibniz Zentrum für Informatik.

Nils Jansen, Bettina Könighofer, Sebastian Junges, Alexandru C Serban, and Roderick Bloem. 2018. Safe reinforcement learning via probabilistic shields. arXiv preprint arXiv:1807.06096 (2018).

Sham Kakade, Michael J Kearns, and John Langford. 2003. Exploration in metric state spaces. In Proceedings of the 20th International Conference on Machine Learning (ICML-03) . 306-312.

Michael Kearns and Satinder Singh. 2002. Near-optimal reinforcement learning in polynomial time. Machine learning 49 (2002), 209-232.

Qingkai Liang, Fanyu Que, and Eytan Modiano. 2018. Accelerated primal-dual policy optimization for safe reinforcement learning. arXiv preprint arXiv:1802.06480 (2018).

Marlos C. Machado, Marc G. Bellemare, Erik Talvitie, Joel Veness, Matthew J. Hausknecht, and Michael Bowling. 2018. Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents. Journal of Artificial Intelligence Research 61 (2018), 523-562.

Sultan Javed Majeed, Marcus Hutter, et al. 2018. On Q-learning Convergence for Non-Markov Decision Processes.. In IJCAI , Vol. 18. 2546-2552.

Andreas Maurer and Massimiliano Pontil. 2009. Empirical bernstein bounds and sample variance penalization. arXiv preprint arXiv:0907.3740 (2009).

Stephen McIlvanna, Nhat Nguyen Minh, Yuzhu Sun, Mien Van, and Wasif Naeem. 2022. Reinforcement learning-enhanced control barrier functions for robot manipulators. arXiv preprint arXiv:2211.11391 (2022).

Rohan Mitta, Hosein Hasanbeig, Jun Wang, Daniel Kroening, Yiannis Kantaros, and Alessandro Abate. 2024. Safeguarded Progress in Reinforcement Learning: Safe Bayesian Exploration for Control Policy Synthesis. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 38. 21412-21419.

Andrew Y Ng, Daishi Harada, and Stuart Russell. 1999. Policy invariance under reward 562 transformations: Theory and application to reward shaping. In Icml , Vol. 99. 278-287. 563

Santiago Paternain, Miguel Calvo-Fullana, Luiz FO Chamon, and Alejandro Ribeiro. 2022. Safe 564 policies for reinforcement learning via primal-dual methods. IEEE Trans. Automat. Control 68, 565 3 (2022), 1321-1336. 566

567

568

569

Aravind Rajeswaran, Igor Mordatch, and Vikash Kumar. 2020. A game theoretic framework for model based reinforcement learning. In International conference on machine learning . PMLR, 7953-7963.

570

571

572

573

574

575

576

577

578

579

580

581

582

583

584

585

586

587

588

589

590

591

592

593

Alex Ray, Joshua Achiam, and Dario Amodei. 2019. Benchmarking safe exploration in deep reinforcement learning. arXiv preprint arXiv:1910.01708 7, 1 (2019), 2.

Alessandro Ronca and Giuseppe De Giacomo. 2021. Efficient PAC reinforcement learning in regular decision processes. arXiv preprint arXiv:2105.06784 (2021).

Yanan Sui, Alkis Gotovos, Joel Burdick, and Andreas Krause. 2015. Safe exploration for optimization with Gaussian processes. In International conference on machine learning . PMLR, 997-1005.

Richard S Sutton, Doina Precup, and Satinder Singh. 1999. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial intelligence 112, 1-2 (1999), 181-211.

Chen Tessler, Daniel J Mankowitz, and Shie Mannor. 2018. Reward constrained policy optimization. arXiv preprint arXiv:1805.11074 (2018).

Brijen Thananjeyan, Ashwin Balakrishna, Suraj Nair, Michael Luo, Krishnan Srinivasan, Minho Hwang, Joseph E Gonzalez, Julian Ibarz, Chelsea Finn, and Ken Goldberg. 2021. Recovery rl: Safe reinforcement learning with learned recovery zones. IEEE Robotics and Automation Letters 6, 3 (2021), 4915-4922.

Garrett Thomas, Yuping Luo, and Tengyu Ma. 2021. Safe reinforcement learning by imagining the near future. Advances in Neural Information Processing Systems 34 (2021), 13859-13869.

Rodrigo Toro Icarte, Ethan Waldie, Toryn Klassen, Rick Valenzano, Margarita Castro, and Sheila McIlraith. 2019. Learning reward machines for partially observable reinforcement learning. Advances in neural information processing systems 32 (2019).

Cameron Voloshin, Hoang Le, Swarat Chaudhuri, and Yisong Yue. 2022. Policy optimization with linear temporal logic constraints. Advances in Neural Information Processing Systems 35 (2022), 17690-17702.

Kim P Wabersich and Melanie N Zeilinger. 2018. Linear model predictive safety certification 594 for learning-based control. In 2018 IEEE Conference on Decision and Control (CDC) . IEEE, 595 7130-7135. 596

Kim Peter Wabersich and Melanie N Zeilinger. 2021. A predictive safety filter for learning-based 597 control of constrained nonlinear dynamical systems. Automatica 129 (2021), 109597. 598

Akifumi Wachi, Yanan Sui, Yisong Yue, and Masahiro Ono. 2018. Safe exploration and opti599 mization of constrained mdps using gaussian processes. In Proceedings of the AAAI Conference 600 on Artificial Intelligence , Vol. 32. 601

602

603

Christopher KI Williams and Carl Edward Rasmussen. 2006. Gaussian processes for machine learning . Vol. 2. MIT press Cambridge, MA.

604

605

606

Eric M Wolff, Ufuk Topcu, and Richard M Murray. 2012. Robust control of uncertain Markov decision processes with temporal logic specifications. In 2012 IEEE 51st IEEE Conference on decision and control (CDC) . IEEE, 3372-3379.

Jorge Nocedal Stephen J Wright. 2006. Numerical optimization. 607

Wenli Xiao, Yiwei Lyu, and John Dolan. 2023. Model-based Dynamic Shielding for Safe 608 and Efficient Multi-agent Reinforcement Learning. In Proceedings of the 2023 International 609 Conference on Autonomous Agents and Multiagent Systems . 1587-1596. 610

Tsung-Yen Yang, Justinian Rosca, Karthik Narasimhan, and Peter J Ramadge. 2020. Projection611 based constrained policy optimization. arXiv preprint arXiv:2010.03152 (2020). 612

Wen-Chi Yang, Giuseppe Marra, Gavin Rens, and Luc De Raedt. 2023. Safe reinforcement 613 learning via probabilistic logic shields. arXiv preprint arXiv:2303.03226 (2023). 614

A Algorithms 615

Algorithm 2 Exact Model Checking [9] Input: model checking parameters ( p , H ), current state ⟨ s , q ⟩ , current action a , product MC M π ⊗D = ( S × Q , P ′ , P ′ 0 , { accept } , L ′ ) Output: true if Pr( ⟨ s , q ⟩ | = ♢ ≤ H accept ) ≤ p 1 Initialize zero vector x (0) ← 0 with size |S| × |Q| Initialize probability matrix A ← ( P ′ ( s , t )) s , t ̸∈ accept (ignoring accepting states) Initialize probability vector b ← ( P ′ ( s , accept )) s ̸∈ accept (going to accepting states) // Iterate over the model checking horizon for i = 1, . . . , H do Compute x ( i ) = Ax ( i -1) + b // Get the corresponding probability Let X ← x ⟨ s , q ⟩ If X < p return true else return false Algorithm 3 Monte-Carlo Model Checking Input: model checking parameters ( ϵ , δ , p , H ), current state ⟨ s , q ⟩ , current action a , policy π , labelling function L , DFA D = ( Q , Σ, ∆, Q 0 , F ) and (approximate) transition probabilities P Output: true if Pr( ⟨ s , q ⟩ | = ♢ ≤ H accept ) ≤ p 1 Choose m ≥ 2 / ( ϵ 2 ) log(2 /δ ) for i = 1, . . . , m do Set s 0 ← s , q 0 ← q and a 0 ← a // Sample a path through the model for j = 1, . . . , H do Sample next state s j ∼ P ( · | s j -1 , a j -1 ) , Compute q j ← ∆( q j -1 , L ( s j )) , Sample action a j ∼ π ( · | s j ) // Check if the path is accepting Let X i ← 1 [ q H ∈ F ] // Construct probability estimate Let ˜ X ← 1 m ∑ m i =1 X i If ˜ X < p -ϵ return true else return false Algorithm 4 Tabular Q-learning (Regular Safety Property) with Counter Factual Experiences [65] Input: MDP M = ( S , A , P , P 0 , R , AP , L ) , DFA D = ( Q , Σ, ∆, Q 0 , F ) , discount factor γ ∈ (0, 1] , learning rate α ∈ (0, 1] , temperature τ > 0 , cost coefficient C and fixed episode length T Initialize: (Q-table) ˆ Q ( s , q , a ) ← 0 ∀ s ∈ S , q ∈ Q , a ∈ A for each episode do Observe s 0 , L ( s 0 ) and q 0 ← ∆( Q 0 , L ( s 0 )) for t = 0, . . . , T do Sample action a t from ⟨ s t , q t ⟩ using the Boltzmann policy derived from ˆ Q with temp. τ Play action a t and observe s t +1 , L ( s t +1 ) and r t (reward is optional). // Generate synthetic data by simulating all automaton transitions for ¯ q ∈ Q do Compute ¯ q ′ ← ∆( q ′ , L ( s t +1 )) Compute cost ¯ c ′ ← C · 1[¯ q ′ ∈ F ] Compute done ← 1[¯ q ′ ∈ F ] // Q-learning step ˆ Q ( s t , ¯ q , a t ) ← (1 -α ) · ˆ Q ( s t , ¯ q , a t ) + α · ( r t +¯ c ′ + γ · done · max a ′ ∈A ˆ Q ( s t +1 , ¯ q ′ , a ′ ) Compute q t +1 ← ∆( q t , L ( s t +1 )) and continue

Algorithm 5 DreamerV3 [34] with Shielding (Regular Safety Property)

Initialize: replay buffer D with S random episodes, world model parameters θ , 'task policy' π task and 'backup policy' π safe randomly.

for each episode do

Sample action a ∼ π task from the task policy

// Estimate the reachability probability using the world model p θ

Use proposed action

a

t

a

else

Play action a t and observe o t +1 , L ( s t +1 ) and r t

Compute q t +1 ← ∆( q t , L ( s t +1 )) ,

Compute cost

c

t

1[

q

t

+1

∈ F

]

Append ( o t , a t , r t , c t , o t +1 ) to the replay buffer D

if update then

// World model learning

Sample a batch B of transition sequences { ( o t ′ , a t ′ , r t ′ , c t ′ , o t ′ +1 ) } ∼ D .

Update the world model parameters θ with maximum likelihood.

// Task policy optimization

'Imagine' sequences { ˆ o t ′ : t ′ + H , ˆ r t ′ : t ′ + H , ˆ c t ′ : t ′ + H } with the 'task policy' π

Update the 'task policy' π

task task with RL (to maximize reward).

Update the corresponding value critics with maximum likelihood

// Backup policy optimization

'Imagine' sequences { ˆ o t ′ : t ′ + H , ˆ r t ′ : t ′ + H , ˆ c t ′ : t ′ + H } with the 'backup policy' π safe

Update the 'backup policy' π safe with RL (to minimize cost)

Update the corresponding value critics with maximum likelihood

B Technical Details 616

B.1 Augmented Lagrangian 617

We first define the following objective functions, 618

(3)

The augmented Lagrangian [72] is an adaptive penalty-based technique for the following constrained 619 optimization problem, 620

where d is some cost threshold. The corresponding Lagrangian is given by, 621

The LHS is an equivalent form for the constrained optimization problem (RHS), since if π is feasible, 622 i.e. J C ( π ) < d then the maximum value for λ is λ = 0 . If π is not feasible then λ can be arbitrarily 623 large to solve this equation. Unfortunately this form of the objective function is non-smooth when 624 moving from feasible to infeasible policies, thus we introduce a proximal relaxation of the augmented 625

Lagrangian [72], 626

630

631

where, 633

634

635

636

637

638

639

640

641

642

643

644

645

646

C Technical Proofs

C.1 Proof of Proposition 3.4

Proposition 3.4 (restated) (Satisfaction probability for P H safe ) . Let M and D be the MDP and DFA from before (Defn. 3.3). For a path ρ ∈ S ω in the Markov chain, let trace H ( ρ ) = L ( ρ [0]), L ( ρ [1]) . . . , L ( ρ [ H ]) be the corresponding finite word over Σ = Pow ( AP ) . For a given state s ∈ S the finite horizon satisfaction probability for P safe is defined as follows,

where H ∈ Z + is some fixed model checking horizon. Similar to before, we show that the finite horizon satisfaction probability can be written as the following bounded reachability probability,

where q s = ∆( Q 0 , L ( s )) is as before and ♢ ≤ H accept is the corresponding step-bounded PCTL path formula that reads, 'eventually accept in H timesteps'.

Proof. Let P safe be a regular safety property and let D = ( Q , Σ, ∆, Q 0 , F ) be the DFA such that L ( D ) = BadPref ( P safe ) . We provide a formal definition for P safe and the corresponding finite horizon property P H safe , respectively:

Let M = ( S , P , P 0 , AP , L ) be a Markov chain and consider the product Markov chain M⊗D 647 from Defn. 3.2. For any path ρ = s 0 , s 1 , s 2 , . . . , there exists a unique run q 0 , q 1 , q 2 , . . . for the trace 648 trace ( ρ ) = L ( s 0 ), L ( s 1 ), L ( s 2 ) . . . , and denote, 649

where start state is ⟨ s 0 , ∆( Q 0 , L ( s 0 )) ⟩ . Before we deal with probabilities let's just consider a fixed path ρ ∈ S ω , the finite trace trace H ( ρ ) = L ( ρ [0]), L ( ρ [1]) . . . , L ( ρ [ H ]) , the unique run q 0 , q 1 , q 2 , . . . , q H and the path ρ + ∈ Σ ω ×Q ω in the product Markov chain. We prove the following statement,

650

651

652

653

where µ k is a non-decreasing penalty multiplier dependent on the gradient step k . The new term 627 that has been introduced here encourages the λ to stay close to the previous value λ k , resulting in a 628 smooth and differentiable function. The derivative w.r.t λ gives us the following gradient update step, 629

At each gradient step, the penalty multiplier µ k is updated in a non-decreasing way by using some small fixed (power) parameter σ ,

The policy π is then updated by taking gradient steps of the following unconstrained objective, 632

654

655

656

657

658

659

660

661

662

663

664

665

666

667

668

669

We start with the ( → ) direction, in particular, ρ ̸| = P H safe if and only if trace H ( ρ ) ∈ L ( D ) . Recall that by definition L ( D ) = { w ∈ Σ ∗ | ∆ ∗ ( Q 0 , w ) ∈ F} , and so trace H ( ρ ) ∈ L ( D ) implies that q H = ∆ ∗ ( Q 0 , trace H ( ρ )) ∈ F , which by construction implies that ρ + | = ♢ accept ≤ H .

The opposite direction ( ← ) is a little more involved, in particular, ρ + | = ♢ accept ≤ H implies that for the unique run q 0 , q 1 , q 2 , . . . , q H there exists t ≤ H such that q t ∈ F . We notice that since L ( D ) = BadPref ( P safe ) then once the DFA reaches an accepting state it will remain in an accepting state for the rest of the run. Therefore, q t ∈ F for t ≤ H implies that q H ∈ F . Then by definition the trace trace H ( ρ ) that determined the unique run q 0 , q 1 , q 2 , . . . , q H must be in the language L ( D ) , which again by definition implies that ρ ̸| = P H safe .

We now deal with the probabilities. First we note that the DFA D does not affect the probabilities of the product Markov chain - it can be shown that for every measurable set P of paths in M ,

see [9]. It now remains to construct this set P in the proper way. In particular, if P is the set of paths starting in some state s ∈ S and that refute P safe in the next H timesteps, i.e.,

̸

and P + is defined as the set of paths starting from the corresponding state ⟨ s , q s ⟩ (where q s = ∆( Q 0 , L ( s )) ) in M⊗D that eventually reach an accepting state of D in the next H steps, i.e.

Then by construction we have,

Finally the probability Pr M ( P ) and Pr M ( s | = P H safe ) are related as follows, 670

671

672

673

674

675

676

677

C.2 Proof of Proposition 4.2

Proposition 4.2 (restated). Let P T safe denote the (episodic) regular safety property for a fixed episode length T . Then satisfying Pr ( ⟨ s t , q t ⟩ | = ♢ ≤ H accept ) ≤ p 1 for all t ∈ [0, T ] guarantees that Pr( s 0 | = P T safe ) ≥ 1 -p 1 · ⌈ T/H ⌉ , where s 0 ∼ P 0 is the initial state.

Proof. Consider splitting up the episode in to ⌈ T/H ⌉ chunks with length at most H . Let X 0 , X 1 , . . . X ⌈ T/H ⌉-1 be the indicator random variables defined as follows,

Since Pr( ⟨ s t , q t ⟩ | = ♢ ≤ H accept ) ≤ p 1 for all t ∈ [0, T ] then the probability Pr( X i = 1) ≤ p 1 . By 678 construction we have, 679

Intuitively we satisfy P safe for the entire episode length if we never enter an accepting state in each of 680 the ⌈ T/H ⌉ chunks. The final result is then obtained by taking a union bound as follows, 681

(27)

682

683

684

685

686

687

688

689

690

691

694

695

696

697

C.3 Proof of Proposition 5.4

Proposition 5.4 (restated). Let ϵ > 0 , δ > 0 , s ∈ S be given. Under Assumption 5.2, we can obtain an ϵ -approximate estimate for Pr( ⟨ s , q ⟩ | = ♢ ≤ H accept ) with probability at least 1 -δ , by sampling m ≥ 1 2 ϵ 2 log ( 2 δ ) paths from the 'black box' model.

Proof. In words, we estimate Pr( ⟨ s , q ⟩ | = ♢ ≤ H accept ) by sampling m paths from a 'black box' model of the environment dynamics. We label each path as satisfying or not and return the proportion of satisfying traces as an estimate for Pr( ⟨ s , q ⟩ | = ♢ ≤ H accept ) . Weproceed as follows, let ρ 1 , . . . ρ m be a sequence of paths sampled from the 'black box' model and let trace ( ρ 1 ), . . . trace ( ρ m ) be the corresponding traces. Furthermore, let X 1 , . . . , X m be indicator r.v.s such that,

Recall that trace ( ρ 1 ) | = ♢ ≤ H accept can be checked in time O ( poly ( H )) . Now let, 692

then by Hoeffding's inequality [40], 693

Bounding the RHS from above by δ and rearranging gives the desired result.

C.4 Proof of Proposition 5.5

We start by introducing the following lemma.

Lemma C.1 (Error amplification for trace distributions) . Let ̂ P ≈ P be such that,

Let the start state s 0 ∈ S be given, and let P t ( · ) and ̂ P t ( · ) denote the path distribution (at time t ) for 698 the two transition probabilities P and ̂ P respectively. Then the total variation distance between the 699 two path distributions (at time t ) are bounded as follows, 700

Proof. We will prove this fact by doing an induction on t . We recall that P t ( · ) and ̂ P t ( · ) denote the 701 path distribution (at time t ) for the two transition probabilities P and ̂ P respectively. Formally we 702 define them as follows, 703

These probabilities read as follows, 'the probability of the sequence s 0 , . . . , s t ⪯ ρ at time t ', or similarly 'the probability that the sequence s 0 , . . . , s t is a prefix of ρ at time t ' Since the start state s 0 ∈ S is given we note that,

Before we continue with the induction on t we make the following observation, for any path ρ ∈ S ω 707 we have by the triangle inequality, 708

Now we continue with the induction on t , 709

The final result is obtained by an induction on t where the base case comes from P 0 ( · ) = ̂ P 0 ( · ) . 710

711

712

Proposition 5.5 (restated). Let ϵ > 0 , δ > 0 , s ∈ S and horizon H ≥ 1 be given. Under Assumption 5.3 we can make the following two statements:

713

714

715

716

717

718

719

720

721

(1) We can obtain an ϵ -approximate estimate for Pr( ⟨ s , q ⟩ | = ♢ ≤ H accept ) with probability 1 by exact model checking with the transition probabilities of ̂ P π in time O ( poly ( size ( M π ⊗D )) · H ) .

(2) We can obtain an ϵ -approximate estimate for Pr( ⟨ s , q ⟩ | = ♢ ≤ H accept ) with probability at least 1 -δ , by sampling m ≥ 2 ϵ 2 log ( 2 δ ) paths from the 'approximate' dynamics model ̂ P π .

Proof. We start by proving statement (1) and then statement (2) will follow quickly. First let Pr( ⟨ s , q ⟩ | = ♢ ≤ H accept ) and ̂ Pr( ⟨ s , q ⟩ | = ♢ ≤ H accept ) denote the acceptance probabilities for the two transition probabilities P and ̂ P respectively. We also let g ( · ) and ̂ g ( · ) denote the average trace distribution (over the next H timesteps) for the two transition probabilities P and ̂ P respectively, where,

Before we continue with the proof of (1) we make the following observations, 722

704

705

706

724

725

· Let f ( x ) : x ∈ X → [0, 1] be a real-valued function. Let P 1 ( · ) and P 2 ( · ) be probability distributions over the space X , then.

We continue by showing the following, 726

(53)

The first inequality (Eq. 47) comes from our earlier observations. The second inequality (Eq. 50) is straightforward and the final inequality (Eq. 51) is obtained by applying Lemma C.1 and Assumption 5.3. We note that this result is similar to the simulation lemma [48], which has been proved many times for several different settings [1, 16, 47, 57].

This concludes the proof of statement (1), since we have shown that ̂ Pr( ⟨ s , q ⟩ | = ♢ ≤ H accept ) is an ϵ/ 2 -approximate estimate of Pr( ⟨ s , q ⟩ | = ♢ ≤ H accept ) , under the Assumption 5.3.

The proof of statement (2) follows quickly. We have established that,

It remains to obtain an ϵ/ 2 -approximate estimate of ̂ Pr( ⟨ s , q ⟩ | = ♢ ≤ H accept ) . By using the same reasoning as in the proof of Proposition 5.4. We can obtain an ϵ/ 2 -approximate estimate of ̂ Pr( ⟨ s , q ⟩ | = ♢ ≤ H accept ) by sampling m paths, ρ 1 , . . . ρ m , from the approximate dynamics model ̂ P . Then provided,

with probability 1 -δ we can obtain ϵ/ 2 -approximate estimate of ̂ Pr( ⟨ s , q ⟩ | = ♢ ≤ H accept ) and by extension an ϵ -approximate estimate of Pr( ⟨ s , q ⟩ | = ♢ ≤ H accept ) . This concludes the proof.

C.5 Proof of Theorem 6.5

Theorem 6.5 (restated). Under Assumption 6.3 and 6.4, and provided that every state action pair ( s , a ) ∈ S ×A has been visited at least O ( H 2 |S| 2 ϵ 2 log ( |A||S| 2 δ )) times. Then with probability 1 -δ the system satisfies the constraints of Problem 4.1, independent of the 'task policy'.

Proof. We split the proof up in to three parts, (1) , (2) and (3) . In part (1) we show that the given sample complexity bound gives us an approximate model of the environment dynamics with high

727

728

729

730

731

732

733

734

735

736

737

738

739

740

741

742

743

744

745

probability. In part (2) we use our assumptions to reason about the probabilistic recoverability 746 of the system when it enters a critical state. In part (3) we put everything together and deal with 747 approximation error ϵ the remaining failure probability that are both unavoidable for the statistical 748 model checking procedures used to shield the system. 749

750

751

(1) We show that the following holds with probability 1 -δ/ 2 ,

when every state action pair ( s , a ) ∈ S × A has been visited at least,

times. First we let #( s , a ) denote the total number of times that ( s , a ) has been observed, similarly 752 we let #( s ′ , s , a ) denote the total number of times that ( s ′ , s , a ) has been observed. The maximum 753 likelihood estimate for the unknown probability P ( s ′ | , s , a ) is ̂ P ( s ′ | s , a ) = #( s ′ , s , a ) / #( s , a ) . 754 Let us fix some ( s , a ) ∈ S × A , and s ′ ∈ S , we let p s ′ = P ( s ′ | s , a ) denote the true probability of 755 transitioning to s ′ from ( s , a ) and we let ˆ p s ′ = #( s ′ , s , a ) / #( s , a ) denote our estimate. We note that 756 E [ˆ p s ′ ] = p s ′ , i.e. ˆ p s ′ is an unbiased estimator for p s ′ . Let m = #( s , a ) also be the number of times 757 that ( s , a ) has been observed, then by Hoeffding's inequality [40] we have, 758

Bounding the LHS from above by 1 -δ/ 2( |A||S| 2 ) and rearranging gives the following lower bound 759 for m , 760

Taking a union bound over all ( s ′ , s , a ) ∈ S × S × A , then for all state action pairs ( s , a ) ∈ S × A 761 we have the following with probability at least 1 -δ . 762

Now fix some s ∈ S and we observe the following, 763

Thus with probability at least 1 -δ/ 2 we have for all s ∈ S that, 764

(2) Using Assumption 6.3 and 6.4 we can argue about the safety of the system. Suppose firstly, 765 that we can check the condition Pr( ⟨ s , q ⟩ | = ♢ ≤ H accept ) ≤ p 1 , precisely and without any failure 766 probability (we will deal with statistical model checking in part (3) ). From any non-critical state we 767 can transition arbitrarily to a critical state, although under Assumption 6.3 this critical state is not 768 irrecoverable with probability ≥ p 1 . We now consider the following two cases: 769

771

772

773

774

775

776

777

778

779

780

781

782

783

784

785

786

787

788

789

790

791

792

793

794

795

796

797

798

799

800

801

802

803

804

805

806

807

808

809

810

811

812

813

814

815

816

817

818

(ii) Pr( ⟨ s , q ⟩ | = ♢ ≤ H accept ) > p 1 under the 'task' policy.

For case (i) we can safely use the 'task' policy and return to a non-critical state within H timesteps with probability at least 1 -p 1 . For case (ii) we deploy the 'safe' policy and under Assumption 6.4 we can return to a non-critical state within H timesteps with probability at least 1 -p 1 . We have now established an invariant, since from every non-critical state we can return to a non-critical state with probability 1 -p 1 and thus satisfy Pr( ⟨ s , q ⟩ | = ♢ ≤ H accept ) ≤ p 1 at every timestep t ∈ [0, T ] .

(3) We now make a similar argument but for the statistical model checking procedure where we can only obtain an ϵ -approximate estimate for the probability Pr( ⟨ s , q ⟩ | = ♢ ≤ H accept ) with high probability. Let us denote our ϵ -approximate estimate ̂ Pr( ⟨ s , q ⟩ | = ♢ ≤ H accept ) , rather than check the condition Pr( ⟨ s , q ⟩ | = ♢ ≤ H accept ) ≤ p 1 , we can check condition ̂ Pr( ⟨ s , q ⟩ | = ♢ ≤ H accept ) ≤ p 1 -ϵ , and if ̂ Pr( ⟨ s , q ⟩ | = ♢ ≤ H accept ) is indeed an ϵ -approximate estimate then this guarantees Pr( ⟨ s , q ⟩ | = ♢ ≤ H accept ) ≤ p 1 . Consider the following two cases:

(i) Our estimate ̂ Pr( ⟨ s , q ⟩ | = ♢ ≤ H accept ) ≤ p 1 -ϵ

(ii) Our estimate ̂ Pr( ⟨ s , q ⟩ | = ♢ ≤ H accept ) > p 1 -ϵ

For case (i) we can safely use the 'task' policy and return to a non-critical state within H timesteps with probability at least 1 -p 1 . For case (ii) we deploy the 'safe' policy and under Assumption 6.4 we can return to a non-critical state within H timesteps with probability at least 1 -p 1 . Again we have established an invariant, since from every non-critical state we can return to a non-critical state with probability 1 -p 1 and thus satisfy Pr( ⟨ s , q ⟩ | = ♢ ≤ H accept ) ≤ p 1 at every timestep t ∈ [0, T ] .

We still need to deal with the failure probability of the statistical model checking procedure at each timestep, by choosing failure probability 1 -δ/ 2 T we can guarantee (by a union bound) an ϵ -approximate estimate for each timestep with probability 1 -δ/ 2 . Finally, taking a union bound over part (1) and (2) gives the desired total failure probability 1 -δ .

D Environment Details

D.1 Colour Gridworld

The colour gridworld environment is a simple 9 × 9 grid, with state space |S| = 81 and action space |A| = 5 , where each action corresponds to the following movements: Left , Right , Up , Down , Stay . The objective is to navigate from the start state in one corner of the grid, to the goal state in the other corner, after reaching the goal state the agent is then sent back to the start state. The agent must navigate to the goal state as many times as possible in a fixed episode length of T = 1000 . The reward function is a sparse reward that gives the agent +1 reward for reaching the goal and 0 otherwise. When the environment is fully deterministic the maximum achievable reward is 58 .

In addition to the goal state, there are three other distinct states, green , blue and purple , each labelled with their corresponding colours, see Fig. 4. The set of atomic propositions is thus AP = { green , blue , purple , goal } , the safety properties are specified over the set AP , in particular we conduct experiments with 3 different safety properties of increasing complexity:

(1) □ ¬

(2) □ goal → ♢ ≤ 10 blue

(3) □ goal → ♢ ≤ 10 □ ≤ 5 purple

Property (1) is a simple invariant property P inv ( ¬ green ) that states the green state must always be avoided. Property (2) and (3) are more complex safety properties that interfere with the goal state. In

819

820

821

822

823

824

825

826

827

828

829

830

831

832

833

834

835

836

837

838

839

840

841

842

843

844

845

846

847

848

849

850

851

852

853

854

855

856

857

858

859

860

861

862

863

864

865

866

867

868

869

870

particular, property (2) states that once the goal state is reached then the blue state must be reached within 10 steps, this actually has no direct consequences on the maximum reward achievable but may interfere with convergence as the goal state seemingly leads to a high penalty if the blue state is not reached.

Property (3) states that once the goal state is reached then the purple state must be reached within 10 steps and then purple must hold for the next 5 timesteps. In safety property both interferes with the goal and has direct consequences on the maximum achievable reward as staying in purple for 5 steps does not lead to progress towards the goal state. In terms of the size of the DFA |Q| , property (1) is an invariant so the cost function is Markov and the size of the DFA is 2, for property (2) and (3) the size of the DFA is 12 and 62 respectively.

Each of the safety properties are tested with the corresponding p value for the environment, detailed in Table 1, which is repeated here for reference. The p value corresponds to the level of stochasticity in the environment. In particular, if p = 0.25 then there is a 25% chance of the agents action being overridden with another random action chosen uniformly. Given

the environment is stochastic then it is difficult to satisfy the safety properties with probability 1 . Through preliminary statistical analysis we computed the maximum satisfaction probabilities for each property, to help inform an appropriate p value to test with. With p = 0.25 , property (1) can be satisfies with very high probability close to 1 , while still achieving maximum reward. With p = 0.25 property (2) can be satisfied with probability ≈ 0.93 while still achieving maximum reward. With p = 0.1 property (3) can be satisfied with probability ≈ 0.75 while still achieving good reward.

Hyperparameter settings. We discuss some of the hyperparameter settings for our shielding approach that are not detailed in Table 5.

Property (1): we use a model checking horizon of H = 3 , and probability threshold p 1 = 1.0 , with the number of samples m = 4096 , we can obtain a roughly ϵ = 0.05 approximate estimate of the finite horizon satisfaction probability with failure probability δ = 0.01 .

Property (2): we use a model checking horizon of H = 10 , and probability threshold p 1 = 0.9 , with the number of samples m = 8192 , we can obtain a roughly ϵ = 0.05 approximate estimate of the finite horizon satisfaction probability with a smaller failure probability δ = 0.001 .

Property (3): again we use a model checking horizon of H = 10 , and probability threshold p 1 = 0.6 , with the number of samples m = 1024 , we can obtain roughly a ϵ = 0.1 approximate estimate of the finite horizon satisfaction probability with failure probability δ = 0.01 .

Extended discussion of results. First we provide slightly larger figures that than provided in the main paper, see Figure 5.

In general we observe that our shielding method is able to effectively trade-off reward and safety, in all cases converging to a system that obtains superior or comparable performance with the baseline. For property (1) we might expect our method to be able to recover the optimal policy that avoids the green state, it is clear in this case that the shielding procedure has harmed convergence and perhaps further investigation and hyperparameter tuning will encourage improvements. For property (2) and (3) the results are what we expect - we can recover the best policy that satisfies the step-wise bounded safety property with the desired probability p 1 .

The intuitive reason for why simply penalising Q-learning doesn't work, is that tuning the cost coefficient C is challenging for stochastic environments, where safety cannot be enforced 'almost surely' (with probability 1), and the precise value of C offers little to no semantic meaning. For different levels of stochasticity p values it is hard to know what desired level of safety we can achieve while still converging to a high reward policy, making tuning C even harder without knowing more about the structure of the environment. In Appendix F we study more closely the effect of C and

p

. Furthermore, we note te sensitivity of our method to the chosen model checking horizon particular, if

H

H

. In is too large we might expect the system to be overly conservative, we also address

D.2 Atari Seaquest 871

872

873

874

875

876

877

878

879

880

881

882

883

884

885

886

887

888

889

890

Our DreamerV3 [34] based shielding procedure is tested on Atari Seaquest, provided as part of the Arcade Learning Environment (ALE)[10, 50]. Seaquest is a partially observable environment meaning we do not have direct access to the underlying state space S , we are however provided with observations o ∈ O as pixel images which correspond to 64 × 64 × 3 tensors. Fortunately DreamerV3 is specifically designed to operate in visual settings and is able to effectively learn a predictive world model that closely approximate the environment dynamics. The action space of Seaquest is finite, specifically |A| = 18 , where each action corresponds to a joystick movement and fire button interaction. Rewards are obtained by 'shooting' an enemy shark or submarine, or by rescuing divers and returning them to the surface. In addition, the agent must manage its oxygen resources and avoid being hit by sharks and the enemy submarines which fire back, see Fig. 6. The environment is also made stochastic by using 'sticky actions' [50], where the agents previous action is repeated with probability p = 0.25 .

In terms of safety properties we experiment with the following two properties,

(2) □ diver ∧ ¬ surface → ♢ ≤ 30 surface

Property (1) states that after diving (i.e. not surface ), the agent must only surface with a diver on 891 board, and never run out-of-oxygen and never get hit by an enemy. The size of the DFA for this 892 property is |D| = 4 . Property (2) states that once a diver is on board the agent must surface within 30 893 timesteps (i.e. rescue the diver). 894

Hyperparameter settings. For our shielding approach almost all the hyperparameters are specified 895 in Appendix E. The only hyperparameter that varies is the model checking horizon H . For property 896 (1) we use H = 30 , empirically this seems adequate enough to avoid running out-of-oxygen and 897 begin surfacing in enough time. For property (2) we use H = 50 , this is to avoid picking up a diver 898 at the bottom of the ocean where it may not be possible to return to the surface in 30 timesteps. 899

900

901

902

903

904

905

906

907

908

909

910

911

912

913

914

915

916

917

918

919

920

921

Extended discussion of results. First we provide slightly larger figures that than provided in the main paper, see Figure 7

For both safety properties DreamerV3 with shielding obtains comparative performance in terms of reward with the unmodified DreamerV3 baseline. Of course this baseline entirely ignores the safety properties and simply maximizes reward. We remark on the differences between the safety properties themselves, property (1) in particular specifies the natural safety properties of the environment, since violating property (1) results in a death, the agent only start with 4 lives (and can gain one more ever 10000 points) and so satisfying property (1) is beneficial for long term reward, short the behaviour satisfying property (1) is correlated with higher reward and we might expect the globally optimal policy in the environment to never violated property (1). Property (2) specifies that once a diver is recovered the submarine must return to the surface in 30 timesteps, we would not expect that the globally optimal policy satisfies this property (2) rather we would expect to converge to a locally optimal policy satisfying property (2) while still obtaining good reward.

With respect to the baseline DreamerV3 (LAG) which has access to the cost function, we see that in both cases it fails to reliable learn a safe policy that simultaneously maximizes reward. For property (2) DreamerV3 (LAG) appear to do slightly better in terms of safety, however when qualitatively inspecting the runs for property (2) we see the DreamerV3 (LAG) agent intentionally get hit by enemy submarines/sharks to re-spawn on the surface without actually having to navigate there. This may be a more effective way to satisfy the safety property with high probability but it clearly leads to worse long term reward.

E Hyperparameters & Implementation Details

E.1 Access to Code

To maintain a high standard of anonymity we provide code for the experiments run on 'colour' 922 gridworld as supplementary material, rather than through GitHub. The colour gridworld environment 923 is implemented with the Gym [14] interface. Tabular Q-learning is implemented with numpy in Python , 924 the model checking procedures (both exact and Monte Carlo) are implemented with JAX [12] which 925 supports vectorized computation on GPU and CPU. The code for the Atari Seaquest experiments 926

927

928

929

930

931

932

933

934

935

936

937

938

939

940

941

942

943

944

are not currently available, although our code base was heavily derived from the code base for Approximate Model-based Shielding (AMBS) [30], see https://github.com/sacktock/AMBS (MIT License).

Training details. For collecting both sets of experiments we has access to 2 Nvidia Tesla A30 (24GB RAM) GPU and a 24-core/48 thread Intel Xeon CPU each with 32GB RAM. For the 'colour' gridworld experiments each run can take several minutes up to a day depending on which property is being tested, for example one run for property (3) can take roughly 1.5 days as the product state space is fairly large. For the Atari Seaquest experiments each run can take 8 hours to 1 day depending on the precise configuration of DreamerV3, in general we see a slow down of × 2 when using shielding compared to the unmodified DreamerV3 baseline. Memory requirements may differ depending on the DreamerV3 configuration used, for the xlarge DreamerV3 configuration 32GB of GPU memory should suffice.

Statistical significance. Error bars are provided for each of our experiments. In particular, we report 5 random initializations (seeds) for each experiment, the error bars are non-parametric (bootstrap) 95% confidence intervals, provided by seaborn.lineplot with default parameters: errorbar=('ci', 95), n_boot=1000 . The error bars capture the randomness in the initialization of the DreamerV3 world model and policy parameters, the randomness of the environment and any randomness in the batch sampling.

E.2 Colour Gridworld 945

E.3 Atari Seaquest 946

948

949

950

951

952

953

954

955

956

957

958

959

960

961

962

963

964

965

966

967

968

969

970

971

972

973

974

975

976

977

978

979

980

981

982

983

984

985

986

987

988

989

990

991

992

993

994

F Ablation Studies

In this section we provide several ablation studies for the 'colour' gridworld environment. We test the most significant hyperparameters and algorithmic components of our method including the baseline (Q-learning with penalties). In particular we demonstrate the counter factual experiences is crucial for learning the safety properties of the environment when the size of the corresponding DFA is non trivial. We also experiment with using exact model checking - demonstrating that we don't loose much by using statistical model checking procedures. Furthermore, we experiment with the cost coefficient C , the model checking horizon H and the level of stochasticity p .

F.1 Counter factual experiences

We run our method and the baseline (Q-learning with penalties) without counterfactual experiences to train the 'backup policy' or penalized task policy (baseline).

For property (2) and (3) we see a significant drop in safety performance, since learning to respect the safety property over the much larger product state space will require much more experience and without exploiting the structure of the DFA (using counter factual experiences) to generate synthetic data the task behaviour will be much more quickly learnt. For property (1), the invariant property, we observe identical performance as the DFA is trivial (only 2 states), and so counter factual experiences is essentially redundant in this case.

F.2 Exact model checking

We run our method (Shielding) with two different configurations: exact model checking with the 'approximate' transition probabilities (learning from experience) and exact model checking with the 'true' transition probabilities. We compare these two methods to the configuration used in the main paper: Monte Carlo (statistical) model checking with the learned transition probabilities.

In all cases we see that Shield (MC-Approx) obtains almost identical performance to Shield (Exact-True), which demonstrates that we don't loose much by statistical model checking with the learned probabilities, when for example we don't have access to the transition probabilities ahead of time, or the MDP is too large to exact model check. We see some variance with Shield (Exact-Approx), which can be explained by sub-optimal convergence in terms of reward, although note that the safety performance is consistent with the other configurations. Perhaps exact model checking with an inaccurate model of the transition probabilities restricts exploration to areas of the state space that are actually safe.

995

996

997

998

999

F.3 Cost coefficient C

We experiment with different values for the cost coefficient C used for our baseline (Q-learning with penalties). In particular, we use C ∈ { 0.1, 1.0, 10.0, 100.0 } , we expect that a larger cost coefficient will penalize unsafe behaviour more harshly and result in 'safer' behaviour (i.e., fewer safety-property violations).

Unsurprisingly, across the board, by increasing 1000 the cost coefficient C we obtain a policy that has 1001 fewer safety-property violations. The improved 1002 'safety performance' is of course at the expense 1003 of reward or task performance, this is a trade-off 1004 we would expect. In particular for C = 100.0 1005 we see that the learned policy essentially avoids 1006 the goal state (achieving zero reward) all but 1007 guaranteeing safety (no safety-violations). The 1008 purpose of this ablation study is to demonstrate 1009 that while we can achieve any desired level of 1010 safety by tuning the cost coefficient C , the actual 1011 value of C offers little to no semantic meaning 1012 for the probability of violating the safety prop1013 erty. 1014

1015

F.4 Model checking horizon H

1016

1017

1018

1019

1020

1021

1022

1023

1024

1025

1026

1027

As was alluded to in the main paper, our method can be very sensitive to the model checking horizon (hyperparameter) H . In particular, if H is too large then we might expect the system to exhibit overly conservative behaviour. As a rule of thumb we suggest that H should be set to roughly the shortest path in the DFA from the initial state to an accepting state - this can easily be computed by using Dijkstra's (shortest-path) algorithm. In this ablation we experiment with much larger H than recommended. This significantly impacts the performance of our proposed approach. However, we do propose a solution, Q-learning (Shield-Rec) which in short, checks that the action proposed by the 'task policy' is recoverable with the 'backup policy', or in other words by playing with the action a ∼ π task proposed by the 'task policy' We can still satisfy Pr( ⟨ s , q ⟩ | = ♢ ≤ H accept ) ≤ p 1 by using the 'backup policy' after playing a .

In general we observe that when H is too large 1028 our original method (Shield) is overly conser1029 vative, sacrificing reward or task performance 1030 for safety guarantees. Our proposed solution 1031 (Shield-Rec) is alleviates this issue partly, pro1032 viding reasonable safety performance and com1033 parable task performance. We note that this 1034 solution is clearly not perfect as is it appears 1035 to be slightly more permissive allowing more 1036 safety-violations than necessary. More investi1037 gation into this framework would be interesting 1038 future work, and perhaps more hyperparameter 1039 tuning, specifically by tuning p 1 , could improve 1040 this method. The goal would be to obtain an al1041 gorithm that is not overly sensitive to H , and as 1042 long as H is sufficiently big to guarantee safety 1043 we don't see much performance degradation by 1044 further increasing H . 1045

1046

1047

1048

1049

1050

1051

1052

1053

1054

1055

1056

1057

1058

1059

1060

F.5 Level of stochasticity p

Finally we investigate the effect of the level of stochasticity of the environment. Specifically, the value p corresponding the the probability that the agent's action is ignored and another action is chosen (uniformly at random) from the action space and played instead. For example, of p = 0.25 and the agent chooses the action Right , there is a 75% chance that the agent goes right and a 25% chance the agent goes a different direction. If p = 0.0 (deterministic environment) then achieving complete safety (zero-violations) becomes easier as the agent has complete control of the environment through their actions.

We experiment with the following p values: p = 0.1 for property (1), p = 0.1 for property (2) and p = 0.05 for property (3). For these smaller p values we would expect it to be easier for our methods including the baseline to achieve a higher-rate of safety and possibly complete safety in some cases.

We see a similar situation as in the main paper, 1061 Q-learning (without penalties) simply finds the 1062 best policy ignoring costs. However, Q-learning 1063 (with penalties) is able to obtain the same perfor1064 mance now as our method Q-learning (Shield), 1065 both in terms of reward and cost. With a smaller 1066 p value the safety-property can be satisfied with 1067 higher probability while still visiting the goal 1068 state frequently and obtaining high reward. In 1069 particular, these p values are chosen such that 1070 each of the safety properties can be satisfies with 1071 probability at least 0.9 from the goal state, thus 1072 penalizing safety-violations with C = 10.0 ap1073 pears to be enough to guarantee safety above 1074 0.9 at each timestep while still achieving high 1075

reward. For different values of C we might expect the baseline to have a different performance 1076 profile. 1077

1078

1079

1080

1081

1082

1083

1084

1085

1086

1087

1088

G Comparison to CMDP

In this additional section we analyze the relationships between our problem setup and other common CMDP settings, for both the finite horizon and corresponding (discounted) infinite horizon problems.

G.1 Finite Horizon

For reference we restate Problem 4.1 here.

Problem 4.1 (restated) (Step-wise bounded regular safety property constraint) . Let P safe be a regular safety property, D be the DFA such that L ( D ) = BadPref ( P safe ) and M be the MDP;

where all probability is taken under the product Markov Chain M π ⊗D , p 1 ∈ [0, 1] is a probability threshold, H is the model checking horizon and T is the fixed episode length.

G.1.1 Expected Cumulative Constraint

Problem 4.4 (restated) (Expected cumulative constraint [4, 58]) .

where d 1 ∈ R + is the cost threshold and T is the fixed episode length. 1089

Proposition G.1. A feasible policy π for Problem 4.1 with parameters p 1 ∈ [0, 1] is also a feasible 1090 policy for Problem 4.4 with parameter d 1 ∈ R + , provided that d 1 ≥ ( T +1) · p 1 . 1091

1092

where, 1093

Proof. For t ∈ [0, T ] we define, the following random variables, X 0 , . . . , X T , where

The argument is straightforward if at every timestep t ∈ [0, T ] we have Pr( ⟨ s t , q t ⟩ | = ♢ ≤ H accept ) ≤ 1094 p 1 then with probability ≤ p 1 we have accept ∈ L ( ⟨ s t , q t ⟩ ) . Then, under mild assumptions 1095 (i.e. C ( ⟨ s t , q t ⟩ ) < ∞ ) we consider the following decomposition of the expected cumulative cost, 1096

We replace the subscript ' ⟨ s t , q t ⟩ ∼ M π ⊗D ' here for brevity. Clearly by linearity of expectations 1097 this statement holds. Although it is worth noting that each expectation is taken under a different 1098 marginal state distribution (i.e. P t ( · ) ), which depends on π (apart from the initial state distribution 1099 P 0 ( · ) ). From now on we will write this is implicitly (i.e. Eq. 72), rather than writing the marginal 1100 state distribution (at time t ) for each expectation. Using our earlier observations we can now bound 1101 the expected cumulative cost from above as follows, 1102

[

T

∑

t

=0

]

1103

1104

1105

1106

1107

1108

1109

1110

=

E

π

[

X

0

] +

E

π

[

X

1

] +

. . .

+

E

π

[

X

T

-

1

] +

E

π

[

X

T

]

(73)

Proposition G.2. The converse is not strictly true, since there may be a feasible policy π for Problem 4.4 with threshold d 1 ≤ ( T +1) · p 1 which does not satisfy the constraints of Problem 4.1.

Proof. We want to prove the following statement, a policy π satisfying,

To prove this we will show that there may be some policy π that satisfies Eq. 75, but does not satisfy Eq. 76 at some timestep t . For simplicity we consider the first timestep (i.e. t = 0 ). First we assume π is such that Eq. 75 holds, assuming H ≤ T then clearly we have,

Let Pr( ⟨ s 0 , q 0 ⟩ | = ♢ ≤ H accept ) denote the proportion of accepting paths from the initial state 1111 s 0 ∼ P 0 ( · ) and automaton state q 0 = ∆( Q 0 , L ( s 0 )) . Suppose π is such that Pr( ⟨ s 0 , q 0 ⟩ | = 1112 ♢ ≤ H accept ) > p 1 . We note that for each path ρ ∈ S ω and corresponding trace ( ρ ) ∈ Σ ω such that 1113 trace ( ρ ) | = ♢ ≤ H accept the sum ∑ H t =0 C ( ⟨ s t , q t ⟩ ) ≥ 1 , and now we have, 1114

E

π

does not imply that,

C

(

⟨

s

t

,

q

t

⟩

)

Now clearly for all p 1 ∈ [0, 1] and T ∈ Z + the following holds, 1115

This implies that there may exist some π satisfying Eq. 75 and such that Pr( ⟨ s 0 , q 0 ⟩ | = ♢ ≤ H accept ) > p 1 , i.e. does not satisfy Eq. 76 at timestep t = 0 .

Proposition G.3. A feasible policy π for Problem 4.4 with threshold d 1 ≤ p 1 , satisfies Pr( ⟨ s t , q t ⟩ | = ♢ ≤ H accept ) ≤ p 1 for all t ∈ [0, T ] . This bound is tight.

Proof. Firstly, a feasible policy π for Problem 4.4 with threshold d 1 ≤ p 1 clearly satisfies,

Assuming H ≤ T , then this implies that for all t ′ ∈ [0, T -H ] we have, 1121

Let Pr( ⟨ s t ′ , q t ′ ⟩ | = ♢ ≤ H accept ) denote the proportion of accepting paths at timestep t ′ , where s t ′ ∼ P t ′ ( · ) . Here P t ′ ( · ) denotes the marginal state distribution at time t ′ . Recall that for each path ρ ∈ S ω and corresponding trace ( ρ ) ∈ Σ ω such that trace ( ρ ) | = ♢ ≤ H accept the sum ∑ t ′ + H t = t ′ C ( ⟨ s t , q t ⟩ ) ≥ 1 . Without loss of generality fix some t ′ ∈ [0, T -H ] and suppose that Pr( ⟨ s t ′ , q t ′ ⟩ | = ♢ ≤ H accept ) > p 1 . This implies that,

Which is a contradiction. Therefore, it must be the case that when Eq. 80 is satisfied then so is 1127 Pr( ⟨ s t , q t ⟩ | = ♢ ≤ H accept ]) ≤ p 1 for all t ∈ [0, T -H ] . For the remaining t ′ ∈ [ T -H , T ] a similar 1128 argument can be made, the only detail is to ensure the sum in Eq. 81 is up to T rather than t ′ + H . 1129 To prove that this bound is tight we can again show the possible existence of a counter example. In 1130 particular, we want to prove the following statement, a policy π satisfying, 1131

for some constant c > 0 , does not imply that, 1132

We will show that there may exist some policy π that satisfies Eq. 83 but does not satisfy Eq. 84 at some timestep t . Firstly, we assume π is such that Eq. 83 holds, this implies that for all t ′ ∈ [0, T -H ] we have,

1133

1134

1135

Fix some t ′ ∈ [0, T -H ] and once again let Pr( ⟨ s t ′ , q t ′ ⟩ | = ♢ ≤ H accept ) denote the proportion of 1136 accepting paths at timestep t ′ . Suppose π is such that Pr( ⟨ s t ′ , q t ′ ⟩ | = ♢ ≤ H accept ) > p 1 . Again recall 1137 that for each path ρ ∈ S ω and corresponding trace trace ( ρ ) ∈ Σ ω such that trace ( ρ ) | = ♢ ≤ H accept 1138 the sum ∑ t ′ + H t = t ′ C ( ⟨ s t , q t ⟩ ) ≥ 1 , and so, 1139

Now clearly for all p 1 ∈ [0, 1] and c > 0 , the following holds, 1140

This implies that there may exist some π satisfying Eq. 83 and such that Pr( ⟨ s t ′ , q t ′ ⟩ | = 1141 ♢ ≤ H accept ) > p 1 , i.e. does not satisfy Eq. 84 at timestep t = t ′ . 1142

1116

1117

1118

1119

1120

1122

1123

1124

1125

1126

G.1.2 Probabilistic Cumulative Constraint 1143

First we restate Problem 4.5. 1144

Problem 4.5 (restated) (Probabilistic cumulative constraint [18, 56]) .

where d 2 ∈ R + is the cost threshold, δ 2 is a tolerance parameter and T is the fixed episode length.

Proposition G.4. A feasible policy π for Problem 4.1 with parameters p 1 ∈ [0, 1] is also a feasible policy for Problem 4.5 with parameters d 2 ∈ R + and δ 2 ∈ (0, 1] , provided that, d 2 ≥ √ ( T +1) / 2 · log(1 /δ 2 ) + ( T +1) · p 1 .

Proof. For t ∈ [0, T ] we define the following random variables, X 0 , . . . , X T , where,

and we make the same following observation,

See the proof of Prop. G.1 for details, the argument is identical. Once again, under mild assumptions 1151 (i.e. C ( ⟨ s t , q t ⟩ ) < ∞ ) we consider the following decomposition of the expected cumulative cost, 1152

Again we replace the subscript ' ⟨ s t , q t ⟩ ∼ M π ⊗D ' here for brevity, see the proof of Prop. G.1 for the 1153 full details. Before we proceed we must first deal with the dependence between the random variables 1154 X 0 , . . . , X T . Strictly speaking it is not the case that Pr( X t = 1 | X t -1 , . . . , X 0 ) = Pr( X t = 1) . 1155 However, we have already established that Pr( X t = 1) ≤ p 1 , as such we can simulate X 0 , . . . , X T 1156 as a sequence of independent coin flips Y 0 , . . . , Y T with probability p 1 , it is then the case that 1157 P [ ∑ T t =0 X t > d 2 ] ≤ P [ ∑ T t =0 Y t > d 2 ] . We can now continue by bounding the probability we care 1158 about, 1159

1145

1146

1147

1148

1149

1150

Proposition G.5. A feasible policy π for Problem 4.5 with parameters δ 2 ≤ p 1 and d 2 < 1 , satisfies 1163 Pr( ⟨ s t , q t ⟩ | = ♢ ≤ H accept ) ≤ p 1 for all t ∈ [0, T ] . This bound is tight. 1164

Proof. A feasible policy π for Problem 4.5 with parameters δ 2 ≤ p 1 and d 2 < 1 clearly implies that, 1165

Assuming H ≤ T , then this implies that for all t ′ ∈ [0, T -H ] we have, 1166

Let Pr( ⟨ s t ′ , q t ′ ⟩ | = ♢ ≤ H accept ) denote the proportion of accepting paths at timestep t ′ , where s t ′ ∼ 1167 P t ′ ( · ) . Again P t ′ ( · ) denotes the marginal state distribution at time t ′ . Recall that for each path ρ ∈ S ω 1168 and corresponding trace ( ρ ) ∈ Σ ω such that trace ( ρ ) | = ♢ ≤ H accept the sum ∑ t ′ + H t = t ′ C ( ⟨ s t , q t ⟩ ) ≥ 1 . 1169 Without loss of generality fix some t ′ ∈ [0, T -H ] and suppose that Pr( ⟨ s t ′ , q t ′ ⟩ | = ♢ ≤ H accept ) > 1170 p 1 . This implies that, 1171

Which is a contradiction. Therefore, it must be the case that when Eq. 101 is satisfied then so is 1172 Pr( ⟨ s t , q t ⟩ | = ♢ ≤ H accept ]) ≤ p 1 for all t ∈ [0, T -H ] . For the remaining t ′ ∈ [ T -H , T ] a similar 1173 argument can be made, the only detail is to ensure the sum in Eq. 102 is up to T rather than t ′ + H . To 1174 prove that this bound is tight we can show the possible existence of a counter example. In particular, 1175 we want to prove the following statement, a policy π satisfying, 1176

for some constant c > 0 does not imply that, 1177

We will show that there may exist some policy π that satisfies Eq. 104 but does not satisfy Eq. 105 1178 at some timestep t . Firstly, we assume π is such that Eq. 104 holds, this implies that for all 1179 t ′ ∈ [0, T -H ] we have, 1180

Fix some t ′ ∈ [0, T -H ] and let Pr( ⟨ s t ′ , q t ′ ⟩ | = ♢ ≤ H accept ) denote the proportion of accepting 1181 paths at timestep t ′ . Suppose that π is such that Pr( ⟨ s t ′ , q t ′ ⟩ | = ♢ ≤ H accept ) > p 1 . Again recall that 1182 for each path ρ ∈ S ω and corresponding trace ( ρ ) ∈ Σ ω such that trace ( ρ ) | = ♢ ≤ H accept the sum 1183 ∑ t ′ + H t = t ′ C ( ⟨ s t , q t ⟩ ) ≥ 1 , and so, 1184

Now clearly for all p 1 ∈ [0, 1] and c > 0 , the following holds, 1185

This implies that there may exist some π satisfying Eq. 104 and such that Pr( ⟨ s t ′ , q t ′ ⟩ | = 1186 ♢ ≤ H accept ) > p 1 , i.e. does not satisfy Eq. 105 at timestep t = t ′ . 1187

G.1.3 Instantaneous constraint 1188

Problem 4.6 (restated) (Instantaneous constraint [23, 60, 69]) .

Proposition G.6. A feasible policy π for Problem 4.6 with threshold d 3 < 1 (otherwise the problem is trivial) is a feasible policy for Problem 4.1 if and only if p 1 = 0 .

Proof. We start by proving the 4.6 ⇒ 4.1 direction. A feasible policy π for Problem 4.6 with d 3 < 1 satisfies,

Then if for all t ∈ [0, T ] , accept ̸∈ L ′ ( ⟨ s t , q t ⟩ ) then we have Pr( ⟨ s 0 , q 0 ⟩ ̸| = ♢ accept ) = 1 , where q 0 = ∆( Q 0 , L ( s 0 )) and by extension we have Pr( ⟨ s t , q t ⟩ ̸| = ♢ accept ≤ H ) = 1 for all t ∈ [0, T ] . This completes the proof of this direction.

Now we prove the 4.1 ⇒ 4.6 direction. A policy π satisfying Pr( ⟨ s t , q t ⟩ | = ♢ accept ≤ H )) = 0 for all t ∈ [0, T ] implies that Pr( ⟨ s t , q t ⟩ ̸| = ♢ accept ≤ H ) = 1 for all t ∈ [0, T ] which implies the following,

1190

1191

1192

1193

1194

1195

1196

1197

1198

1199

1200

1201

1202

1203

1204

1205

1206

1207

1208

1209

1210

1211

1212

1213

1214

1215

1216

1217

which implies that, and by Defn. 4.3,

and by Defn. 4.3, which implies that,

which concludes the proof.

G.2 Infinite Horizon

While in this paper we only consider finite horizon problems with a fixed episode length T , we note that we can also make a set of similar statements for the infinite horizon (discounted) setting. In this section we provide the corresponding statements and proofs for the infinite horizon setting. Firstly, we consider the following infinite horizon problem.

Problem G.7 (Step-wise bounded regular safety property constraint) . Let P safe be a regular safety property, D be the DFA such that L ( D ) = BadPref ( P safe ) and M be the MDP;

where all probability is taken under the product Markov chain M π ⊗D , p 1 ∈ [0, 1] is a probability threshold H is the model checking horizon .

G.2.1 Expected Cumulative Constraint

Problem G.8 (Expected cumulative constraint) .

where d 1 ∈ R + is the cost threshold and γ ∈ [0, 1) is the discount factor.

Proposition G.9. A feasible policy π for Problem G.7 with parameters p 1 ∈ [0, 1] , is also a feasible policy for Problem G.8 with parameter d 1 ∈ R + , provided that d 1 ≥ T · p 1 , where T = 1 / (1 -γ ) is the effective horizon.

Proof. For t = 0, 1, 2, . . . we define, the following random variables, X 0 , X 1 , X 2 , . . . , where, 1218

The argument for this is straightforward. If at every timestep t = 0, 1, 2, . . . we have Pr( ⟨ s t , q t ⟩ | = 1220 ♢ ≤ H accept ) ≤ p 1 then with probability ≤ p 1 we have accept ∈ L ( ⟨ s t , q t ⟩ ) . Let T = 1 / (1 -γ ) 1221 be the effective horizon, then under mild assumptions (i.e. C ( ⟨ s t , q t ⟩ ) < ∞ ) we can consider the 1222 following decomposition of the expected cumulative cost, 1223

We replace the subscript ' ⟨ s t , q t ⟩ ∼ M π ⊗D ' here for brevity. Clearly by linearty of expectations 1224 this statement holds. Although it is worth noting that each expectation is taken under a different 1225 marginal state distribution (i.e. P t ( · ) ), which depends on π (apart from the initial state distribution 1226 P 0 ( · ) ). From now on we will write this is implicitly (i.e. Eq. 121), rather than writing the marginal 1227 state distribution (at time t ) for each expectation. Using our earlier observations we can now bound 1228 the expected cumulative cost from above as follows, 1229

1230

1231

1232

1233

1234

where, 1219

Proposition G.10. The converse is not strictly true, since there may be a feasible policy π for Problem G.8 with threshold d 1 ≤ T · p 1 which does not satisfy the constraints of Problem G.7

We want to prove the following statement, a policy π satisfying,

Proof. To prove this we will show that there may be some policy π that satisfies Eq. 125, but does 1235 not satisfy Eq. 126 at some timestep t . For simplicity we consider the first timestep (i.e. t = 0 ). First 1236 we assume π is such that Eq. 125 holds, then clearly we have, 1237

Let Pr( ⟨ s 0 , q 0 ⟩ | = ♢ ≤ H accept ) denote the proportion of accepting paths from the initial state s 0 ∼ 1238 P 0 ( · ) . Suppose π is such that Pr( ⟨ s 0 , q 0 ⟩ | = ♢ ≤ H accept ) > p 1 . We note that for each path ρ ∈ S ω 1239

does not imply that,

and corresponding trace ( ρ ) ∈ Σ ω such that trace ( ρ ) | = ♢ ≤ H accept the sum ∑ H t =0 γ t C ( ⟨ s t , q t ⟩ ) ≥ 1240 γ H , and so, 1241

1242

1243

1244

1245

1246

1247

1248

Now clearly for all p 1 ∈ [0, 1] , γ ∈ [0, 1) , H ∈ Z + and T = 1 / (1 -γ ) the following holds,

This implies that there may exist some π satisfying Eq. 125 and such that Pr( ⟨ s 0 , q 0 ⟩ | = ♢ ≤ H accept ) > p 1 , i.e. does not satisfy Eq. 126 at timestep t = 0 .

Proposition G.11. A feasible policy π for Problem 4.4 with threshold d 1 ≤ p 1 · γ T + H satisfies Pr( ⟨ s t , q t ⟩ | = ♢ ≤ H accept ) ≤ p 1 up to the effective horizon T = 1 / (1 -γ ) . This bound is tight.

Proof. Let T = 1 / (1 -γ ) be the effective horizon. A feasible policy π for Problem 4.4 with threshold d 1 ≤ p 1 · γ T + H clearly satisfies,

which implies that for all t ′ ∈ [0, T ] we have, 1249

Let Pr( ⟨ s t ′ , q t ′ ⟩ | = ♢ ≤ H accept ) denote the proportion of accepting paths at timestep t ′ , where 1250 s t ′ ∼ P t ′ ( · ) . Here P t ′ ( · ) denotes the marginal state distribution at time t ′ . Recall that for 1251 each path ρ ∈ S ω and corresponding trace ( ρ ) ∈ Σ ω such that trace ( ρ ) | = ♢ ≤ H accept the sum 1252 ∑ t ′ + H t = t ′ γ t -t ′ C ( ⟨ s t , q t ⟩ ) ≥ γ H . Without loss of generality fix some t ′ ∈ [0, T ] and suppose that 1253 Pr( ⟨ s t ′ , q t ′ ⟩ | = ♢ ≤ H accept ) > p 1 . This implies that, 1254

Which is a contradiction. Therefore, it must be the case that when Eq. 130 is satisfied then so is 1255 Pr( ⟨ s t , q t ⟩ | = ♢ ≤ H accept ]) ≤ p 1 for all t ∈ [0, T ] . To prove that this bound is tight we can again 1256 show the possible existence of a counter example. In particular, we want to prove the following 1257 statement, a policy π satisfying, 1258

for some constant c > 0 , does not imply that, 1259

We will show that there may exist some policy π that satisfies Eq. 136 but does not satisfy Eq. 137 at 1260 some timestep t . For simplicity we consider timestep t = T , although we note that with a little extra 1261

work we could come up with a proof for any t ∈ [0, T ] . Firstly, we assume π is such that Eq. 136 1262 holds, then we have, 1263

Let Pr( ⟨ s T , q T ⟩ | = ♢ ≤ H accept ) denote the proportion of accepting paths at timestep T . Suppose π 1264 is such that Pr( ⟨ s T , q T ⟩ | = ♢ ≤ H accept ) > p 1 . We note that for each path ρ ∈ S ω and corresponding 1265 trace ( ρ ) ∈ Σ ω such that trace ( ρ ) | = ♢ ≤ H accept the sum ∑ T + H t = T γ t C ( ⟨ s t , q t ⟩ ) ≥ γ T + H , and so, 1266

Now clearly for all p 1 ∈ [0, 1] , γ ∈ [0, 1) , c > 0 , H ∈ Z + and T = 1 / (1 -γ ) , the following holds, 1267

This implies that there may exist some π satisfying Eq. 136 and such that Pr( ⟨ s T , q T ⟩ | = ♢ ≤ H accept ) > p 1 , i.e. does not satisfy Eq. 137 at timestep t = T .

G.3 Probabilistic Cumulative Constraint

Problem G.12 (Probabilistic cumulative constraint) .

where d 2 ∈ R + is the cost threshold, δ 2 is a tolerance parameter and γ ∈ [0, 1) is the discount factor.

Proposition G.13. A feasible policy π for Problem G.7 with parameters p 1 ∈ [0, 1] , is also a feasible policy for Problem G.12 with parameters d 2 ∈ R + and δ 2 ∈ (0, 1] , provided that, d 2 ≥ √ ( ⌈ log( T ) ⌉ · T ) / 2 · log(1 /δ 2 ) + ⌈ log( T ) ⌉· T · p 1 +1 , where T = 1 / (1 -γ ) is the effective horizon.

Proof. Again t = 0, 1, 2, . . . we define the following random variables, X 0 , X 1 , X 2 , . . . , where,

and we make the following observation,

See the proof of Prop. G.9, the argument is identical. Under mild assumptions (i.e. C ( ⟨ s t , q t ⟩ ) < ∞ ) 1279 we consider the following decomposition of the (undiscounted) expected cumulative cost up to 1280 timestep ⌈ log( T ) ⌉ · T -1 , 1281





⌈

log(

T

)

⌉·

∑

t

=0

T

-

1





1268

1269

1270

1271

1272

1273

1274

1275

1276

1277

1278

E

π

C

(

⟨

s

t

,

q

t

⟩

)

=

E

π

[

X

0

] +

E

π

[

X

1

] +

. . .

+

E

π

[

X

⌈

log(

T

)

⌉·

T

-

1

]

(147)

Again we replace the subscript ' ⟨ s t , q t ⟩ ∼ M π ⊗D ' here for brevity, see the proof of Prop. G.9 for 1282 more details. Before we proceed we must first deal with the dependence between the random variables 1283 X 0 , . . . X ⌈ log( T ) ⌉· T -1 . Strictly speaking it is not the case that Pr( X t = 1 | X t -1 , . . . , X 0 ) = 1284

Pr( X t = 1) . However, we have already established that Pr( X t = 1) ≤ p 1 , as such we can simulate 1285 X 0 , . . . , X ⌈ log( T ) ⌉· T -1 as a sequence of independent coin flips Y 0 , . . . , Y ⌈ log( T ) ⌉· T -1 with probability 1286 p 1 , it is then the case that P [ ∑ ⌈ log( T ) ⌉· T -1 t =0 X t > d 2 ] ≤ P [ ∑ ⌈ log( T ) ⌉· T -1 t =0 Y t > d 2 ] . Now we can 1287 bound the probability that we care about, 1288

Here the first inequality (Eq. 152) comes from the following two facts, certainly 1289 ∑ ⌈ log( T ) ⌉· T -1 t =0 γ t X t ≤ ∑ ⌈ log( T ) ⌉· T -1 t =0 X t and we have that ∑ ∞ t = ⌈ log( T ) ⌉· T γ t X t ≤ 1 . The sec1290 ond fact is a little harder to see, first we note that lim γ → 1 γ T = 1 /e , where T = 1 / (1 -γ ) is the 1291 effective horizon. Then we can rewrite, 1292

The second inequality (Eq. 153) comes from our earlier construction. The final inequality (Eq. 156) 1293 is obtained from Hoeffding's inequality [40] for bounded random variables. Finally, by bounding the 1294 final expression (Eq. 157) from above by δ 2 and rearranging gives the desired result. 1295

Proposition G.14. A feasible policy π for Problem G.12 with parameters δ 2 ≤ p 1 and d 2 < γ T + H , 1296 satisfies Pr( ⟨ s t , q t ⟩ | = ♢ ≤ H accept ) ≤ p 1 up to the effective horizon T = 1 / (1 -γ ) . This bound is 1297 tight. 1298

Proof. A feasible policy π for Problem G.12 with parameters δ 2 ≤ p 1 and d 2 < γ T + H clearly 1299 implies that, 1300

and certainly for all t ′ ∈ [0, T ] we have that, 1301

Let Pr( ⟨ s t ′ , q t ′ ⟩ | = ♢ ≤ H accept ) denote the proportion of accepting paths at timestep t ′ , where 1302 s t ′ ∼ P t ′ ( · ) . Here P t ′ ( · ) denotes the marginal state distribution at time t ′ . Recall that for 1303 each path ρ ∈ S ω and corresponding trace ( ρ ) ∈ Σ ω such that trace ( ρ ) | = ♢ ≤ H accept the sum 1304 ∑ t ′ + H t = t ′ γ t -t ′ C ( ⟨ s t , q t ⟩ ) ≥ γ H . Without loss of generality fix some t ′ ∈ [0, T ] and suppose that 1305 Pr( ⟨ s t ′ , q t ′ ⟩ | = ♢ ≤ H accept ) > p 1 . This implies that, 1306

Which is a contradiction. Therefore, it must be the case that when Eq. 161 is satisfied then so is 1307 Pr( ⟨ s t , q t ⟩ | = ♢ ≤ H accept ]) ≤ p 1 for all t ∈ [0, T ] . To prove that this bound is tight we can show 1308 the possible existence of a counter example. In particular, we want to prove the following statement, 1309 a policy π satisfying, 1310

for some constant c > 0 does not imply that, 1311

We will show that there may exist some policy π that satisfies Eq. 168 but does not satisfy Eq. 169 at 1312 some timestep t . Firstly, we assume π is such that Eq. 168 holds, this implies that for all t ′ ∈ [0, T ] 1313 we have, 1314

Fix some t ′ ∈ [0, T ] and let Pr( ⟨ s t ′ , q t ′ ⟩ | = ♢ ≤ H accept ) denote the proportion of accepting paths 1315 at timestep t ′ . Suppose that π is such that Pr( ⟨ s t ′ , q t ′ ⟩ | = ♢ ≤ H accept ) > p 1 . Again recall that 1316 for each path ρ ∈ S ω and corresponding trace ( ρ ) ∈ Σ ω such that trace ( ρ ) | = ♢ ≤ H accept the sum 1317 ∑ t ′ + H t = t ′ γ t -t ′ C ( ⟨ s t , q t ⟩ ) ≥ γ H , and so, 1318

Now clearly for all p 1 ∈ [0, 1] , and c > 0 , the following holds, 1319

This implies that there may exist some π satisfying Eq. 168 such that Pr( ⟨ s t ′ , q t ′ ⟩ | = ♢ ≤ H accept ) > 1320 p 1 , i.e. does not satisfy Eq. 169 at timestep t = t ′ . 1321

1322

1323

1324

1325

1326

1327

1328

1329

1330

1331

1332

1333

1334

1335

1336

1337

1338

1339

1340

1341

1342

1343

1344

1345

1346

1347

1348

1349

1350

1351

1352

1353

1354

1355

1356

1357

1358

1359

1360

1361

1362

1363

1364

1365

1366

1367

1368

1369

1370

1371

1372

1373

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The proof of safety guarantees (Theorem 6.5) is provided in Appendix C.5. Furthermore, we provide experimental results demonstrating the scalability of our approach in Section 7.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations of our framework are discussed under in Section 7 in the paragraph Separating Reward and Safety , furthermore the limitations of our proposed method are also discussed and explored in Appendix F.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

1374

1375

1376

1377

1378

1379

1380

1381

1382

1383

1384

1385

1386

1387

1388

1389

1390

1391

1392

1393

1394

1395

1396

1397

1398

1399

1400

1401

1402

1403

1404

1405

1406

1407

1408

1409

1410

1411

1412

1413

1414

1415

1416

1417

1418

1419

1420

1421

1422

1423

1424

1425

1426

1427

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: For the key proofs in the main paper we explicitly provide the assumptions used, the proofs of each theoretical result from the main paper can also be found in Appendix C.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: other than access to code, we provide pseudo-code for the algorithms used in our experiments (see Appendix A)

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in

1428

1429

1430

1431

1432

1433

1434

1435

1436

1437

1438

1439

1440

1441

1442

1443

1444

1445

1446

1447

1448

1449

1450

1451

1452

1453

1454

1455

1456

1457

1458

1459

1460

1461

1462

1463

1464

1465

1466

1467

1468

1469

1470

1471

1472

1473

1474

1475

1476

1477

1478

1479

some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide access to the code for our first set of experiments in the supplementary material, with a corresponding script to reproduce the results in the main paper. For the second set of experiments we provide directions to the code base that we adapted and throughout the paper and appendices we provide sufficient details to reproduce these results without too much difficulty.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide a thorough description of the environmental settings in Appendix D.1 and D.2, furthermore, hyperparameters and details with regards to access to the code are provided in Appendix E.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes] ,

Justification: We provide error bars for all of our experiments, over 5 random initializations (seeds), provided by seaborn.lineplot , see Appendix E for details.

1480

1481

1482

1483

1484

1485

1486

1487

1488

1489

1490

1491

1492

1493

1494

1495

1496

1497

1498

1499

1500

1501

1502

1503

1504

1505

1506

1507

1508

1509

1510

1511

1512

1513

1514

1515

1516

1517

1518

1519

1520

1521

1522

1523

1524

1525

1526

1527

1528

1529

1530

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide details the compute resources used for our experiments in Appendix E and the expected time and memory requirements for an individual run.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have reviewed the NeurIPS Code of Ethics and we do not think the research in this paper has any particular ethical concerns, to the best of our ability we have tried to maintain anonymity.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

1531

1532

1533

1534

1535

1536

1537

1538

1539

1540

1541

1542

1543

1544

1545

1546

1547

1548

1549

1550

1551

1552

1553

1554

1555

1556

1557

1558

1559

1560

1561

1562

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA] .

Justification: As our paper is mostly foundational we do not foresee any immediate positive or negative societal impact of this research.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA] . 1563

1564

1565

Justification: We do not believe that the new assets provided in the paper pose any such risks.

1566

1567

1568

1569

1570

1571

1572

1573

1574

1575

1576

1577

1578

1579

1580

1581

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

1582

1583

1584

1585

1586

1587

1588

1589

1590

1591

1592

1593

1594

1595

1596

1597

1598

1599

1600

1601

1602

1603

1604

1605

1606

1607

1608

1609

1610

1611

1612

1613

1614

1615

1616

1617

1618

1619

1620

1621

1622

1623

1624

1625

1626

1627

1628

1629

1630

1631

1632

Justification: We are the original creators/owners of the code used for the first set of experiments, for the second set of experiments we explicitly cite the paper and provide the URL for the code that we have adapted in this paper, which is available under the MIT License as stated in Appendix E.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We provide

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA] .

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an online shielding approach for safe RL that does not assume prior knowledge of environment dynamics and utilizes finite-horizon model checking with learned approximations of the environment dynamics. It specifically focuses on RL with regular safety properties provided as a PCTL formula. The authors present a framework that dynamically identifies unsafe actions and deploys a safe backup policy when necessary. The main technical contributions of the paper are:\n\n- Definition of a constrained RL problem based on regular safety properties.\n- Presentation of model checking algorithms to verify finite-horizon satisfaction probability.\n- Development of sample complexity results for statistical model checking procedures.\n\nThe novelty of the paper lies in its approach to reinforcement learning with regular safety properties without requiring prior knowledge of environment dynamics. Unlike traditional shielding approaches that need full environment models or simulators, this framework uses learned approximations and finite-horizon model checking. The authors represent the synthesis problem as finding an optimal policy under a constraint that the resultant product Markov chain (from the policy) and the DFA (from the safety property) has probability $\\leq p_1$ of violating the safety property in finite horizon $H$. The authors make use of the CMDP formulation to separate the reward and safety considerations by treating the safety property as a cumulative constraint in the CMDP.\n\nSTRENGTHS:\nOverall, this is a very strong submission and very clearly written. The authors present their problem statement precisely and also compare against many other related settings. This is challenging to do in safe RL since it is such a wide field with many parallel approaches, but I feel the authors did a commendable job here, especially in demonstrating how related and alternative formulations can be represented in their setting. \n\nThis is a very interesting combination of shielding from temporal logic specifications with a CMDP formulation. Especially, as the framework allows for regular safety properties instead of just invariant properties, it allows for quite general specifications. \n\nThe authors also do a good job in showing the generality of the work as it pertains to levels of model knowledge.\n\nWEAKNESSES:\nThe main weakness I can see in this paper is perhaps a lack of experiments in settings with more complex models and safety specifications. Especially settings in which safety and optimality of the reward are in conflict, it would be interesting to see how the agent is able to achieve a trade off. \n\nAnother consideration is perhaps motivation of using a CMDP as a framework for safe RL. I don't see this as well motivated in the paper. There are several approaches to safe RL that use MDPs and are able to use probabilistic model checking type techniques to give guarantees for cost thresholds.\n\nQUESTIONS:\nIs there any similarities in the approach from the literature in shielding for POMDPs? E.g.,\"Safe Reinforcement Learning via Shielding for POMDPs\" by Carr et. al. Especially as one of the contributions in this paper is that the authors assume no knowledge of the dynamics. While the setting is not quite the same, I think there may be overlaps in the concept on how Carr et al, are able to shielding guarantees on a partial model\n\nTypos\nline 188 - \" The hyperparameter p1 is be direct\" grammar issue\n\nlines 370, 408, 952, 983 - \"loose\" -> \"lose\"\n\n[45,46] are the same reference.\n\nLIMITATIONS:\nThere is some discussion of limitations in the section before the conclusion. I agree with the authors on the downsides of separating reward and safety.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a new safe RL approach, building on safety shields.\nThe idea is to leverage model-checking techniques during the RL training to block actions that are identified as unsafe in the shield and use a learned backup policy if this is the case. In contrast to previous approaches, the \"meta-algorithm\" presented does not require an a-priori known model of the safety aspects of the environment. The approach comes with theoretical guarantees on the satisfaction of finite-horizon PCTL specifications and the evaluation assesses the potential of the proposed algorithm.\n\nSTRENGTHS:\nI'd first like to pinpoint that the paper is quite clear and very well written.\n\nThe proposed approach has several key advantages. First, it doesn't necessarily require providing a shield beforehand, in contrast to previous work. Second, the properties the agent needs to enforce during training are specified via PCTL, a well-established specification formalism that is not prone to exponential blow-up in the size of the automaton for translating the formula (as is the case for LTL). Third, the approach comes with guarantees on the shield: (i) when the hyper-parameters of the optimization procedure are well-chose, one can bound the probability of failure of the system from the initial state, (ii) the optimal policy found under the PCTL constraint is ensured to be a feasible policy for standard constrained MDP objectives. Finally, the authors discuss and provide guarantees under different assumptions, namely, the access to a model of the safety-relevant aspect of the environment (as in previous work), under a black-box model, and when one has access to an approximate model such that the total variation between the true and approximate transition probabilities is bounded. Additional statistical guarantees are provided for these last two assumptions. \n\nExperiments successfully highlight the potential of the approach.\n\nWEAKNESSES:\n**Beyond tabular settings.**\nThe main concern I have with this paper is the fact that the guarantees seem to solely hold in the tabular setting, i.e., when the state-action space is finite and tractable. Notably, in the second round of experiments, the authors use Dreamer-v3 to learn an approximate model of the environment. Although the resulting method seems to outperform constrained RL methods, the theoretical guarantees do not hold in practice. \n\n**On the assumptions.**\nAssumption 5.2 is confusing. Indeed, when reading it for the first time, I thought one needs to have access to a *generative model*, i.e., a black box model of $\\mathcal{P}$ that can be requested at any time, under any state and action, akin to the setting of [1, 2]. Specifically, this would mean that one doesn't necessarily need to sequentially execute the environment to obtain samples (as this is the case in RL), but, at any time, for any given state-action pair $(s, a)$, one could request the model to obtain a finite number of samples from $\\mathcal{P}(\\cdot \\mid s, a)$. Note that this is not compliant with RL. When reading further, it seems that Monte-Carlo model checking only needs to produce episodes, which is fully compliant with RL. Thus, the distinction is really important here.\n\nOn another point, Assumption 5.3 looks rather restrictive. How to ensure that the approximate model learned through Algorithm 1 (line 300) yields a bounded total variation? This should be discussed in the main text. Moreover, the linked guarantees (Proposition 5.5) are not evaluated in the experiments. It could be interesting to have an example of how the statistical guarantees can be applied in practice.\n\n[1] Michael J. Kearns, Yishay Mansour, Andrew Y. Ng: A Sparse Sampling Algorithm for Near-Optimal Planning in Large Markov Decision Processes. Mach. Learn. 49(2-3): 193-208 (2002)\\\n[2] Yujia Jin, Aaron Sidford: Towards Tight Bounds on the Sample Complexity of Average-reward MDPs. ICML 2021: 5055-5064\n\nQUESTIONS:\n- How do your guarantees apply when using dreamer as an approximate model of the environment? I guess the guarantees vanish here, and if it is the case, under which assumption Proposition 5.5 can be extended to still provide some guarantees? \n- Could you clarify Assumption 5.2 (see the related point in the weaknesses).\n- About Assumption 5.3, the total variation distance seems difficult (or even impossible) to formally check. Could you discuss this further?\n- Line 110: you require the reward function to be bounded in $[0, 1]$. What is the impact of this assumption?\n\n**Other minor remarks.**\n- I think there is a slight bug in Definition 3.3; $trace(\\rho)$ yields a word of infinite length while $\\mathcal{L}(\\mathcal{D})$ consists of words of finite length. So $trace(\\rho)$ should never belong to  $\\mathcal{L}(\\mathcal{D})$. \n- line 169: $\\mathbb{Z}_{+}$: why not simply write $\\mathbb{N}$?\n- line 204: aren't you this way defining an $\\omega$-automaton?\n- line 249: I don't see why $H$ should be small. Most model-checkers rely on value-iteration-based engines with very large horizon, so I think $H$ shouldn't be a problem here, but rather the size of the formula and the size of the state space. \n- line 289 in Algorithm 1: I guess the probability in the **if** statement is computed through $\\hat{\\mathcal{P}}$. This should be explicitly indicated.\n- line 326: you first use $\\text{Pr}^{min}$ and then $\\inf_{\\pi} \\text{Pr}$. Doesn't the $\\inf$ coincide with the $\\min$ here? I found that formulation a bit confusing. \n- In Table 1, invariant properties are unbounded. Isn't that disallowed in your approach?\n- line 360: to me, the usage of PCTL* is not a slight detail; it embeds LTL that is quite different from PCTL; notably model checking algorithms are not polynomial in the size of the formula (but in the worst case doubly exponential).\n\nLIMITATIONS:\nApart from the points raised above, the limitations of the work have been successfully addressed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-10"}

comments: SUMMARY:
This paper studies RL with 'regular' safety properties. The constraint of safe RL is based on the satisfaction of a logic formula in probability. The action from the 'backup' policy will proactively override the potentially unsafe action from RL to ensure/optimize safety, a typical shielding mechanism in formal safe control methods. The authors demonstrated the effectiveness of their approach against CMDP and regular RL (Q-learning) in two examples.

STRENGTHS:
The approach is sound, as a typical shielding method, should work well in a safe RL setting, in 2 examples shown in the paper. 
The problem studied in this paper is important.

WEAKNESSES:
1. Novelty. Novelty is my biggest concern for this paper. First, Problem 4.1 has already been discussed and solved in [1][2]. Second, the shielding approach is nothing new, a very standard way in formal methods. You can even trace back to simplex architecture with an advanced controller with safety back control. Third, the model-checking approach of this paper is not novel. 
2. Significance. The experiments are weak with only 2 baselines (1 as original RL, 1 as CMDP) on 2 simple examples.

[1] Wang, Yixuan, et al. "Enforcing hard constraints with soft barriers: Safe reinforcement learning in unknown stochastic environments." International Conference on Machine Learning. PMLR, 2023.
[2] Wachi, Akifumi, et al. "Safe exploration in reinforcement learning: A generalized formulation and algorithms." Advances in Neural Information Processing Systems 36 (2024).

You may want to extend the related works part including two papers above and more.

QUESTIONS:
Please check the weakness part above.

LIMITATIONS:
The reviewer would like to know the limitations discussion by the authors in the rebuttal phase.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-10

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an approach to online shielding for reinforcement learning agents. Namely, safety is formulated in probabilistic temporal logic with a parametric threshold as an indicator for reachability of the goal state. The proposed algorithm checks the reachability probability threshold in each state of the environment and raises a warning when the threshold is violated. A pre-trained backup policy is then proposed to be deployed which overrides the action of the agent. The approach is evaluated on tabular and visual RL benchmarks.\n\nSTRENGTHS:\nThe paper addresses an interesting and valuable problem. \n\nEvaluation includes visual RL benchmarks, which are interesting to provide online safety for.\n\nWEAKNESSES:\nThe paper gives a lot of choice to the reader to compose a problem setting of their interest. It does not, however, provide precise enough approach description for each of them. This makes the contributions blurred. Described problem settings have been extensively studied before and the proposed approach does not significantly improve on them. It is also claimed that the approach can be used both during training and deployment, it is, however, not clear if the authors formulate these as two distinct settings and evaluate separately or not. In the latter case, it would be a dangerous simplification. Figures in the evaluation section are not readable.\n\nPresentation: The presentation suffers from imprecise narrative leaving multiple questions until the evaluation section. Assumptions are introduced twice and it is not clear what exact problem the authors propose to address, or to what exact problem setting it generalizes. The use of \"etc.\" and \"some other\" give the impression that more problems can be addressed than presented in the evaluation.\n\nMinor:\n- p.7: \"if need be we\"\n- \"don't\" --> do not\n\nQUESTIONS:\n1. What problem settings and RL controllers does the meta-algorithm apply to precisely?\n2. How exactly is the approximate model updated in case of the breach of probability threshold?\n3. Hoeffding's inequality and Bernstein bounds for Monte-Carlo simulations are known to explode for rare events. How do the authors propose to tackle the challenge of guaranteeing safety for rare violations which are potentially of most interest when it comes to online safety?\n4. When sampling from \"approximate\" model, what are the expected guarantees to be achieved,  i.e., how much is approximation error expected to contribute to the final statistical guarantee?\n5. In evaluation: how does hyperparameter tuning effect the probability threshold in the probabilistic reachability specification?\n\nLIMITATIONS:\nThe approach is of limited novelty and employs existing components. Technical details in terms of the exact problem setting and guarantees are insufficient to judge the contribution. The approach is not yet placed in the larger context of online safety for RL agents, which would require more precise problem formulation and discussion of limitations.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses safety and constraint compliance in deploying reinforcement learning (RL) systems. Such issues have triggered a vast body of research in the area of safe RL over the last few years. The paper introduces a safe RL framework for so-called regular safety properties, focusing on satisfying these properties with high probability. The paper compares and places this setup to common constrained Markov decision processes (CMDP) settings and presents a meta-algorithm with provable safety guarantees to prevent violations of regular safety properties during training and deployment. The approach is evaluated in both tabular and deep RL settings.\n\nSTRENGTHS:\nSafe RL is of utmost importance in order to enable RL agents to be successfully deployed in the real world. This paper tackles an important problem: How to ensure safety if the safety criteria go beyond standard reward or simple reachability features. The motivation of the paper is well done, and the placement in the literature is mostly complete and extensive.\n\nWEAKNESSES:\nWhile this paper addresses an important research question, I feel it is not ready for publication at a major ML conference. I will first summarize the issues that I see and then elaborate in more detail.\n\n1. The claims made in the intro are not properly met. This concerns particularly the claim that the approaches are most permissive in terms of prior knowledge. \n\n2. The evaluation is not sufficient. The experiments are very sparse and miss important information to assess the quality of the research.\n\n3. The paper is packed with formal definitions, most of them being standard in the formal methods community. Conversely, little space is spent on describing the methods in detail and providing a proper evaluation.\n\nI overall feel that the paper aims too high and wants to solve all kinds of aspects in model-based safe RL with shields. It would have been better to pick certain key aspects more clearly. As an example, the paper claims to \n- not need a model\n- to learn a model\n- to use most of the available model checking paradigms (and even introduce them)\n- evaluate with tabular and deep RL\n\nMany of these aspects are then not properly discussed. \n\nI will now comment on the previously listed weaknesses.\n\n1. In the introduction, the authors claim to operate in a most permissive setting where environment dynamics are not known. Yet, then, they introduce three model-checking paradigms, which are essentially standard numerical model checking, Monte Carlo-based statistical model checking, or model checking with approximate models. In particular, the last version is shady. What is a guarantee for a learned model? How can a claim be made for a most permissive setting but still obtain hard guarantees when, as a consequence, all guarantees rely effectively on statistics or a learned model? Other approaches are simply very clear about their assumptions, such as knowing a model. The model learning procedure is standard, and if the whole 'permissiveness' of the setting depends on the fact that a model can learned, I do not see a novel approach here. \n\nVery importantly: While a model is learned, no safety guarantees can be given, defying the notion of shielded RL. This aspect is not discussed in the paper. A step further, even the safe fallback policy that is explained in Section 6 relies on learning. \n\n2. The evaluation only considers a few environments and compares a simple tabular and deel RL agent. What I would have liked to see is how the different assumptions and model checking paradigms affect the learning, the safety, the performance. I believe, with a thorough evaluation in this direction, the paper would be much stronger. If we can believe that it is feasible to learn a model, then I would like to see a comparison between the strong assumption of knowing the model, and having learned a model, and what the effects on safety guarantees are. \n\n3. The contributions only start at page 5, and are then still interleaved with standard definitions. It is important to have a paper safe-contained, but, as an example, why is it necessary to introduce both LTL, PCTL, DFA in detail? And then, the evaluation even uses PCTL^*. To me this seems as if the space had been filled up with long definitions, while it should have been used more on the contributions of the paper. \n\nGenerally, I feel the paper follows a great direction, especially investigating Shielded RL with a learned model. With a stronger evaluation and emphasis of this part, the research and contribution could be much improved in my opinion. \n\nMinor comments:\n\n- proposition 4.2 is obvious and seems like an overformal statement of a simple fact. \n\n- Def 4.3. Make clear that this is reward engineering\n\n- Non-Markovian cost: Compare to reward machines\n\n- The tradeoff between safety and exploration has (for instance) been investigated in \n\nCarr et al.: Safe Reinforcement Learning via Shielding under Partial Observability. AAAI 2023.\n\n- Learning the model, proposition 5.5: I think the topology (graph) of the model needs to be known to estimate the probabilities.\n\nQUESTIONS:\nBeyond the points I raise above, I have the following questions.\n\n1. Please comment on the evaluation and effects of different assumptions on prior knowledge.\n\n2. Do you need to know the topology of the MDP (Markov chain) to effectively learn a model? \n\n3. Can you comment on the limitations of your work and thereby sharpen the contribution?\n\nLIMITATIONS:\nThe limitations of the work in terms of assumptions and their real-world relation are not properly explained in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-19"}