Explainably Safe Reinforcement Learning
Authors: Sabine Rieder
Stefan Pranger
Debraj Chakraborty
Jan Kretinsky
Bettina Könighofer
Venue: NeurIPS 2025 poster ()
Keywords: Reinforcement Learning
Shielding
Decision Tree
Safe AI
Explainability
Primary area: reinforcement_learning
TL;DR: We represent a shield for safe reinforcement learning as a hierarchy of decision trees to explain the rules of the shield.

Abstract: Trust in a decision-making system requires both safety guarantees and the ability to interpret and understand its behavior. This is particularly important for learned systems, whose decision-making processes are often highly opaque. Shielding is a prominent model-based technique for enforcing safety in reinforcement learning. However, because shields are automatically synthesized using rigorous formal methods, their decisions are often similarly difficult for humans to interpret. Recently, decision trees became customary to represent controllers and policies. However, since shields are inherently non-deterministic, their decision tree representations become too large to be explainable in practice. To address this challenge, we propose a novel approach for explainable safe RL that enhances trust by providing human-interpretable explanations of the shield's decisions. Our method represents the shielding policy as a hierarchy of decision trees, offering top-down, case-based explanations. At design time, we use a world model to analyze the safety risks of executing actions in given states. Based on this risk analysis, we construct both the shield and a high-level decision tree that classifies states into risk categories (safe, critical, dangerous, unsafe), providing an initial explanation of why a given situation may be safety-critical. At runtime, we generate localized decision trees that explain which actions are allowed and why others are deemed unsafe. Altogether, our method facilitates the explainability of the safety aspect in the safe-by-shielding reinforcement learning. Our framework requires no additional information beyond what is already used for shielding, incurs minimal overhead, and can be readily integrated into existing shielded RL pipelines. In our experiments, we compute explanations using decision trees that are several orders of magnitude smaller than the original shield.

Explainably Safe Reinforcement Learning
Sabine Rieder*
Stefan Pranger"
Masaryk University
Graz University of Technology
Technical University of Munich
stefan. pra.fa..prangertuurr....m
sabine. riederêmail mwni. cz,
Debraj Chakraborty
Jan Křetínsky
Masaryk University
Masaryk University
chakraborty@fi muni.cs oz
Technical University of Munich
jan kww.insktkeeeeke..kkkkkkkkkkkkke fi muni cz,
Bettina Könighofer
Graz University of Technology
bett vettina.ee koenighoferrtturre.cco
Abstract
Trust in decision-makinn makinn system requires both safety guarantees and the ability to interpret and understand its behavior. This is particularly important for learned systems, whose decision decision-making processes are often highly opaque. Shielding is a prominent model-based technique for enforcing safety in reinforcement learn- ing. However, because shields are automatically synthesized using rigorous for- mal methods, their decisions are often similarly difficult for humans to interpret. Recently, decision trees became customary represent controllers and policies. However, since shields are inherently non-deterministic, their decision tree repre- sentations become too large to be explainable in practice. To address this chal- lenge, we propose novel approach for explainable safe RL that enhances trust by providing human- ciiman-nnterpretbblle explanations of the shield's 's decisions. Our method represents the shielding policy as a hierarchy of decision trees, offering tow-down, case-based explanations. At design time, safety risks of executing actions in given states. Based on this risk analysis, we construct both the shield and high-level decision tree that classifies states into risk categories (safe, critical, dangerous, unsafe), providing an initial explanation of why given situation may be safety-critical. At runtime, we generate localized decision trees that explain which actions are allowed and why others are deemed unsafe. Altogether, method facilitates the explainability of the safety aspect in the safe-by-shieldig ditional information beyond what already used for shielding, incurs minimal overhead. and can be readily integrated into existing shielded RL pipelines. In our experiments, we compute explanations using decision trees that are several orders of magnitude smaller than the original shield.
world model analyze the
reinforcement learning. Our framework requires no ad-
1 Introduction
Deep reinforcement learning (RL) [42] is a powerful machine learning technique for intelligent sequential decision-making. Despite its successes, its application in safety-critcaal systems remains limited due to safety concerns. RL agents learn by exploring their environment through trial and emoor a process that inherently carries the risk of taking unsafe actions.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Safety Explanations during Runtime
L1: Risk per State
L2: Risk of Action per State
13: Consequences Actions
d
5
Px
Po
keep
scfe
d t 3
keep, brake
brake
P.Pa
critical
brake
0.2
0.8
dangerous
unssfe
P.
break
7, Po
Shielded
Explain the risk of keeping the velocity in state (Pa 3. 7)
Shield
Observation
Safe Actions
Environment
RL Agent
Action
Figure Overview of our method for explainable-safi RL.
Shielding [1] is a prominent approach towards safe RL. A shield blocks ("shields") unsafe actions from the learning agent runtime, either during learning or evaluation, as depicted at the bottom of Fig. 1. At each step, the shield provides a list of all safe actions, from which the agent can select one. Shielded RL thus belongs to class of methods that combine symbolic AI, which provides formal safety guarantees but suffers from limited scalability, with machine learning, which offers high scalability but lacks guarantees. However, even though shields add safety guarantees to RL, trusting a shielded system remains challenging due to the lack of explainability.
Policies learned via RL are notoriously difficult interpret, as they are represented by highly opaque deep neural networks. Unfortunately, understanding the decisions made by a shield is almost equally challenging. Shields are typically implemented large lookup tables that define the set of allowed actions for each state. These tables are inherently hard for humans to interpret, as they provide no insight into why a particular action considered safe or unsafe given state. The combined opacity of both the shield and the RL policy makes it difficult to predict, understand, or trust the behavior of shielded learning agent.
In this paper, we propose an approach to explainably safe RL by explaining the decisions of the shield to the user, making the safety aspect of the process explainable.
Decision Trees (DTs) have recently gained popularity for representing controllers and policies due to their inherent simplicity and human-readability [17]. This is documented by numerous case studies and tool support, e.g. [5, 4, 7, 12). However, directly computing DTss to represent a shield often re- sults in excessively large trees, undermining their utility for explainability. Computing compact DTs especially challenging in the context of shielding due to the shield's 's inherent non-determinism: a shield permits the agent to explore any action that safe, resulting in more complex representations.
Our Approach for Explainable-Safe RL. Rather than presenting the user with a large tree that explains all safety-critical aspects at once, we propose provide explanations in a hierarchical manner. Our method offers a compact, top-down, case-based explanation. An overview of our method is shown in Fig. 1. First, our approach computes the shield along with its explanations to help users understand how critical the current state and the risks associated with exccuting specific actions. Second, the risk of each state is explained using a decision tree constructed at design time. Third, at runtime, additional explanations identify which actions are safe and clarify why others considered unsafe. Our framework takes input abstract world model M in the form of Markov Decision Process (MDP), along with a safety specification oo in temporal logic.
At Design-Time. Based on the model M, we apply value iteration to compute, for every state- action pair (8, a). the risk of violating the safety specification yp when executing action a in state s. Based on this safety analysis, we categorize the states in M into four risk categories:

---

Safe: From safe states, executing any action carries a low risk of violating safety in the future.
Critical: In critical states, some, but not all, actions carry too-high risk of violating safety.
Dangerous: In dangerous states, all actions carry an unacceptably high risk of violating safety.
Unsafe: States in which safery has already been violated.
Shield Computation. From this safety analysis, we first derive the shield. The task of the shield is to block actions from the RL agent that pose too high risk. Thus, in safe states, the shield allows all actions. In critical states, the shield allows all actions whose risk is below the user-defined threshold. If no absolute-safety guarantees are enforced (i.e., the user sets the threshold to a value less than 1, allowing the agent to take some risks), the agent may end up in dangerous state or even an unsafe one. In dangerous states, all actions induce risk inevitably above the safety threshold, meaning safety is at risk but, for now, has not been violated. In this case, fallback shielding strategy must be defined, such as selecting a predefined action (c.g., braking or landing). We follow a common approach and implement shield that, in such situations, allows only the safest available action [23]. In unsafe states, safety is violated and the shield allows no actions (deadlocks). The resulting shield a non-deterministic policy mapping states to allowed actions.
Level 1: Explaining the Risk of States. The highest-level explanation is represented in the form of decision tree (DT) that categorizes states into the four risk categories (safe, critical, dangerous, and unsafe), helping users to determine whether the system is currently in a safety-critical situation and to assess the severity of the risk. For given state, the predicates along the path in the DT provide an explanation for its categorization. Example: In Fig. 1, a human operator asks for an explanation why the action of keeping the velocity is not allowed in the current state s, where the agent is at position 3 and an object is at position 7. The Level 1 DT explains, that the current state is critical because the distance d to the object is 3m S S 10m. Therefore, not all actions are safe to execute.
At Runtime, The user may wish to understand why certain actions are considered safe or unsafe in the current situation. To explain the safety risk of actions, we compute a decision tree that explains the categorization of actions in the current state. If the user requests more detailed information about why a particular action is classified as unsafe, we compute an execution tree that summarizes the potential consequences of executing that action.
Level 2: Explaining the Risk of Actions. A Level 2 DT explains the shielding policy for the set of states represented by a leaf in the Level DT. If the current state is categorized as critical in Level 1, the Level DT explains which actions pose low safety risk and therefore permitted. If the current state is categorized as dangerous, the Level 2 DT explains what action is the safest one to execute. Example: Continuing the scenario from Fig. 1, the Level 2 DT explains that, in the current state, only breaking is classified as safe action because the distance to the obstacle is less than 5m. If the distance were greater than 5m, maintaining the current velocity would also be considered safe.
Level 3: Explaining the Consequences of Actions. If executing action a in a given state 8 is categorized unsafe, we compute yet another tree that provides evidence for the violation of the property & resulting from exccuting a in s. This final explanation is provided in the form of an Execution Tree (ET), which summarizes traces in the MDP M that start with executing in s, followed by taking only the safest available action thereafter. This tree demonstrates that, even when the safest actions are taken after a, the property y is violated with a probability that exceeds the safety threshold. Example: Returning to the scenario described above, the Level 3 ET demonstrates that, if the speed is maintained in (Pa 3. po, 7). the next state will be (p.. 5, Po 7). From there, even braking would result in an unsafe state (d = 0) within the next two steps.
To compute the trees, the user can optionally provide predicates to guide the DT learning process. Note that the world model M and the safety specification & are required in any shielded RL setting. As a result, our method can be casily integrated into any shielded RL application, as it does not require any additional information beyond what is already used to construct the shield.
Key Contributions of this Paper. (1) To the best of our knowledge, we introduce the first frame- work that combines explainability with formal safety guarantees for RL. (2) We provide a method for generating concise, case-based explanations that enable humans to understand the cause of safety violations as they occur. We overcome the non-scalability of a naive approach to explainability- via-DT by the hierarchical decomposition of explanations. (3) We evaluate our implementation on several challenging RL. benchmarks, showing that the resulting explainable shields are compact and comprehensible, in contrast to traditional shields, which typically involve thousands of states.
3

---

2 Related Work
Shields were initially introduced in the context of reactive systems [10]. Later, [1] extended the con- cept to RL, ensuring absolute safety guarantees. In contrast, [23] adopted quantitative perspective on safety, designing shields that provide probabilistic safety guarantees, thereby permitting the agent to take some calculated risks. These two fundamental concepts of shielded RL assume environments modeled as MDPs with discrete state and action spaces. Many extensions exist [34, 43, 35, 26]. For example, [15] studied shielded RL for partially observable environments where only a part of the state can be observed. Shields for multi-ager systems have been studied, in both centralized and de- centralized settings [18, 29]. Shields for quantitative fairness properties have been studied in [6, 14]. In this work, we make a step toward bridging the gap between formal methods and explainable AI by computing explanations for rigorously computed shields, as introduced by [23].
A comprehensive literature review on explainable RL can be found in [31, 49]. Research directions include human-in-the
approaches [40], policy summarization [2. 44], training process visu- alization [32]. and methods that identify performance-critict states [16, 20, 36]. These methods primarily focus on explaining why the agent's 's decisions are optimal for maximizing the expected
reward. The computation of case-based explanations has been explored in [46, 13]. In contrast to explaining the full decision-make maaingg intent of the RL policy, approach explains why certain actions are considered safety-critical
DT's are widely used in explainable Al due to their intuitive, human-readable structure [17]. In the context of explainable RI,, DT's have been constructed to mimic the trained policy of RI. agents [9, 30, 47, 19, 38]. In contrast, that specifically focus on the safety aspects of the problem Furthermore, our approach preserves both optimality and scalability by retaining the RL agent for decision-making Recently, several works have explored the computation of DTs to represent optimal policies in MDPs [25, 3, 48]. However, these approaches focus explaining the available actions, not the safety aspects, and are often not scalable to the large environments and non-deterministic policies of the RL context.
compute DTs explain the shield, resulting in compact explanations
Background
Markov Decision Processes. Markov decision process (MDP) [42] is a tuple M (S, so. A, A. where S is a finite set of states, E S is the initial state, .A is finite set of actions, and P the probabilistic transition function. For all s E S, the available actions are A(s) {a E A 3s', (s", & 0} and we assume set of features. state s E S can be represented as a tuple s = (VI..... Un) where iy = s(U)) E Z is the assigned value to feature U/.
S x A + Dist(S)
[A(s)] > 1. Let V {VI, ... Un be
Choices in an MDP are resolved via policies. A (memoryless non-randomizing) non-deterministic policy is a relation S + 24. A (memoryless non- non-randomizing) deterministic policy is a function : S + A. Applying a deterministic policy : S + A to an MDP M induces a Markov chain (MC) M = (S, so P) with P S + (ist(S) where all nondeterminism is resolved. A finite trace = { S Sn } in MC is sequences of states such that P(s 8(41) > The probability mass of such trace is defined as P (SuuS122...., = Ilostnn P(st, $1+1).
Probabilistic Model Checking. We consider safety propertics expressed in Computation Tree Logic (CTL) [8]. Informally, safety property specifies that 'something bad never happens'. Such properties can represent invariants like "collisions are never allowed' as well as temporal properties like: 'The agent must not reach unsafe position within 30 steps.' Probabilistic model check- ing [8] computes the probability of satisfying a safety property & over a finite or infinite horizon, using adaptations of value iteration, policy iteration, linear programming. We define the prop- erties below with a bound n. For the unbounded horizon, n co. For given MDP M, and a property y in CTL, model checking computes the following probabilities:
P :p Sx N -> [0, 1] is the expected probability to satisfy & in the MC M from a state E S within steps for deterministic policy π. pmin (s, (s, h) min, P. "PMrrp(s h) is the minimal expected probability in the MDP M over all policies from a state s within h steps.

---

Safe Reglon
Region 3a: risk(s, s
Dangerous Region
va: (s,
ri:kisk(s,
Unsafe Reglon:
Figure 2: Risk-based Categorization States.
Shields. A shield Wahield S 2 for an MDP M can be represented a non-deterministic policy. Shields are computed from safety specification environment that captures the safety-relevant aspects of the full MDP [23]). Computing the shield using only safety-relevant fragment of the full MDP enables the scalability of shielded RL. Dif- ferent types of shields are distinguished by the safety guarantees they provide and the types of used world models. We follow the approach of Jansen et [l3] and compute shields that offer proba- bilistic safety guarantees. We discuss the details in Section 4.
in CTL and an abstract model of the
Reinforcement Learning. In reinforcement learning (RL) [42], an agent learns a task via interac- tions with unknown environment modeled by an MDP M with associated reward function R S + R. In each state S, the agent chooses an action E A. The environment then moves state s' with probability P(s, a, s'). The return zet, of an execution p the discounted cumu- lative reward defined by ret,- = = (Rts), using the discount factor E [0, 1]. The objective of the agent learn an optimal policy π + A that maximizes the expectation of the return.
Decision Trees. A DT over the set of features V is a tuple T (Tr, A, where Tr is finite, rooted, binary, ordered tree consisting of a set of inner nodes and a set of leaves L.
We use decision trees (DTs) [11] to represent nondeterministic policies in MDPs.
The set of predicates over V is denoted by r. We define set of basic predicates I [vi const], where U E V, const E Z, and ~E (S, "5,,,,,,=)) The function p N assigns to every inner node N predicate
I. The set A specifies set of labels. The labeling function X L + A
assigns to each Icaf E L, label 7 E
A DT T defines function f 2d + Z, with d = [V]. as follows. Given an input vector (a state in the MDP)
(V),- Va) E Zá, one follows a unique path from the root to a leaf E L s.t. for each inner node E N on the path, the predicate p(n) evaluates to under the substitution गे iff the first (typically left) child of n lies on p. We define T(x) = to be the leaf the state reaches. The value of the function on s is defined f(s) = f(TI...... ... Un) X(1). We define 1, L - 2 with S zd the function that maps leaf to the set of states whose path ends in I.
Execution Trees. For given MC M (S, sp, P) and set of finite traces II { ... nn of M, an execution free (ET) represents ..... of M in a tree structure. An execution tree T (N, E) a rooted tree consisting of a set of nodes N and set of edges E N + N. Nodes are labeled with states p in T. Thus, each path = (n1, 72,... ... n, corresponds to a trace т, = {31. $2.... sn E such that the node n; is labeled with the state s; for <ism.
S. An edge E corresponds to transition in P. Each trace II defines path
4 Computing Hierarchical Safety Explanations
In this section, we present our algorithm for explainable safe RL detail. We begin by describing the construction of the shield and the Level 1 DT in Sec. 4.1. In Sec. 4.2, introduce the Level 2 and Level trees, which provide case-based explanations at runtime. To compute the shield and the explanations, our algorithm takes the safety specification &p and an MDP M A, A,P). Optionally, accepts user-defined predicates to guide the DT learning.
4.1 Shield Computation and High-Level Safety Explanations
Executing an action in a given state may carry some risk of violating safety at some point in the future. Based on this risk, states are categorized as safe, critical, dangerous, or unsafe.
5

---

Definition 1. (Risk of Safery Violation) Given an MDP M = (S, s0, A.P), and a finite horizon h, we define the risk of violating vp from state s E S in the next h steps as function risk M.p: S x N - [0, 1] as follows:
A.P)), a safety property i,
4s E S: risk MiskM,psssh = Pin M (s.
The risk of violating v from a state E S after executing action a E A in the next h steps is 1] as follows:
defined via the function risk M.: S x A x
Ys EsES,Va va E A: risk (s, a. h) = (P(s, a,s, risk h 1)).
s'ES
The risk of a state s is defined as the minimal expected probability of reaching a state that violates ip within the next h steps, quantified over all policies; that is. the probability of a safety violation under the safest available policy. The risk of executing an action a in a given state s is the accumulated, weighted risk of the successor states reached by executing in s. The risk across all states in the state space can be computed using standard probabilistic model checking algorithms, such as value iteration dynamic programming [8], with tools like PRISM [27], STORM [22], or TEMPEST [34]. Definition 2. (Safety of Actions) Given an MDP M. a safety property yp a finite horizon h, and a user-defined safety threshold [0, 1]. For a given state action pair (s, a) S x A, action is called safe in 8 if M. a, h) S c. Otherwise, a is unsafe in 8.
We partition the state space into S = S, US. U Sa U U., according to the risk per state as follows: Definition 3. (Safe States S.) Vs E S. s S, iff Va A: risk a,) S. Definition 4. (Critical States S.) Vs E S. s E S. iff Ea E A: risk (s,,,,) S and
3a E A: risk M.p a', h)) >6
Definition 5. (Dangerous States Sa) E S. s SE Sa iff F and Va E A: M as, > Definition 6. (Unsafe States S.,) VS E S. Sa iff H.p
The categorization of states is illustrated in Fig. 2. In safe states, all available actions are safe, i.e., they have risk of at most In critical states, only some actions are safe, while others may exceed the safety threshold. In dangerous states, all available actions are unsafe, i.e., each carries a risk of violating the safety property greater than e, although no safety violation has occurred yet. A state is considered unsafe it violates the safety property.
Computation of the shield Tshleld- Using the computed risk function Mis and the classification of states, the shielding policy Tshield: S - 2- is computed as follows:
(risk a, h) S, s Sa and risk (s, = min risk (ssk,,, a', h).
a) E Wshield
a'EAA
The shield permits all actions that are classified safe and blocks all unsafe actions. The only ex- ception is in dangerous states, where the shield allows the agent take the action with the lowest risk. Note that different approaches exist for defining shielding strategies in dangerous states, for example, selecting a predefined fallback action such as halting. Next, we compute the Level-1 DT Th1, which classifies states in the state space according to their risk category, providing an initial explanation of why a given state may be safety-critical
Computation of DT Tia- We construct the decision tree Tr.1 = (TrIa, IL,, PLI. ALi, Ara) ) for given MDP M, finite horizon h, and safety threshold e through the following steps. First, we define the set of predicates L1 as follows. P contains the basic predicates [rg ~ const). where 2 E V, const e Σ, and ~E {S, <, >, =}. Additionally, this set can be extended by user-defined predicates to incorporate domain knowledge. For example, predicates measuring the distance of a state s to the unsafe region S, often enable intuitive explanations of safety-critical dynamics. Such functions could capture, for example, the distance between two agents or the difference between the current temperature and an overheating threshold. These user-defined predicates are of the form P: k2 (21, . 2x) ~ const, with k; S + Z. Second, the classification of the states serves as the labels for the leaves of Tr.- Thus, AL = {s. c, d, u). Lastly, we learn Tr.1 (i.e.. its underlying tree Tra and functions PL1 and Ara) using standard DT learning algorithms [33, 39j. We learn an exact DTT s.t. for all leaves of Tr,, 1,(1) S S, for x E cs u). Intuitively, every E S is classified exactly, and the tree achieves perfect accuracy.

---

(a) A Frozen Lake environment.
(b) A Highway environment.
(c) The Taxingg environment.
Figure 3: The RL benchmarks used for evaluation our approach.
4.2 Case-based Safety Explanations
At runtime, our framework provides case-based explanations via two additional trees. Given cur- rent state 8, a Level decision tree explains which actions the RL agent allowed explore from For given unsafe action a, the Level 3 execution tree explains why executing a is unsafe in s.
Computation of DT TL2- Let be the current state. If E S., all actions are classified as safe. If s E S., safety is already violated. Therefore, we compute action-dependent safety explanations only for critical and dangerous states. critical states E Se. some actions are safe and some are unsafe (i.e., Ea: riskM,, a. h) S and 3a' risk a', > c.). A TT2 for critical state explains why actions are safe unsafe in In dangerous states, all actions are unsafe, i.e, Va: risk a, h) > c. In such states, our shield allows the agent to explore the safest available action (which still exceeds the safety threshold é). Then, the Tr.2 provides an explanation for why the selected action is considered the safest among the available options.
Let be the current state and 1 Thi (s) be the leaf node reached by state TLI. If S. over the states (1) as
or s E Sd, compute Level-2 DT Ti (Tri.2, T12: P1.2 2,,2,11222 follows. The set of predicates r7 is equal to IL1- The set of labels A/.2 is defined sets over actions, since the Tie explains which actions are allowed by the shield, i.e., A122 = 2
M/22)
Computation of ET TL3- Given a current state Sear and an unsafe action a., the ET TLa explains that executing ag in Seur leads to unsafe states with a probability greater than The tree provides this explanation by representing a set of traces that start in state reached after executing an in Scur and ending in an unsafe state, and that provide enough evidence to demonstrate that is unsafe. For given state pair (Scar, Qu), we compute the T T.aund as follows. First, the MDP M is transformed into an MC M = (S, 80, P) via the following policy
ag arg min risk ri,, aeA
8 = Seur: and otherwise.
ss : Ts) =
Thus, the policy ı picks in any state the safest available action. Only in the current state Scur, the policy selects the action au- Therefore, traces sampled from this policy demonstrate possible consequences of executing ag in Scar: if afterward only the safest actions were executed. We call finite trace 7 {so..... } unsafe if s, S,, We compute set of traces II = {TI, T2.... } that start state 8eur in the MC M such that the probability mass these traces exceeds the safety threshold e, i.e. Σvem P(r) 2 We follow the algorithm from [21] to compute the most probable traces the MC M that are unsafe and their probability mass exceeds the threshold e. The algorithm uses the recursive enumeration algorithm for computing the k shortest paths weighted digraph [24], substituting probabilities in M with distances. This algorithm computes the shortest paths recursively until the probability mass of the traces is greater than e. The resulting traces are mrefix-merged to construct the execution tree T (scar.au)
73
Experimental Evaluation
In this section, we present the experimental evaluation of our approach. We consider the size of shield Istictal as the size of the lookup table and the size of a tree [T] as the number of its nodes.
The tree size serves as our metric for evaluating explainability. The model checking queries were computed using TEMPEST [34], and the DT representations of shields using DTCONTROL. [5]. All experiments were conducted on a laptop with an Intel Core 1 1800H CPU at 2.3 GHz with 32
7

---

dar
North
{2.3..5)
du
0.95
0.017
00017
dy
day
dv $ 2.
6.6
5.7
7,7
S
North
True's False
0.95
0,017
dw
x
dg x
6,5
7,6
East
True Fulse
True ; False
0.017
#$
dg
E
7,5
Fast
Truc' False
True;; False
0,017
7.6
(a) TLi
(b) TLZ, dangerous leaf
(c) T
(6.7).N
Figure 4: Exemplary explanations for the Frozen Lake environment.
Taacal
Tendal
Size
10a
Teal
10a
Tuzl
10a
10
20
30
40
50
Grid Size
Figure 5: Shield and tree sizes for diffcrent configurations of the Frozen Lake environment.
GB of RAM. All details the experimental setup can be found in the Appendix. We provide the implementation supplementary material.
5.1 Frozen Lake
We performed first set of experiments using the Farama Frozen Lake environment [45]. A Frozen Lake environment an n X n grid, consisting of blue slippery tiles and holes. The agent can move in cardinal directions, where every movement carries probability of 0.05 of slipping into different direction. The RL agent has to reach the goal G while not falling into hole. The shield preventing the agent from falling into hole is computed with horizon h The state classification of an example environment is shown in Fig, 3a. Red tiles indicate unsafe states (holes), orange tiles indicate dangerous states, yellow tiles indicate critical states, and blue tiles indicate safe states. The set of predicates r (x, y, dar dN, dr, ds, dw consists of the agent's coordinates (ar, y). the Manhattan distance dm to the nearest hole, and the distance d, the nearest hole in the cardinal direction a E {N, S, E, W).
and risk threshold
0.075.
Explaining Safety. Fig. 4a shows TL for the environment depicted in Fig. 3a. TL can concisely capture the states in which the shield needs to interfere, by using the user-defined distance predicates. Fig. 4b shows Tra for the dangerous states in Tz1- The tree summarizes that, to exit dangerous state, the agent should move east the distance to a hole that direction permits it; otherwise, needs to move to the nearest safe state. Finally, Fig. 4c shows why moving the North at position
(6, 7) is not allowed. The represented traces in 7 an unsafe state, which exceeds the allowed risk of 0.075.
have probability mass of 0.08 of reaching
Results. We evaluated the scalability of our approach using randomly generated instances Frozen Lake environments of increasing size X n with E {5, 10.... 50). Per size, we generate 10 random instances and compare the sizes of the computed shields and tree representations. Fig. shows the average sizes of the shielding policy [shietd] the shield represented as one single tree [Taaietal, and the trees [Tzal, and |Tzz over grid size n x.. The results show that, using our approach, we obtain DT's that are several orders of magnitude smaller than the shielding policy. The average

---

drl S 20
Ni
True
False
shielal
[Tahielal
[Tzil
|Tza
0
de S
True False
Thucks False
HW2-f
358
7
7
HW2-c
3553
113
9
33
rdps 0
N.A
HW3-f
8355
18
17
2.5
HW3-c
232216
555
57
45.1
True False
True's False
TaxiNet
1105
39
21
N.A.B
N.B
TaxiNet
1107
37
33
(a) TEa for HW2-c
(b) Tıa2 for HW2--c
(c) Sizes of the shielding policy and the DTs
Figure 6: Exemplary DTs for HW2-c and the comparison of the sizes of shield and decision trees.
time for computing the shield ranges from 0.15s for 5 to 15.4s for n. 50. The average time to learn the DTs (TE) or TT22 is approximately 2s for all grid sizes.
5.2 Highway Cruise Control
We conducted our second set of experiments in the Farama Highway environment [28], illustrated in Fig. 3b. In this environment, the agent controls self-driving car operating on highway populated with other vehicles. The agent's 's actions are: switching lanes (S1, SR), accelerating (A), braking (B), or performing collisions. Additionally, receives positive reward for driving in the leftmost lane. We compute shield that ensures collision avoidance by enforcing safe distance of 20m. In this example, the shield prohibits the agent from taking any risks (i.c., e = 0) with h = cc. consider two parameters: two three lanes (HW2/HW3) and whether the agent can change its velocity (-f(ixed)/- chhangeable)). This results in the scenarios HW2-f, HW2-c, HW3-f, and HW3-c. We discuss the results of HW2-c. Appendix A.2 provides further results and precise predicate definitions. The set of predicates = [di.le: dz, N1, rd,, rdy, includes: the distance d1 to the nearest vehicle in lane I; the current lane of the ego vehicle le; the distance dy the closest vehicle in the ego vehicle's lane; N1 indicating whether the ego vehicle is next to a vehicle in lane I; and final predicates encoding the remaining distance other when the agent accelerates (rd or brakes (rdn).
operation (N). The agent's goal is to reach the end of the road without
Explaining Safety. Fig. 6a shows the Tr,, and shows the Tz2 for the topmost critical node. The trees explain that if the ego car is next to a car in the adjacent lane, switching lanes is not allowed. Moreover, the agent may only accelerate or brake within velocity bounds 20 29m/s.
Results. Table 6c shows the size of the shielding policy and the average sizes of DTs of different levels and scenarios. The results show, expected, that representing shields via trees instead of lookup tables yields more compact representation. Furthermore, using the hierarchical explana- tions TL1 and TLa is, most cases, more compact than using a single tree. The time for computing the shield ranges from 0.1s for HW2-f to 50s for HW3-c. The time to learn the DTs ranges from 1.3s for Tn of HW2-f 17.4s of Taa for HW3-f.
5.3 Boeing Taxinet
For our final set of experiments, we applied our approach to an autonomous taxiing environ- ment [41], shown in Fig. 3c. In this environment, the agent must steer an aircraft ensure remains aligned with the centerline. Depending on the current heading, the aircraft's 's position rel- ative to the centerline changes. The safety objective is to avoid exceeding a heading of 20 and maximum deviation of 0.8m from the centerline. The shield is computed with horizon h = oc and a risk threshold e 0.0. The set of predicates r (he, ele, [he,, |clel, , d) consists of the heading emoor he, centerline error cle, their absolute values, and the distance d to the maximum allowed centerline error, depending on the current heading. The set of actions A T,, N, R allows the agent to steer left (T.) right (R) in steps of 50, to take no action (N). We have applied our approach

---

to two different instances and show the resulting sizes for the shields and the DTs in Table 6c. The results again show that our approach yields compact representations of the shielding policy. We provide the corresponding DT's and further discussion in the Appendix.
6 Conclusion & Future Work
We presented a method for "explainably safe" RL. via proposed explainable shields. Our approach provides case-based explanations of the shielding policy and hierarchy of decision trees that ex- plains the risks associated with states and actions, as well as the consequences of executing unsafe actions. Our experiments show the capability of our method in providing small trees even for com- plex scenarios. In future work, we will explore automated methods for predicate generation to reduce reliance user input. Besides, we plan to investigate the potential of generative AI to ren- der case-based explanations understandable to non-specialists. Furthermore, user study on the understandability of explanations is also of interest. Lastly, we plan to extend approach by computing compact explanations for states that the trained RL policy visits frequently. As a trained RL policy rarely operates over the entire state space, focusing explanations on the most frequently visited states will provide clearer insights into the safety-relevant aspects of the agent's 's behavior.
Acknowledgments
Bettina Könighofer and Stefan Pranger were supported by the State Government of Styria, Austria- Department Zukunftsfonds Steiermark and by the Austrian Science Fund (FWF) 10.55776/COE12. This research has furthermore received funding from the European Union under Grant Agree- ment No. 101171844, ERC project Intelligence-Oriente OVationCS), and from the European Union's Horizon Europe program under Grant Agreement No. 101212818, Robustifying Generative AI Through Human-Centric Intergration of Neural and Symbolic Methods (RobustifAD). Views and opinions expressed are, however, those of the authors only and do not necessarily reflect those of the European Union or European Research Execu- tive Agency. Neither the European Union nor the granting authority can be held responsible for them. This research has also received funding from the MUNI Award in Science and Humanities MUNMV/1757/2021 of the Grant Agency of Masaryk University.
Verification&Conttollla Synthesis (In-
ZUK SS NDS FWF
Austrian Science Fund
erc
STEIERMARK
References
[1] Mohammed Alshiekh, Roderick Bloem, Rüdiger Ehlers, Bettina Könighofer, Scott Niekum, and Ufuk Topcu. Safe reinforcement learning via shielding. In Sheila A. Mcllraith and Kilian Q- Weinberger, editors, Proceedings of the 32nd AAAI Conference Artificial Intelligence, AAAI 2018, pages 2669-2678. AAAI Press, 2018
[2] Ofra Amir, Finale Doshi-Velez, and David Sarne. Summarizing agent strategies. Autonomous Agents and Multi-Agent Systems, 33(5):628-66) 644,, 2019.
[3] Roman Andriushchenko, Milan Ceska, Sebastian Junges, and Filip Macák. Small decision trees for mdps with deductive synthesis. Ruzica Piskac and Zvonimir Rakamarie, editors, Proceedings of the 37st International Conference on Computer Aided Verification, CAV 2025, volume 15932 of Lecture Notes in Computer Science, pages 169-192. Springer, 2025.
[4] Pranav Ashok, Jan Kretínský, Kim Guldstrand Larsen, Adrien Le Coënt, Jakob Haahr Taankvist, and Maximilian Weininger. SOS: safe, optimal and small strategies for hybrid markov decision processes. In David Parker and Verena Wolf, editors, Proceedings of the 16th International Conference Quantitative Evaluation of Systems, QEST 2019, volume 11785 of Lecture Notes in Computer Science, pages 147-164. Springer, 2019.
[5] Pranav Ashok, Mathias Jackermeier, Jan Kretínsky, Christoph Weinhuber, Maximilian Weininger, and Mayank Yadav. dtcontrol 2.0: Explainable strategy representation via decision
10

---

tree learning steered by experts. In Proceedings of the 27th International Conference on Tools and Algorithms for the Construction and Analysis of Systems, TACAS 2007, volume 12652 of Lecture Notes in Computer Science, pages 326-345. Springer, 2021.
[6] Guy Avni, Roderick Bloem, Krishnendu Chatterjee, Thomas A. Henzinger, Bettina Könighofer, and Stefan Pranger. Run-time optimization for learned controllers through quantitative games. In Isil Dillig and Serdar Tasiran, editors, Proceedings of the 31st International Conference on Computer Aided Verification, CAV 2019.
[7] Muqsit Azeem, Debraj Chakraborty, Sudeep Kanav, Jan Kretínský, MohammadSadegh Mo- hagheghi, Stefanie Mohr, and Maximilian Weininger. 1-2-3-go! policy synthesis for parame- terized markov decision processes via decision-tree learning and generalization. In Shankara- narayanan Krishna, Sriram Sankaranarayanan, and Ashutosh Trivedi, editors, Proceedings of the 26th International Conference on Verification, Model Checking, and Abstract Interpretation, VM- CAI 2025, volume 15530 of Lecture Notes in Computer Science, pages 97-120. Springer, 2025.
[8] Christel Baier and Joost-Pieter Katoen. Principles of Model Checking. MIT press, 2008
[9] Osbert Bastani, Yewen Pu, and Armando Solar-Lezama. Verifialle reinforcement learning via policy extraction. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference Neural Information Processing Systems 2018, NeurlPS 2018, pages 2999, 2018
[10] Roderick Bloem, Bettina Könighofer. Robert Könighofer, and Chao Wang. Shield synthesis: runtime enforcement for reactive systems. In Christel Baier and Cesare Tinelli, editors, Pro- ceedings of the 21st International Conference Tools and Algorithms for the Construction and Analysis of Systems, TACAS 2015, volume 9035 of Lecture Notes in Computer Science, pages 533-548 Springer, 2015.
[11] Tomás Brázdil, Krishnendu Chatterjee, Martin Chmelik, Andreas Fellner, and Jan Kretínsky. Counterexample explanation by learning small strategies in markov decision processes. In Daniel Kroening and Corina S. Pasareanu, editors, Proceedings of the 27th International Conference on Computer Aided Verification, CAV 2015, volume 9206 of Lecture Notes Computer Science pages 158-177. Springer, 2015.
[12] Carlos E. Budde, Pedro R. D' Argenio, and Arnd Hartmanns. Digging for decision trees: A case study in strategy sampling and learning. In Bernhard Steffen, editor, Proceedings of the 2nd International Conference on Bridging the Gap between Al and Reality, AlSoLA 2023, volume 15217 of Lecture Notes in Computer Science, pages 354-378. Springer, 2024.
[13] Carrie J. Cai, Jonas Jongejan, and Jess Holbrook. The effects of example- based explanations in a machine learning interface. In Wai-Tat Fu, Shimei Pan, Oliver Brdiczka, Polo Chau, and Gaelle Calvary, editors, Proceedings of the 24th International Conference on Intelligent User Interfaces, IUI 2019, pages 258-262. ACM, 2019.
[14] Filip Cano, Thomas A. Henzinger, Bettina Könighofer, Konstantin Kueffner, and Kaushik Mallik. Fairness shields: Safeguarding against biased decision makers. In Toby Walsh, Julie Shah, and Zico Kolter, editors, Proceedings of the 39nd AAAI Conference on Artificial Intelli- gence, AAAI 2025, pages 15659-15668. AAAI Press, 2025.
[15] Steven Carr, Nils Jansen, Sebastian Junges, and Ufuk Topcu. Safe reinforcement learning via shielding under partial observability. In Brian Williams, Yiling Chen, and Jennifer Neville, editors, Proceedings of the 37th AAAI Conference on Artificial Intelligence, AAAI 2023, pages 14748-14755 AAAI Press, 2023.
[16] Zelei Cheng, Xian Wu, Jiahao Yu. Wenhai Sun, Wenbo Guo, and Xinyu Xing, Statemask: Ex- plaining deep reinforcement learning through state mask. Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Infor- mation Processing Systems 36: Annual Conference on Neural Information Processing Systems, NeurlPS 2023, 2023.
11

---

[17] Rudresh Dwivedi, Devam Dave, Het Naik, Smiti Singhal, Omer F. Rana, Pankesh Patel, Bin Qian. Zhenyu Wen, Tejal Shah, Graham Morgan, and Rajiv Ranjan. Explainable AI (XAI): core ideas, techniques, and solutions. ACM Computing Survey, 55(9): 1-14::9: 2023.
[18] Ingy Elsayed-Aly, Suda Bharadwaj, Christopher Amato, Rüdiger Ehlers, Ufuk Topeu, and Lu Feng. Safe multi-agent reinforcement learning via shielding. In Frank Dignum, Alessio Lo- muscio, Ulle Endriss, and Ann Nowé, editors, Proceedings of the 20th International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2021, pages 483 491. ACM, 2021.
[19] Claire Glanois, Paul Weng, Matthieu Zimmer, Dong Li, Tianpei Yang, Jianye Hao, and Wulong Liu. A survey on interpretable reinforcement learning. Machine Learning. 113(8):5847-5890,
2024.
[20] Wenbo Guo, Xian Wu, Usmann Khan, and Xinyu Xing. EDGE: explaining deep reinforcement learning policies. In Mare' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34; Annual Conference on Neural Information Processing Systems, NeurlPS 2021, pages 12222- 12236, 2021.
[21] Tingting Han and Joost-Pieter Katoen. Counterexamples in probabilistic model checking. In Orna Grumberg and Michael Huth, editors, Proceedings of the 12th International Conference on Tools and Algorithms for the Construction and Analysis of Systems, TACAS 2007, volume 4424 of Lecture Notes in Computer Science, pages 72-86. Springer, 2007.
[22] Christian Hensel, Sebastian Junges, Joost-Pieter Katoen, Tim Quatmann, and Matthias Volk. The probabilistic model checker storm. International Journal Software Tools for Technology Transfer, 24(4)5589-610 2022.
[23] Nils Jansen, Bettina Könighofer, Sebastian Junges, Alex Serban, and Roderick Bloem. Safe reinforcement learning using probabilistic shields. In Igor Konnov and Laura Kovács, editors, Proceedings of the 31st International Conference on Concurrency Theory, CONCUR 2020, vol- ume 171 of LIPlcs, pages 3:1-3:16. Schloss Dagstuhl Leibniz-Zentrumm für Informatik, 2020.
[24] Victor M. Jiménez and Andrés Marzal. Computing the K shortest paths: A new algorithm and an experimental comparison. In Jeffrey Scott Vitter and Christos D. Zaroliagis, editors, 3rd International Workshop Algorithm Engineering, WAE '99, volume 1668 of Lecture Notes in Computer Science, pages 15-29. Springer, 1999.
[25] Florian Jüngermann, Jan Kretínský, and Maximilian Weininger. Algebraically explainable controllers: decision trees and support vector machines join forces. International Journal on Software Tools for Technology Transfer, 25(3):249-265 22622 2023.
[26] Bettina Könighofer, Roderick Bloem, Rüdiger Ehlers, and Christian Pek. Correct-by- construction runtime enforcement in AI A survey. In Principles of Systems Design Essays Dedicated to Thomas A. Henzinger on the Occasion of His 60th Birthday, volume 13660 Lec- ture Notes in Computer Science, pages 650 663. Springer, 2022.
[27] Marta Z. Kwiatkowska, Gethin Norman, and David Parker. PRISM 4.0: Verification of prob- abilistic real-time systems. In Ganesh Gopalakrishnan and Shaz Qadeer, editors, Proceedings of the 23rd International Conference Computer Aided Verification. CAV 2011, volume 6806 of Lecture Notes in Computer Science, pages 585-591. Springer, 2011.
[28] Edouard Leurent. An environment for autonomous driving decision-making. https://github. com/cleurendhighwwayeen 2018.
[29] Daniel Melcer, Christopher Amato, and Stavros Tripakis. Shield decentralization for safe multi-agent reinforcement learning. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Bel- grave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: An- nual Conference on Neural Information Processing Systems, NeurIPS 2022, Curran Associates, Inc., 2022.
12

---

[30] Stephanie Milani, Zhicheng Zhang, Nicholay Topin, Zheyuan Ryan Shi, Charles A. Kamhoua, Evangelos E. Papalexakis, and Fei Fang. MAVIPER: learning decision tree policies for inter- pretable multi-agent reinforcement learning. In Massih-Reza Amini, Stéphane Canu, Asja Fis- cher, Tias Guns, Petra Kralj Novak, and Grigorios Tsoumakas, editors, Proceedings of the 19th European Conference on Machine Learning and Knowledge Discovery in Databases, ECML PKDD 2022, volume 13716 of Lecture Notes in Computer Science, pages 251-266. Springer, 2022.
[31] Stephanie Milani, Nicholay Topin, Manuela Veloso, and Fei Fang. Explainable reinforcement learning: A survey and comparative review. ACM Computing Surveys,56(7::1655 16:7):168::1168:333 2024.
[32] Aditi Mishra, Utkarsh Soni, Jinbin Huang, and Chris Bryan. Why? why not? when? visual explanations of agent behaviour in reinforcement learning. In 15th IEEE Pacific Visualization Symposium, PacificVis 2022, pages 111-120. IEEE, 2022.
[33] Tom M. Mitchell. Machine learning, International Edition. McGraw-Hill Series in Computer Science. McGraw-Hill Hill, 1997.
[34] Stefan Pranger, Bettina Könighofer, Lukas Posch, and Roderick Bloem. TEMPEST Synthesis Tool for Reactive Systems and Shields in Probabilistic Environments. In Proceedings of the 19th International Symposium on Automated Technology for Verification and Analysis, ATVA 2021, volume 12971, pages 222 228. Springer, 2021.
[35] Stefan Pranger, Bettina Könighofer, Martin Tappler, Martin Deixelberger, Nils Jansen, and Roderick Bloem. Adaptive shielding under uncertainty. In Proceedings of the 2021 American Control Conference, ACC 202/, pages 3467-3474. IEEE, 2021.
[36] Stefan Pranger, Hana Chockler, Martin Tappler, and Bettina Könighofer. Test where decisions matter: Importance-driven testing for deep reinforcement learning. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomezak, and Cheng Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurlPS 2024, 2024.
[37] Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernsstss, and Noah Dormann. ::able-baselines:: Reliable reinforcement learning implementations. Journal of Ma- chine Learning Research, 22(268):1-8, 2021.
[38] Aaron M Roth, Dinesh Manocha, Ram D Sriram, and Elham Tabassi. Explainable and Inter- pretable Reinforcement Learning for Robotics. Springer, 2024.
[39] Shai Chalev-Shwarte and Shai DeviD. Understanding Machine Learning: From Theory to Algorithms. Cambridge University Press, USA, 2014.
[40] Mohan Sridharan and Ben Meadows. Towards theory of explanations for human- col- laboration. Künnstlccee Intell., 33(4):331-342, 2019.
[41] Tyler C Staudinger, Zachary D Jorgensen, and Dragos D Margineantu. Xplane-ml-an environ- ment for learning and decision systems for airplane operations. 2018
[42] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning An Introduction, 2nd Edi- tion. MIT Press, 2018
[43] Martin Tappler, Stefan Pranger, Bettina Könighofer, Edi Muskardin, Roderick Bloem, and Kim G. Larsen. Automata Learning Meets Shielding. In Proceedings of the 11th International Symposium Leveraging Applications of Formal Methods, Verification and Validation, ISoLA 2022, pages 335-359. Springer, 2022.
[44] Martin Tappler, Ignacio D. Miguel, Sebastian Tschiatschek, and Ezio Bartocci. Rule- guided reinforcement learning policy evaluation and improvement. In James Kwok, editor, Pro- ceedings of the 34th International Joint Conference on Artificial Intelligence, IJCAI-25, pages 6254-6262. International Joint Conferences on Artificial Intelligence Organization, 8 2025. Main Track.
13

---

[45] Mark Towers, Ariel Kwiatkowski, Jordan K. Terry, John U. Balis, Gianluca De Cola, Tris- lan Deleu, Manuel Goulão, Andreas Kallinteris, Markus Krimmel, Arjun KG, Rodrigo Perez- Vicente, Andrea Pierré, Sander Schulhoff, Jun Jet Tai, Hannah Tan, and Omar G. Younis. Gym- nasium: A standard interface for reinforcement learning environments. CoRR, abs/2407.17032, 2024.
[46] Jasper der Waa, Elisabeth Nieuwburg, Anita H. M. Cremers, and Mark A. Neerinex. Eval- uating XAI: comparison rule-based and example-based explanations. Artificial Intelligence, 291:103404, 2021.
[47] Marko Vasic, Andrija Petrovic, Kaiyuan Wang, Mladen Nikolic, Rishabh Singh, and Sarfraz Khurshid. Moër: Interpretable and verifiable reinforcement learning via mixture of expert trees. CoRR, abs/1906.06711 17, 2019.
[48] Daniël Vos and Sicco Verwer. Optimal decision tree policies for markov decision processes. In Proceedings of the 32nd International Joint Conference on Artificial Intelligence, IJCAI 2023, pages 5457-5465. ijeai.org, 2023.
[49] Lindsay Wells and Tomasz Bednarz. Explainable Al and reinforcement learning A systematic review of current approaches and trends. Frontiers in Artificial Intelligence, 4:550030, 2021.
14

---

Figure 7: A randomly generated Frozen Lake environment with approximately 15% holes.
Jraiicll
10a
[aanall
Size
[Taal
iTbal
10
10
20
30
40
50
Grid Size
Figure 8: Size comparison between the different approaches using random Frozen Lake instances with approx- imately 4% of holes.
Appendix
10a
Taxaa
Size
10
Thal
10a
Thal
10a
40
Grid Size
Figure 9: Size comparison between the different approaches using random Frozen Lake instances with approx- imately 20% of holes.
A.1 Additional Results Frozen Lake
Scalability. For the experiments on the scalability of our approach, we randomly generated instances of the Frozen Lake environment of various sizes. To ensure that all environments have a meaningful structure, divide the overall size N x N of the instance into tiles of size 5 x 5. In the center of each tile, we place a randomly shaped connected hole of size to 5. Fig shows such a randomly generated environment. Red areas represent holes, orange areas the dangerous states, and yellow the critical states. For the experiment presented in the main part of the paper, Figure 5, we create the holes in such way that the expected value of holes is 15% of all fields. The figure shows that the size DT is already smaller than the size of the original shield. Our method reduces the size further, even for large problem instances.
In Figure 8, we perform the same experiment with randomly generated instances, but only allow holes of sizes 0 to 2, which leads to expected value of 4% of all fields being holes. In this case, all DT's are small and remain small for very large instances.
On the other hand, when creating instances with 20% of the entire grid being holes, the size of the DT' grows. This is caused by fact that holes are close to each other because of their size. When a field is surrounded by several holes, it becomes dangerous instead of critical. Therefore, having many holes interacting with each other causes the overall problem to become more complicated and makes it harder to distinguish critical from dangerous states.
15

---

rds
True's False
ddg <
20 < Idzl.
True's
False
17 di
d 61
True's False
True'FFalse Fulse
Sz
N
(a) Tia1 for scenario HW2-f
(b) Tu.2 for the critical leaf
Figure 10: Decision Trees for scenario HW2-f
A.2 Additional Results -Highway Cruise Control
In this section, discuss the remaining scenarios for the Highway environment and elaborate on the suggested predicates.
HW2-f. In the base scenario, the highway has two lanes, and all vehicles are driving at a fixed speed. The agent is slightly faster with a value of va than the vehicle on the right lane. The available actions switch left (5.). switch right (Sx). and noop (N). All actions succeed with probability 1. The feature set of the environment is V = {d1, J. the distance to the vehicle on lane which the rightmost lane, and the current lane of the ego vehicle. We extend the set of basic predicates = {di, le, dz, rd] with the following predicates:
d₁ == (dl if le = 1 else 100) is the distance to the vehicle on the current lane of the ego vehicle. If there no vehicle on the lane, the distance is set to high value.
rd = d1 va- -20 is the remaining distance until action has to be taken immediately. It can seen as the distance to the safety buffer around the other vehicle. To avoid comparison with the constant Vd. subtract it.
The DT Tra describing this instance depicted in Fig. 10a. It consists of 7 nodes. The DT investi- gates whether the remaining distance to the car on lane remains large assuming no action is taken. If this the case, every action allowed. Otherwise, the DT compares to the safety threshold to filter out unsafe states. Lastly, when the agent is in front of the vehicle lane it may perform any action. Otherwise, as the root tests that the agent is close to the vehicle on lane 1, the state is critical. Figure 10b shows Tz2 for the critical state. If the agent is the left lane, shown by high lane distance, may not switch to the right lane. Otherwise, it has to switch to the left lane to avoid a collision. Computation of Tua was completed within less than 1.4 seconds, T22 computation required on average 1.6 seconds and shield creation 0.1 seconds.
HW2-c. In this scenario, the agent is allowed to change its speed and accelerate or decelerate by fixed value of A.. Therefore, the set of features extended with the velocity the ego vehicle and the vehicle on lane 1 to V {d1, le, Ve, 01). The agent has the additional actions accelerate (A) and brake (B). We provide the predicates
(di, le, Ce, U1, dr, N1, rd a.ddrrd rdg) ) with the addition of:
rdd == (d1 (ve + A. v1) - 20 if lg = 1 else 100) is the remaining distance until the safety requirement is violated, assuming the agent accelerates and the other vehicle is on the same lane.
rdB == (di - (we A, - vi) + if 1, else 100) the remaining distance until the safety requirement is violated, assuming the agent decelerates and the agent is on lane 1.
N₁ == (dj E [-20 + Va, 20 + vd) which captures whether in the next time step, after the ego has approached the other vehicle by va space units, it will be within the safety zone of the other vehicle. In this case, the ego vehicle is considered to be next to the other vehicle.
The new predicates are a refinement of rd to allow for a better understanding of the behavior of different actions.
16

---

M
drl S
Ni
True " False
rd 0
6e S
Trlse
Truc's False
c
N.A
Truo's False
True False
N.A.B
N.B
(a) Level-1 DT for scenario HWW2-c
(b) T1.2 for the rightmost critical leaf
Figure 11: DTs for scenario HW2-c
20 [dx].
Nj
N2
Mu] s
Truc't False
rdi s
Mi
True False
S
N₂
|du 75
(b) Tıa for the critical leaf with N₁
0 and N₂-
True
False
20 ddl 73
rdj 5 3
Ni = 0,
No
True
False
roj 5
[Mxl 75
drl S 22
True Faalse
True False
ruly x
0
[dy] s 23
St.
True
False
True False
Sc.N
(a) TIi for scenario HW3-f
(c) Tua for the leftmost critical leaf
Figure 12: DT's for scenario HW3-f
Tr.1 is depicted in Figure 1la. It explains that when the ego vehicle is next to another vehicle, some actions are prohibited. Furthermore, when the current distance falls below the safety distance, the state is unsafe. In all other cases, the criticality depends on whether acceleration and braking are safely possible. In this example, no dangerous states exist as the environment is deterministic. Level-2 DTs consist, average, of 33 nodes. This is caused by others remain small, similar to the one in Figure 6b, which is also depicted in Figure llb. Computing Ti required 1.3 seconds, the average time for Tra was again 1.3 seconds, and creation of the shield took 0.7 seconds.
tree having 81 nodes while all
HW3-f. This scenario is an extension of HW2-f with an additional lane. All vehicles drive at a fixed speed, with the ego vehicle being slightly faster by value of va. The set features extended with the distance to the vehicle on lane 2 V = {d1 d2,1c). We extend the set of predicates = {d), d2, le, dr, N1, N2, rd, co) We define N, and N₂ accordingly for the vehicles on lanes and 2, and define rdj and rdg as rd for the vehicles lanes I and 2, respectively.
17

---

The Level-1 DT is depicted in Figure 12a. It shows that the classification of the states depends on the distance to the vehicle in front and whether the agent is next to another vehicle. If the vehicle is next to one on lane and has vehicle close in front, it cannot avoid a collision.
On average, the Level-2 DT's have size of 2.5 nodes, with the largest one having 5 nodes. Examples are shown in Figures 12b and 12c. Figure 12b corresponds to the critical leaf where N1 0 and N2 1. 2₂ = 1 already encapsulates that the agent currently on the middle lane. The available actions then depend on whether it is too close to the vehicle on lane 1, which disallows action N. In any case, switching to the leftmost lane is allowed. For the leftmost critical leaf, which is shown in Figure 12c, we know that the agent is not next to any vehicle. However, the current remaining distance to the vehicle on lane is low. Therefore, the agent is on lane it should switch left to avoid a scenario where overtaking the vehicle on lane 2 is no longer possible. If it is on lane 1, the available actions depend to the distance of the vehicle on lane Computation of TT,1 was completed within 1.8 seconds, TL2 required, on average, less than 1.3 seconds, and shield creation less than 2.7 seconds.
HW3-c. In the last scenario, we consider a highway with three lanes and allow the ego vehicle to change its velocity. The state space V is extended with pa, which shows what action was taken in the previous step. It is required to precisely model the Highway RL environment, where the speed after no operation or switching lanes also depends on the action of the previous time step.
We allow all predicates the previous cases and add a new predicate sbd, safe braking distance, which compares the braking distance (bd) needed for the ego vehicle brake until it is most as fast as the vehicle in front of it, to the remaining distance: sbd := (dz 2 bd). It is an extension of rdg rdg considers switching lanes
possible action, whereas rbd only allows braking. We obtain Level-1 DT of slightly below 70 nodes, which is depicted in Figure 13. The size of the tree results from the special cases in which the ego vehicle can switch to the right-most lane and break before hitting the vehicle in front of it, or switch back to the middle lane. Introducing predicate for these special cases reduces the size drastically but requires more detailed domain knowledge. This shows the need for the automatic finding of intelligent predicates. Similar behavior can be observed for the Level-2 DTs. which have an average size of 36 nodes. Computation of the TI1 required 17.4 seconds. Level-2 DTs are computed on average in less than 3.6 seconds. Shield computation was performed within 50 seconds
18

---

True's False
N2
N2
abd
rda S
True't False
rdy < O
True't False
True's False
26
True False
da 26
4
False
(0. 2)
23
de
True
False
True False
drl
29
True False
True False
dzl $
3
N. B
True
True False
dg 28
de 30
True False False
True
False
dzl < 32
d
<
28
True False
True False
True
False
pa
c
us 28
pu
True
False
N
de
pa
True à False
drl <
False
Figure 13: Level-1 DT for scenario HW3-f
19

---

Arl S
True : False
S
True ; False
hel <
True
False
d
d ds
True False
True
False
d d=1
U
True': False
S -1
ctel $
12 < [he] s 17. letel $ 79, 0 04
True ; False
True
False
hel 7
h 0
True's False
False
s
ctes -71
73
True's False
True
True
False
R
N.R
L.N
(a) The decision tree Tr.1
(b)
Tr.2 critical leaf.
Figure 14: Decision trees for the taxiing environment.
A.3 Additional Results Boeing Taxinet
In this section, discuss the results for two different autonomous taxiing environments: model where steering always follows a deterministic update and slippery model, where steering might cause the airplane to slip, causing heading to change by 10 instead of 5 Initially, the airplane is positioned at the centerline with heading of 0%. Depending on its current heading, the airplane moves be decimeter per time step. '5
Taxiing. In this base scenario the agent will only be restricted by the shield if either it would reach the maximum heading error 20" or in case it would diverge from the centerline by 80 decimenter. The decision tree TLi and TL2 for critical Ieaf shown in Fig. 14a and 14b, respectively. The level-1 DT TI1 nicely shows how user-defined predicates can help explain the shield in concise way. After classifying the states in which the property has already been violated, the TIi differentiates between states which the heading error is smaller or larger than 12". It can then immediately make use of the defined distance function. In case the heading error is still small and the distance is sufficiently large, all states are safe and no interference needed. If the distance is 40 decimeter TE classifies the different scenarios depending on conerete distance, heading and centerline errors. If the heading error exceeds 12" and safety has not yet been violated, the shield has to interfere. The restrictions the shield imposes are shown in 14b. This Tra nicely shows how a compact representation allows us to legibly explain how the shield interferes.
The synthesis time for the shielding policy is 0.19 seconds and the time needed to compute either TI1 or TL2 tree 1.5 seconds.
Taxingg on Slippery Ground. In this second experiment, the aircraft might slip an additional 5 with a probability of 10% in case it is steered to the left or to the right. The resulting decision trees are shown in 15. The Level-1 DT Tr.1, shown in 15a, captures this by more finely distinguishing between the current centerline error and distance to the maximum centerline error. Similarly to the Level-1 DT from the previous experiment, it classifies all states in which the heading error is
20

---

Rel 17
True 1 False
Iel s
True's False
off S
<
True False
True " False
d
ds
ds
True False
True False
True
False
ds
True' False
True
False
True False
7< lhel + 12. letel = d
ote| 7
$
12
True False
True's False
cte 0
Tiue T False
True's : False
R
L
sel
(b) A T1a for a critical leaf.
True; False
12 < [he] s 17. letel & d 0
a'
True's False
ctlls
clc S 0
True: Fulse
True's False
I
(a) The decision tree TLa
(c) A T2z for dangerous Icaf.
Figure 15: Decision trees for the slippery taxiing environment.
sufficiently small, < 7", and the distance to the maximum centerline error is large enough, safe. In case the heading error is small, but the centerline error close to the maximum allowed value, Tr. distinguishes between safe and critical states by capturing the different values for the distance and centerline error. In case the heading error is > 7", the decision tree Tr. has to classify states that either critical, dangerous, or already violating the safety specification. Fig. 15b and Fig. 15c show two examples for Tr2-- We want to highlight these two examples for Tr.2, as they show how our approach able explain both how the shield has interfere order enforce the safety specification, as well as when the safety specification cannot be adhered to anymore and only safest actions can be allowed by the shield anymore.
As above, the synthesis time for the shielding policy is 0.19 seconds and the time needed to compute either TL1 or TLa tree is 1.5 seconds.
21

---

4,000.0
1.0
wro shield
Reward
Reached
Violation
300.0
w/ shield
2,000.0
200.0
-1.0
0.0
0k
50k
100k
Uk
50K
100k
0k
50k
100k
Timesteps
Timesteps
Timesteps
(b) Accumulated number of cessful episodes
Accumulated safety viola-
(a) Rewards for Frozen Lake en- vironment
tions
B Supplementary Material
We have trained agents on the Frozen Lake environment from Fig. 3a and the four different settings the Highway environment using the implementations from Stable-Baselines3 [37]. The library provides MaskablePPO implementation, which is perfectly suited for safe RL via shielding. We have compared shielded training with training without shield using PPO. All training runs have been conducted using the default parameters.
Due to the simulator for the Boeing TaxiNet environment being closed-source, we were not able to train agents for these problem instances.
B.1 Frozen Lake Environment
The task of the agent in this environment is reach the goal state without falling into hole. The agent can move in any cardinal direction and will succeed with a probability of 0.95. With a probability of 0.05 will slip in any of the other cardinal directions where not obstructed by wall. Upon reaching the goal the agent receives a reward of 1, otherwise, when the agent falls into hole receives negative reward of -1.
Fig. 16a, 16b, and 16c show the training results averaged over runs. These results show that the shield enables the agent finish its task, while training without the shield does not succeed. When training without a shield, the agent is not able to explore the critical areas around the holes and therefore stays at the safe area near the initial position only. We want remark that due to the stochasticity exhibited in this environment, complete safety cannot be guaranteed.
B.2 Highway Environment
The task of the agent in this environment is reach the end of the highway fast possible without causing a crash. The agent can switch lanes, do nothing, or change its velocity, is not fixed. The agent receives a reward of 1.0 for driving the rightmost lane, reward of 0.5 on the leftmost lane, and a reward of 0.75 for driving on the middle lane in the environments HW3-f and HW3-c. If the agent crashes into another car, it receives negative reward of 1..0.
We show the training results, averaged over 5 runs Figures 17a to 17h. We want use the example of environment HW3-c to highlight the need for accurate world models: Due to the world model not being 100% accurate, we have exhibited small number of crashes the shielded training run.
22

---

25.0
800.0
w/o shield
Reward
20.0
# Craches
600.0
shield
400.0
15.0
200.0
10.0
0.0
20k
Timesleps
Timesteps
(a) Rewards for HW2-f
(b) Accumulated Crashes for HW22f
30.0
200.0
wio shield shield
Reward
25.0
# Crashes
100.0
20.0
15.0
0.0
20k
40k
0k
20k
40k
Timesteps
Timesteps
(c) Rewards for HW2-c
(d) Accumulated Crashes for HW2-c
30.0
800.0
wlo shiekd
Reward
25.0
a Crabbes
600.0
shicld
20.0
15.0
200.0
0.0
10.0
20k
Timesteps
Timesteps
(c) Rewards for HW3-F
(f) Accumulated Crashes for HW3-F
300.0
shield shield
Reward
25.0
Crashes
200.0
100.0
15.0
20k
40k
0k
20k
40k
Timesteps
Timesteps
(g) Rewards for HW3-c
(h) Accumulated Crashes for HW3-c
Figure
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We explain how we generate the described explanations and show in the scalability study in Figure 5 the size of the shield and the decision trees.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We clearly highlight in the conclusion and experiments how our method de- pends on well-defined predicates for larger problem instances.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to system might not be used reliably provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
24

---

Answer:
[NA]
Justification:
There are no theoretical results that require proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the of the paper should be comple- mented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Ys]]
Justification: We explain the set of predicates we use for computing the DTs. Furthermore, we name the used tools. Furthermore, we provide the code to reproduce the results in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurlPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature the contribution. For example
(a) the contribution primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results way produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
25

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification:
We provide the code in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cc/ppblic guidds/CaueStbmiisiiooollly for more details.
While we encourage the release of code and data, understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: httppuul/ggiidee/oooeuuiiiiiisss
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We explain the parameters used for instantiating the RL environments and provide the world models in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should presented in the core the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question:
Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: For the randomized scalability experiments, the plots show the mean and highlight the area plus/minus standard deviation around Each test case was run 10 times with different random seeds. In all other experiments, the results are deterministic.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
26

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We state in the beginning of our section on evaluation which machine we
used.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
No human participants were involved in the research. Furthermore, all exper- iments are based on models, which are not recorded from human interactions. We cannot forsee negative usecase of our research.
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In the introduction we explain the need for safe RL learning. Our method can have impact in this direction. We forsee negative socictal impacts.
27

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not create any data or model that has high risk of misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We cite the sources of the used RL environments.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
28

---

For scraped data from a particular source (e.g., website), the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms of use the pack- age should be provided. For popular datasets, paperswitheode. seeerssithhod.cccd/dataatttet com/datasets has
rated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [NA]
Justification:
We do not introduce new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the pa- per include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Question:
Answer: [NA]
Justification:
This paper does not involve crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, cura- tion, or other labor should paid at least the minimum wage the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
This paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
29

---

Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: Our proposed method does not involve LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy https://eeertssscccommceeeeessss for what should or should not be described.
30


Reviews:
[{"review_id": "1QHhCASH4G", "reviewer": "Reviewer_RZs6", "summary": "A shield in RL is a component of an RL agent that prevents it from performing any action that has been deemed to be unsafe by the model designer. This paper proposes that by explaining an RL agent's shield, that agent can be used in a safe and trustworthy manner. The contribution of this paper is a shield *explainer*. The shield explainer works by explaining the shield's decisions at different levels of granularity corresponding to levels of execution and safety. For example, they write that a \"level 1\" tree classifies actions as safe or unsafe, a \"level 2\" tree is used to explain which actions the RL agent can choose safely and a \"level 3\" tree is used to explain why an unsafe deemed unsafe would be unsafe to execute.", "questions": "- Is generally practical to create a shield for an RL agent? Obviously the applicability of this paper depends on how easy it is to create shields. \n- Building off of the first question, how hard would it be to extend your method from working on the farama foundation environment to an actual self-driving car context?\n- Please further justify the choice of explaining the shield with DTs. Could concept bottle neck models (CBMs) be used instead, for example?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "- The idea of this paper (explaining an RL shield) is neat.\n- I enjoyed reading this paper. It was easy to follow and very clear. I appreciated the level of detail that the authors described their methodology at. The structure of the paper and use of formal definitions helped my reading. \n- I think practitioners and researchers will want to use this method if it is open source and can be easily integrated into their workflows.\n- As I will mention in the next section, explaining RL agents using DTs is not new, but I do think the specific use case here where a hierarchy of explanations of a shield is constructed using 3 DTs is novel and clever idea. \n\n## Weaknesses\n\n- The authors should give citations/evidence for why case-based explanations are good explanations. \n- I am not convinced that, in general, the 3 trees used in explanations would be of sufficiently low complexity so that a user can understand the shield. I would like more detail on how the DTs were computed. Are they trained to be sparse to help ensure that they can be understood? Sparsity of the trees is especially important for this paper because we have to look at 3 of them. \n- Explaining RL agents using decision trees is not a new idea. I think VIPER was one of the first to do it and I have seen other works use the idea since the publication of VIPER. \n- Please further justify the choice of explaining the shield with DTs. Could concept bottleneck models (CBMs) be used instead, for example?\n- No human evaluation study is given", "was_revised": true, "final_justification": "Having looked at the authors' response to my critique and to the critiques of the other reviewers I have decided to keep my scores the same. However, I do want to point out that the authors chose not to respond to the following point I raised in my review in their rebuttal:\n> Explaining RL agents using decision trees is not a new idea. I think VIPER was one of the first to do it and I have seen other works use the idea since the publication of VIPER.", "rebuttal": "Thank you for your helpful insights. In the following, we respond to your comments and questions.\n\nQ: Case-based explanations: \nA: We shall put references. However, note that a global explanation that is too big to even display and thus useless is necessarily less useful than an explanation of a concrete case. Further, such methods have been discussed in the setting of AI under the name example-based explanation.\n\nQ: Practicality of creating a shield, and difficulty of applying it to a self-driving car environment:\nThis is fundamental research for creating safe RL that can be understood by humans. It is a combination of ML and model-based techniques, and, due to the cost of the latter, should only be applied to small, safety-critical aspects of the overall application. The costs of computing the shield need to be outweighed by the benefit of added safety. As the research in this area is currently still very fundamental, we cannot yet apply it to self-driving cars, but we hope to be able to do so in the future.\n\nQ: CBMs \nA: A DT can be learned until exhaustion, meaning until it represents the dataset perfectly. Since we are aiming to capture a safety-critical element precisely, this is an important property. Of course, there is a chance of achieving similar accuracy with a CBM. When considering the naive approach of creating a CBM, one could use all the predicates the DT uses as concepts in the bottleneck of the CBM. In this case, this particular layer would basically be a list of all important information about the domain. We think that a DT is more explainable than such a list, but we would be interested in other ideas.\n\nQ: Human evaluation\nA: We agree that a study on the interpretability of DTs would be of interest for the topic. Nevertheless, the amount of literature on this topic indicates that there is at least some consensus that DTs are more explainable than lookup tables. Further, we think that such a study should be based on a solid methodology and warrants its own paper."}, {"review_id": "LlHcjEgtqr", "reviewer": "Reviewer_EFbP", "summary": "This paper introduces a novel framework for \"explainably safe\" RL that aims to build trust in safety-critical systems. The core problem addressed is that while \"shielding\" is a prominent technique to guarantee safety in RL by blocking unsafe actions, the shield's decision-making process is often as opaque as the RL agent's neural network policy. This opacity hinders human understanding and trust. The proposed solution provides human-interpretable explanations for the shield's decisions using a hierarchy of decision trees. The framework was evaluated on three RL benchmarks: Frozen Lake, Highway Cruise Control, and Boeing Taxinet. The results demonstrate that this hierarchical approach produces explanatory decision trees that are several orders of magnitude smaller and more compact than representing the entire shield as a single decision tree.", "questions": "Please see my questions in the weakness", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\nThis paper proposes a novel method to shield a policy while simultaneously providing decision tree-based explanations.\n\nThe definition of the risk of safety violation is reasonable and can be efficiently computed in tabular MDPs using dynamic programming.\n\nWeaknesses:\n\nDefinition 1, which is central to the algorithm—defining safe, critical, and dangerous states—can only be effectively estimated in tabular MDPs. When both the state and action spaces are continuous (e.g., in Mujoco environments), this estimation becomes challenging.\n\nThe experiments are limited to simple, simulated RL environments. The proposed approach does not generalize well to more complex scenarios where the state is represented by continuous vectors that lack clear physical interpretation.\n\nLack of Human-Subject Evaluation:  The paper's central motivation is to provide \"human-interpretable\" explanations and enhance trust. However, the evaluation is entirely quantitative, focusing on the size of the decision trees.  There is no user study or qualitative analysis to validate whether humans actually find these trees understandable, useful, or trust-inspiring.\n\nIn the related work section, the authors only discuss decision tree-based explanation methods. However, other explanation techniques have been proposed for RL, such as EDGE (1) and Statemask(2) approaches. These should be considered for a more comprehensive comparison.\n\n[1] Edge: Explaining deep reinforcement learning policies\n\n[2] Statemask: Explaining deep reinforcement learning through state mask", "was_revised": true, "final_justification": "I will keep my rating as I have discussed with author.", "rebuttal": "Thank you for your feedback and especially for the pointers to related work. We are happy to include them. In the following, we will respond to the raised concerns. \n\nQ: Continuous spaces\nA: Indeed, Model checking in continuous spaces is more challenging. However, this is an underlying problem for computing the shield, as the shield already contains this information. Therefore, we do not set out to solve this issue with our method.\n\nQ: Human evaluation\nA: We agree that a study on the interpretability of DTs would be of interest for the topic. Nevertheless, the amount of literature on this topic indicates that there is at least some consensus that DTs are more explainable than lookup tables. Further, we think that such a study should be based on a solid methodology and warrants its own paper."}, {"review_id": "UesVvoWhDc", "reviewer": "Reviewer_DwWi", "summary": "The authors introduce a novel method for explainability in Reinforcement Learning, whereby a user has access to a human-interpretable hierarchy of explanations about unsafe situations (as defined by hand) in terms of formally verifiable guarantees from a 'shield'. The core challenge they address is that while shields provide formal safety guarantees by blocking unsafe actions during RL training or execution, these shields are generally implemented as large lookup tables that are inherently opaque and difficult for humans to interpret. This opacity makes it hard to understand why particular actions are considered safe or unsafe in given states, limiting trust in safety-critical applications.\n\nIn the current work, the user has access to information about the risk categorisation of states, of action-level explanations about the current state and a tree showing why the specific action may be unsafe. This three-level hierarchical explanation system works with Level 1 providing a decision tree that categorizes states into four risk categories (from safe to unsafe) based on value iteration analysis, explaining why the current situation may be safety-critical. Level 2 provides runtime explanations that identify which actions are permitted in the current state and clarify why others are blocked by the shield. Then Level 3 provides execution trees that show the potential consequences of taking unsafe actions by showing traces that lead to safety violations even if safest actions are taken afterward.\n\nThe novelty lies in taking the traditional 'shields' approach for safe RL, and adding to it a Decision Tree format of case-based explanations. Rather than attempting to represent the entire shield as a single decision tree, the authors propose the hierarchical decomposition that gives compact, top-down explanations. This addresses the scalability problem that arises because shields permit agents to explore any safe action which undermines explainability. The method requires no additional information beyond what is already used for shield construction (which has to be hand-coded anyway), making it readily integrable into existing shielded RL pipelines.\n\nAfter the method is introduced, it is shown to provide useful outputs on three RL environments. This covers Frozen Lake, Highway, and Aircraft Taxiing, showing that the three layers of decision trees are several orders of magnitude smaller than the original shields while providing interpretable safety explanations. The results show consistent compression across different problem scales and complexity levels, with computation times remaining reasonable in these limited settings.", "questions": "Would you be able to look at human feedback on the explainability of the DT's in realistic settings? For whom would these actually be useful in a practical setting?\n\nHow does your method scale for much larger spaces than 50x50? For instance in pixel-based scenarios?\n\nHave you experimented with automatically defined predicates? The user-defined ones again are not going to be scalable in more complex settings.\n\nBeyond the tree-size, what metrics can you use for explainability?\n\nSome minor comments: \n\nLine 238: \"Dustance\"\nIn figure 3a, I would mark in the position which is used for the DT in figure 4c, otherwise it is hard to connect the two.", "limitations": "In the appendix, this is marked as yes, because \"We clearly highlight in the conclusion and experiments how our method de-\npends on well-defined predicates for larger problem instances.\". However, I believe that there are important limitations simply on the scaling to larger instances overall", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Overall, the paper is very well written, and even for a non-expert on Shields, and the formal verification of safety guarantees, it is easy to follow the method both of the shield and of the DT formalism that is used to create the human-interpretable explanations. \n\nThis method introduced is clearly a useful step forward, as explainability within RL, particularly within a context of formal verification, is clearly exceptionally important. The empirical work does indeed show good promise for these ideas.\n\nThe major weakness lies in the lack of discussion and exposition around more complex environments. The methods (and this goes for shields in general) require user-defined predicates, which in simple settings may be easy to define, but anything even slightly complex is likely to be very hard to define, and difficult to define predicates could lead to unwanted behaviour. \n\nThe argument made is that the DT's are a significantly compact representation compared to the shielding policies themselves, but there is still seemingly a linear scaling with the size of the environment for the first layer in the DT hierarchy. There is no discussion of what happens for larger action spaces, as all of the action spaces in the setup are minimal.\n\nBecause there is no experiment within any more complex settings, it is hard to know how this could scale in a way whereby, in situations where interpretability becomes important, it would remain human-interpretable. Having some sense of the computational complexity of the risk function for large state AND action spaces, or long horizons would be useful to know whether this method would be practically useful in critical and realistic situations. While there is some discussion about the scaling in frozen lake, this isn't sufficient to truly understand how useful this is in those realistic settings. In particular, how would this scale in pixel-based settings?", "was_revised": true, "final_justification": "I have read through the rebuttal to my questions and those of the other reviewers. I still have some concerns about the actual utility of this method to relatively large action spaces, but there has been some attempt at explaining how this could be overcome. In terms of who would use it, it still feels like for large scale practical situations it would not be useful, but I can see that perhaps in some areas of cybersecurity or situations where there is always going to be a low dimensional action/state space it could be used. I have thus upgraded from a 3 to a 4.", "rebuttal": "Thank you for the detailed review. In the following, we respond to questions and concerns.\n\nQ: Larger action and state spaces:\nA: The size of the action space only significantly influences the size of the Level 2 DTs. As these trees need to distinguish all different actions, more available actions would increase their size. However, this cannot be avoided as it is an underlying property of the problem and needs to be considered in every explanation.\nFor large state spaces, e.g., in pixel-based settings, the size of the DTs depends on the quality of predicates. In the worst case, splits would be performed on the level of x and y coordinates of the pixels, which would not be very explainable. If the predicates are adequate, we do not expect too many issues.\n\nQ: Who would use this method? \nA: Resilient RL systems in critical environments need the ability to have a human in the loop. Our method is an approach to make the safety-relevant aspect of an environment understandable for humans and allow them to better understand why actions are prohibited. This method can be used by engineers and scientists analyzing a safety-critical environment when applying RL. Further, it can be used to explain to stakeholders why some profitable actions are disallowed in some positions.\n\nQ: Automatically defined predicates:\nA: We have used dtControl to automatically synthesize predicates. One issue is that the automatically synthesised predicates may not necessarily be explainable, in which case we have to reject some explanations and take others. Nevertheless, our method can profit from new methods for the automated synthesis of predicates, which is a very active research area.\n\nQ: Metrics beyond tree size: \nA: We agree that tree size is only a proxy for evaluating explainability. However, we are unaware of a better fitting metric except for a human case study. Such a study on the interpretability of DTs would, of course, be interesting for the topic."}, {"review_id": "RGwT0Hzt61", "reviewer": "Reviewer_CBXv", "summary": "The paper tackles the challenge of interpretability in safe RL. Existing methods are based on shielding, which synthesizes a model-based “safety filter” that blocks unsafe actions, but the resulting lookup table policies are often as unclear as the neural policies they represent.  The authors propose representing the shield as a hierarchy of decision trees that give top-down, case-based explanations.  The method requires no additional information beyond standard shielding inputs. Experiments demonstrate the approach in three environments and highlight its ability to avoid unsafe states while offering interpretable explanations.", "questions": "- It seems like the abstraction process model checking may not scale well to large or high-dimensional MDPs. Any ideas on how this be addressed?\n - What happens when the abstraction function fails to distinguish semantically different but perceptually similar states?\n- Can this method be integrated with DRL agents in complex continuous domains, or does it rely on symbolic states?\n\nHuman-factor validation, request: run and report a small user study (e.g., 8 to 12 engineers or graduate students) measuring 1. time to identify an unsafe action and 2. Self reported confidence with the hierarchical DT vs the raw shield table. Score impact: Demonstrating statistically significant gains in comprehension or speed would directly strengthen the “interpretability” claim and raise Significance ALOT. \n\nPredicate dependency: clarify how many user defined predicates were needed per domain and run an ablation where only automatic CART splits (no handcrafted predicates) are allowed. Score impact: showing that useful explanations still emerge with minimal domain knowledge would make the approach broadly applicable, improving Significance. \n\n \n\nSafety threshold sensitivity. request: Vary the user defined risk threshold (e.g., 0.01 to 0.05 to  0.1) and plot shield permissiveness and  Level-1 tree depth, and learning agent performance. Showing graceful scaling would confirm robustness.  If explanations remain compact across thresholds, I would raise Quality.", "limitations": "No. The paper is open about several technical limitations, but the risk and uncertainty associated with the human factor are not addressed. The work assumes that compact decision-tree explanations automatically foster correct human understanding. At the very least, a brief discussion of potential misinterpretation (e.g., operators over-trusting a tree that is still an approximation) are essential in a paper on explainability.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n- Merges explainability and safety—a key concern for deploying RL in real-world systems.\n\n- The layered design (low-level learner + high-level symbolic checker) enables plugging into existing RL methods with minimal architectural changes.\n\n- Formally grounded method, the shield is derived with probabilistic model-checking (value iteration on an abstract MDP) and inherits the same risk‐bound guarantees as prior “probabilistic shields.” \n\n- Hierarchical explanations: the three-level tree hierarchy is generated directly from the shield’s risk analysis, ensuring fidelity to the original representation. \n\n - Experiments show that hierarchical trees are orders of magnitude smaller than raw lookup-table shields while preserving safety. \n- Illustrative figures, figures clearly walk the reader through each decision-tree level. \n\n- Because the method reuses what a shielding pipeline already computes (abstract MDP + risk table) and only adds a few calls to an off-the-shelf DT learner, it can be adopted with almost no extra engineering or hyperparameter tuning (this is, however, also a shortecoming, since the approach is quite straigthforward). \n\nWeaknesses: \n\n- The approach assumes access to domain-specific symbolic rules and an abstraction function.\n\n- The way safety layers are integrated within the RL training loop is not fully specified.\n\n- While explanations are shown qualitatively, no human studies or metrics assess explanation quality.\n\n- The text is sometimes hard to follow in terms of phrasing and wording\n\n- Each building block (probabilistic shielding, decision-tree explanations, hierarchical DTs) appears in prior literature. Thus, the contribution is best viewed as a novel integration and pruning strategy rather than a fundamentally new algorithmic insight. \n\n- Runtime-latency uncertainty is not addressed. Because Level 2 and Level 3 trees are generated on the fly, worst-case latency could matter in real-time systems. A bound or at least a discussion of the importance of this issue must be provided.", "was_revised": true, "final_justification": "​I have read the authors response and the other reviews. Based on the authors' acknowledgement of the importance of the issues I have raised, I have updated my score.\n​", "rebuttal": "We thank the reviewer for the suggestions on how to improve the paper and answer the posed questions:\n\nQ: Integration into the RL training loop: \nA: This approach is currently not integrated into the training loop. The idea is to bring the human into the loop when trying to understand why an action is blocked from the RL agent, and is one of the first approaches to explain the safety-critical aspects of RL to humans. Nevertheless, the question of how our method could benefit the training process is very interesting for future work.\n\nQ: Runtime latency: \nA: For convenience, as less crucial information, we included the computation time in the appendix (which, of course, the reviewers are not required to read). Depending on the size of the DT, the computation time is only a few seconds and thus not so interesting. As these DTs are only computed when a human tries to understand them, we think that a computation time of seconds is reasonable. If memory space is available, they can also be precomputed, or relevant information on the model checking calls for Level 3 can be stored, which improves latency.\nWe value the feedback and agree that it is not clearly discussed in the paper, and we will add it. \n\nQ: Model Checking in large or high-dimensional MDPs: \nA: Model checking is not scalable. This is a fundamental issue of shield computation and formal methods in general. It is also the reason why model-free techniques like reinforcement learning are applied to safety-critical use cases. To ensure their safety, the research area combining model-free techniques and model-based methods was created. Shields are an example of methods in this area. It is no longer required to compute an entire strategy for a safety-critical application, but only to check safety properties and integrate them into a shield.  Of course, shielding is only applied when the computation costs are outweighed by the requirement for a safe behavior.\n\nQ: Failure of the abstraction function: \nA: Of course, the MDP abstraction might be wrong, and, in consequence, also the shield. However, our aim is to explain the shield (no matter how inaccurate it is). On the contrary, if the shield is wrong due to a nonsensical abstraction, the explanation might pinpoint the fishy areas and mistakes in the abstraction function. DTs have been used exactly this way to validate models via explanations of the optimal policies. (e.g., [4])\n\nQ: Continuous domains: \nA: This paper focuses on discrete settings. However, in continuous settings, shields can be computed with controlled barrier certificates. Our method can also explain such shields. We will discuss this in the paper.\n\nQ: Human evaluation\nA: We agree that a study on the interpretability of DTs would be of interest for the topic. Nevertheless, the amount of literature on this topic indicates that there is at least some consensus that DTs are more explainable than lookup tables. Further, we think that such a study should extend beyond a few graduate students and be based on a solid methodology, which warrants its own paper."}]

Author rebuttal:

Meta-review: The reviewers found this paper to be a valuable contribution to the important intersection of safety and interpretability in RL. They were in agreement that the core idea—using a hierarchy of decision trees to explain the behavior of a safety shield—is novel, well-motivated, and clearly presented. The reviewers appreciated the method's formal grounding and the strong empirical evidence showing that the proposed explanations are orders of magnitude more compact than the original shield policies. While both reviewers raised valid concerns, primarily centered on the method's reliance on hand-coded symbolic rules and questions about its scalability to more complex, realistic environments, these points were seen as limitations of the current scope rather than fundamental flaws. The consensus is that the paper makes a clear and useful step forward in a challenging domain, and despite the identified avenues for future work, the current contribution is strong enough to warrant acceptance.
Decision: Accept (poster)