Title: Explainably Safe Reinforcement Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Reinforcement Learning

Shielding

Decision Tree

Safe AI

Explainability
Abstract: Trust in a decision-making system requires both safety guarantees and the ability to interpret and understand its behavior. This is particularly important for learned systems, whose decision-making processes are often highly opaque. Shielding is a prominent model-based technique for enforcing safety in reinforcement learning. However, because shields are automatically synthesized using rigorous formal methods, their decisions are often similarly difficult for humans to interpret. Recently, decision trees became customary to represent controllers and policies. However, since shields are inherently non-deterministic, their decision tree representations become too large to be explainable in practice. To address this challenge, we propose a novel approach for explainable safe RL that enhances trust by providing human-interpretable explanations of the shield's decisions. Our method represents the shielding policy as a hierarchy of decision trees, offering top-down, case-based explanations. At design time, we use a world model to analyze the safety risks of executing actions in given states. Based on this risk analysis, we construct both the shield and a high-level decision tree that classifies states into risk categories (safe, critical, dangerous, unsafe), providing an initial explanation of why a given situation may be safety-critical. At runtime, we generate localized decision trees that explain which actions are allowed and why others are deemed unsafe. Altogether, our method facilitates the explainability of the safety aspect in the safe-by-shielding reinforcement learning. Our framework requires no additional information beyond what is already used for shielding, incurs minimal overhead, and can be readily integrated into existing shielded RL pipelines. In our experiments, we compute explanations using decision trees that are several orders of magnitude smaller than the original shield.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tackles the challenge of interpretability in safe RL. Existing methods are based on shielding, which synthesizes a model-based “safety filter” that blocks unsafe actions, but the resulting lookup table policies are often as unclear as the neural policies they represent.  The authors propose representing the shield as a hierarchy of decision trees that give top-down, case-based explanations.  The method requires no additional information beyond standard shielding inputs. Experiments demonstrate the approach in three environments and highlight its ability to avoid unsafe states while offering interpretable explanations.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- Merges explainability and safety—a key concern for deploying RL in real-world systems.\n\n- The layered design (low-level learner + high-level symbolic checker) enables plugging into existing RL methods with minimal architectural changes.\n\n- Formally grounded method, the shield is derived with probabilistic model-checking (value iteration on an abstract MDP) and inherits the same risk‐bound guarantees as prior “probabilistic shields.” \n\n- Hierarchical explanations: the three-level tree hierarchy is generated directly from the shield’s risk analysis, ensuring fidelity to the original representation. \n\n - Experiments show that hierarchical trees are orders of magnitude smaller than raw lookup-table shields while preserving safety. \n- Illustrative figures, figures clearly walk the reader through each decision-tree level. \n\n- Because the method reuses what a shielding pipeline already computes (abstract MDP + risk table) and only adds a few calls to an off-the-shelf DT learner, it can be adopted with almost no extra engineering or hyperparameter tuning (this is, however, also a shortecoming, since the approach is quite straigthforward). \n\nWeaknesses: \n\n- The approach assumes access to domain-specific symbolic rules and an abstraction function.\n\n- The way safety layers are integrated within the RL training loop is not fully specified.\n\n- While explanations are shown qualitatively, no human studies or metrics assess explanation quality.\n\n- The text is sometimes hard to follow in terms of phrasing and wording\n\n- Each building block (probabilistic shielding, decision-tree explanations, hierarchical DTs) appears in prior literature. Thus, the contribution is best viewed as a novel integration and pruning strategy rather than a fundamentally new algorithmic insight. \n\n- Runtime-latency uncertainty is not addressed. Because Level 2 and Level 3 trees are generated on the fly, worst-case latency could matter in real-time systems. A bound or at least a discussion of the importance of this issue must be provided.\n\nQUESTIONS:\n- It seems like the abstraction process model checking may not scale well to large or high-dimensional MDPs. Any ideas on how this be addressed?\n - What happens when the abstraction function fails to distinguish semantically different but perceptually similar states?\n- Can this method be integrated with DRL agents in complex continuous domains, or does it rely on symbolic states?\n\nHuman-factor validation, request: run and report a small user study (e.g., 8 to 12 engineers or graduate students) measuring 1. time to identify an unsafe action and 2. Self reported confidence with the hierarchical DT vs the raw shield table. Score impact: Demonstrating statistically significant gains in comprehension or speed would directly strengthen the “interpretability” claim and raise Significance ALOT. \n\nPredicate dependency: clarify how many user defined predicates were needed per domain and run an ablation where only automatic CART splits (no handcrafted predicates) are allowed. Score impact: showing that useful explanations still emerge with minimal domain knowledge would make the approach broadly applicable, improving Significance. \n\n \n\nSafety threshold sensitivity. request: Vary the user defined risk threshold (e.g., 0.01 to 0.05 to  0.1) and plot shield permissiveness and  Level-1 tree depth, and learning agent performance. Showing graceful scaling would confirm robustness.  If explanations remain compact across thresholds, I would raise Quality.\n\nLIMITATIONS:\nNo. The paper is open about several technical limitations, but the risk and uncertainty associated with the human factor are not addressed. The work assumes that compact decision-tree explanations automatically foster correct human understanding. At the very least, a brief discussion of potential misinterpretation (e.g., operators over-trusting a tree that is still an approximation) are essential in a paper on explainability.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce a novel method for explainability in Reinforcement Learning, whereby a user has access to a human-interpretable hierarchy of explanations about unsafe situations (as defined by hand) in terms of formally verifiable guarantees from a 'shield'. The core challenge they address is that while shields provide formal safety guarantees by blocking unsafe actions during RL training or execution, these shields are generally implemented as large lookup tables that are inherently opaque and difficult for humans to interpret. This opacity makes it hard to understand why particular actions are considered safe or unsafe in given states, limiting trust in safety-critical applications.\n\nIn the current work, the user has access to information about the risk categorisation of states, of action-level explanations about the current state and a tree showing why the specific action may be unsafe. This three-level hierarchical explanation system works with Level 1 providing a decision tree that categorizes states into four risk categories (from safe to unsafe) based on value iteration analysis, explaining why the current situation may be safety-critical. Level 2 provides runtime explanations that identify which actions are permitted in the current state and clarify why others are blocked by the shield. Then Level 3 provides execution trees that show the potential consequences of taking unsafe actions by showing traces that lead to safety violations even if safest actions are taken afterward.\n\nThe novelty lies in taking the traditional 'shields' approach for safe RL, and adding to it a Decision Tree format of case-based explanations. Rather than attempting to represent the entire shield as a single decision tree, the authors propose the hierarchical decomposition that gives compact, top-down explanations. This addresses the scalability problem that arises because shields permit agents to explore any safe action which undermines explainability. The method requires no additional information beyond what is already used for shield construction (which has to be hand-coded anyway), making it readily integrable into existing shielded RL pipelines.\n\nAfter the method is introduced, it is shown to provide useful outputs on three RL environments. This covers Frozen Lake, Highway, and Aircraft Taxiing, showing that the three layers of decision trees are several orders of magnitude smaller than the original shields while providing interpretable safety explanations. The results show consistent compression across different problem scales and complexity levels, with computation times remaining reasonable in these limited settings.\n\nSTRENGTHS AND WEAKNESSES:\nOverall, the paper is very well written, and even for a non-expert on Shields, and the formal verification of safety guarantees, it is easy to follow the method both of the shield and of the DT formalism that is used to create the human-interpretable explanations. \n\nThis method introduced is clearly a useful step forward, as explainability within RL, particularly within a context of formal verification, is clearly exceptionally important. The empirical work does indeed show good promise for these ideas.\n\nThe major weakness lies in the lack of discussion and exposition around more complex environments. The methods (and this goes for shields in general) require user-defined predicates, which in simple settings may be easy to define, but anything even slightly complex is likely to be very hard to define, and difficult to define predicates could lead to unwanted behaviour. \n\nThe argument made is that the DT's are a significantly compact representation compared to the shielding policies themselves, but there is still seemingly a linear scaling with the size of the environment for the first layer in the DT hierarchy. There is no discussion of what happens for larger action spaces, as all of the action spaces in the setup are minimal.\n\nBecause there is no experiment within any more complex settings, it is hard to know how this could scale in a way whereby, in situations where interpretability becomes important, it would remain human-interpretable. Having some sense of the computational complexity of the risk function for large state AND action spaces, or long horizons would be useful to know whether this method would be practically useful in critical and realistic situations. While there is some discussion about the scaling in frozen lake, this isn't sufficient to truly understand how useful this is in those realistic settings. In particular, how would this scale in pixel-based settings?\n\nQUESTIONS:\nWould you be able to look at human feedback on the explainability of the DT's in realistic settings? For whom would these actually be useful in a practical setting?\n\nHow does your method scale for much larger spaces than 50x50? For instance in pixel-based scenarios?\n\nHave you experimented with automatically defined predicates? The user-defined ones again are not going to be scalable in more complex settings.\n\nBeyond the tree-size, what metrics can you use for explainability?\n\nSome minor comments: \n\nLine 238: \"Dustance\"\nIn figure 3a, I would mark in the position which is used for the DT in figure 4c, otherwise it is hard to connect the two.\n\nLIMITATIONS:\nIn the appendix, this is marked as yes, because \"We clearly highlight in the conclusion and experiments how our method de-\npends on well-defined predicates for larger problem instances.\". However, I believe that there are important limitations simply on the scaling to larger instances overall", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel framework for \"explainably safe\" RL that aims to build trust in safety-critical systems. The core problem addressed is that while \"shielding\" is a prominent technique to guarantee safety in RL by blocking unsafe actions, the shield's decision-making process is often as opaque as the RL agent's neural network policy. This opacity hinders human understanding and trust. The proposed solution provides human-interpretable explanations for the shield's decisions using a hierarchy of decision trees. The framework was evaluated on three RL benchmarks: Frozen Lake, Highway Cruise Control, and Boeing Taxinet. The results demonstrate that this hierarchical approach produces explanatory decision trees that are several orders of magnitude smaller and more compact than representing the entire shield as a single decision tree.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThis paper proposes a novel method to shield a policy while simultaneously providing decision tree-based explanations.\n\nThe definition of the risk of safety violation is reasonable and can be efficiently computed in tabular MDPs using dynamic programming.\n\nWeaknesses:\n\nDefinition 1, which is central to the algorithm—defining safe, critical, and dangerous states—can only be effectively estimated in tabular MDPs. When both the state and action spaces are continuous (e.g., in Mujoco environments), this estimation becomes challenging.\n\nThe experiments are limited to simple, simulated RL environments. The proposed approach does not generalize well to more complex scenarios where the state is represented by continuous vectors that lack clear physical interpretation.\n\nLack of Human-Subject Evaluation:  The paper's central motivation is to provide \"human-interpretable\" explanations and enhance trust. However, the evaluation is entirely quantitative, focusing on the size of the decision trees.  There is no user study or qualitative analysis to validate whether humans actually find these trees understandable, useful, or trust-inspiring.\n\nIn the related work section, the authors only discuss decision tree-based explanation methods. However, other explanation techniques have been proposed for RL, such as EDGE (1) and Statemask(2) approaches. These should be considered for a more comprehensive comparison.\n\n[1] Edge: Explaining deep reinforcement learning policies\n\n[2] Statemask: Explaining deep reinforcement learning through state mask\n\nQUESTIONS:\nPlease see my questions in the weakness\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nA shield in RL is a component of an RL agent that prevents it from performing any action that has been deemed to be unsafe by the model designer. This paper proposes that by explaining an RL agent's shield, that agent can be used in a safe and trustworthy manner. The contribution of this paper is a shield *explainer*. The shield explainer works by explaining the shield's decisions at different levels of granularity corresponding to levels of execution and safety. For example, they write that a \"level 1\" tree classifies actions as safe or unsafe, a \"level 2\" tree is used to explain which actions the RL agent can choose safely and a \"level 3\" tree is used to explain why an unsafe deemed unsafe would be unsafe to execute.\n\nSTRENGTHS AND WEAKNESSES:\n- The idea of this paper (explaining an RL shield) is neat.\n- I enjoyed reading this paper. It was easy to follow and very clear. I appreciated the level of detail that the authors described their methodology at. The structure of the paper and use of formal definitions helped my reading. \n- I think practitioners and researchers will want to use this method if it is open source and can be easily integrated into their workflows.\n- As I will mention in the next section, explaining RL agents using DTs is not new, but I do think the specific use case here where a hierarchy of explanations of a shield is constructed using 3 DTs is novel and clever idea. \n\n## Weaknesses\n\n- The authors should give citations/evidence for why case-based explanations are good explanations. \n- I am not convinced that, in general, the 3 trees used in explanations would be of sufficiently low complexity so that a user can understand the shield. I would like more detail on how the DTs were computed. Are they trained to be sparse to help ensure that they can be understood? Sparsity of the trees is especially important for this paper because we have to look at 3 of them. \n- Explaining RL agents using decision trees is not a new idea. I think VIPER was one of the first to do it and I have seen other works use the idea since the publication of VIPER. \n- Please further justify the choice of explaining the shield with DTs. Could concept bottleneck models (CBMs) be used instead, for example?\n- No human evaluation study is given\n\nQUESTIONS:\n- Is generally practical to create a shield for an RL agent? Obviously the applicability of this paper depends on how easy it is to create shields. \n- Building off of the first question, how hard would it be to extend your method from working on the farama foundation environment to an actual self-driving car context?\n- Please further justify the choice of explaining the shield with DTs. Could concept bottle neck models (CBMs) be used instead, for example?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}