Title: Approximating Shapley Explanations in Reinforcement Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Reinforcement learning

Shapley values

explainable artificial intelligence

explainable reinforcement learning

feature-based explanations
Abstract: Reinforcement learning has achieved remarkable success in complex decision-making environments, yet its lack of transparency limits its deployment in practice, especially in safety-critical settings. Shapley values from cooperative game theory provide a principled framework for explaining reinforcement learning; however, the computational cost of Shapley explanations is an obstacle for their use. We introduce FastSVERL, a scalable method for explaining reinforcement learning by approximating Shapley values. FastSVERL is designed to handle the unique challenges of reinforcement learning, including temporal dependencies across multi-step trajectories, learning from off-policy data, and adapting to evolving agent behaviours in real time. FastSVERL introduces a practical, scalable approach for principled and rigourous interpretability in reinforcement learning.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies Shapley value explanations in RL and proposes a new efficient approximation method. Exactly computing SVs is known to be a computationally costly procedure in high-dimensional domains hindering practical scalability. The problem has been extensively addressed in the SL domain. However, there is no method tailored to the unique challenges the RL framework poses, such as temporal dependencies and coverage issues when only offline/off-policy data are available. The authors propose FastSVERL: a parametric approach for approximating state feature contributions (to policy action probs; expected and realized values) via SV in an efficient way.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- The paper provides a comprehensive summary of SVERL [1,2], allowing readers unfamiliar with this work to easily follow along. In general, the paper is very well-written and easy to read.\n- The problem addressed is nicely posed and well motivated.\n- I found the proposed approach to be reasonable, albeit most components are derived from existing work. I found the single conditioned policy idea (line 203) a nice touch.\n\n\n**Weaknesses**\n\n- The main weak point of this paper is the limited empirical evaluation (main reason for giving a low score). I would typically expect from such a paper to conduct extensive experiments in more complex (\"real-world\") environments. Furthermore, since the authors chose to use simpler simulated benchmarks, where ground truth attributions can be computed, the use of more baseline methods for comparison are needed. Such baseline methods would include SL techniques mentioned in the background section.\n- It was not clear from the write-up how some of the challenges recognized as unique to the RL method were explicitly addressed by the FastSVERL approach (see my questions below).\n- [Minor] since there are works in (MA)RL using SV for different purposes than the one considered in the paper, it would good to specify ealrier in the paper (even from appendix) the use-case of focus.\n\nQUESTIONS:\nIt was not clear to me how temporal dependencies, characteristic of RL environments, were explictly addressed by your framework.\n\nLIMITATIONS:\nMentioned", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a scalable method for approximating Shapley values for RL policies. This method can be used to quantify the influence of individual state features on the agent's behavior, its performance, and its value estimates. The paper also addresses two key challenges: computing Shapley values from off-policy data, and doing so while the policy is being trained.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. The paper tackles the important problem of interpreting RL policies. It starts from a computationally expensive method and proposes an approximation that improves scalability.\n\n2. The experimental setting is sound, and the results are promising. The accuracy of the approach is compared against the exact method, showing low MSE. Moreover, the method is adapted for use with offline data collected from a different policy, and for explaining continually evolving policies, both of which are important use cases.\n\n**Weaknesses:**\n\n1. The introduction and background sections argue that RL poses additional challenges compared to supervised learning, particularly due to the temporal dependencies between state features and returns. However, this concern primarily applies when measuring the influence of state features on performance, not when measuring their influence on behavior or value estimation. The proposed approach for the latter two cases appears equivalent to existing supervised learning methods described in the background section. This raises the question of why these are presented as novel contributions if existing techniques can be directly applied.\n\n2. Section 3.2 is somewhat difficult to follow, particularly the discussion of the performance characteristic function. See Questions 1–3 below.\n\n3. Section 5 introduces a new method that, according to Figure 3c, appears more accurate and more scalable than the previously proposed method. It is unclear why this method is only introduced toward the end of the paper and not analyzed in greater depth earlier. This is especially puzzling given that, in line 180, the Monte Carlo estimation approach is dismissed for lacking scalability and generalization across different feature subsets.\n\n4. Aside from demonstrating accuracy on a toy problem where Shapley values can be computed exactly, the paper lacks experiments that showcase the method’s usefulness for explaining the behavior of RL policies in high-dimensional and complex environments.\n\nQUESTIONS:\n1. *Section 3.2:* How is the single-conditioned policy \"defined\"? I understand it as a policy that takes the initial action based solely on the features in $\\mathcal{C}$ and then follows the actual policy afterward. But how are the probabilities computed for that first action? Are they approximated using a model trained with the loss defined in Equation 12?\n\n2. *Section 3.2:* How is $\\mathcal{B}$ obtained? The text states, \"we uniformly sample $s_e$ and $\\mathcal{C}$ at each decision stage.\" What does this mean? Does it imply the environment can be reset to any arbitrary state?\n\n3. *Section 3.2:* Could you elaborate on how the off-policy (Equation 14) and on-policy (Equation 15) versions work? This is confusing, as in both cases $s'$ and $a$ seem to be obtained while following the current policy. Additionally, Equation 15 is described as SARSA-style (an on-policy method), yet is presented as off-policy, which is contradictory.\n\n4. *Section 3.2* presents both an on-policy and off-policy method. Then, in Section 4, the challenge of learning the Shapley approximation from off-policy data is raised again, this time proposing importance sampling to reweight the loss and address distribution shift. Why is the first off-policy approach insufficient in this context? Is this instead referring to **offline** learning rather than off-policy learning?\n\n5. Could you provide intuition for why the method proposed in Section 5 converges faster than the one that uses the exact behavior characteristic function?\n\nLIMITATIONS:\nAlthough there is no specific section that explicitly discusses the limitations, some limitations are mentioned throughout the paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe submission studies Shapley values for explaining reinforcement learning models. Similar to FastSHAP, the submission proposes FastSVERL to amortize estimation of Shapley values for reinforcement learning as introduced in SVERL. The submission comprehensively discusses amortization for both behavior, performance, and value estimation. Experiments on both synthetic and real-world datasets show the behavior of the proposed methods and discuss its limitations and potential future research directions.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. Explainability for reinforcement learning is a challenging setting that is worth exploring\n2. Amortization has proven to be a successful strategy in the past\n3. Experimental analysis is thorough\n\n**Weaknesses**\n\n1. Presentation could do a better job at separating behavior, performance, and evaluation estimation, which is hard to follow\n2. Certain claims are hand-wavy and could be clarified\n3. Sampling strategies could be given more space\n\nI have a few clarifying questions and I am looking forward to discussing with the authors!\n\nQUESTIONS:\n* Several times throughout the paper it is stated that \"Shapley value explanations do not generalize\" hence are \"inefficient for repeated explanations\". Could the authors clarify what they mean? What does it mean for an explanation to \"generalize\"?\n\n* Lines 202-204: I have a general question about explaining the performance of a model. Why is the explained state $s_e$ always the start state? How could one explain the performance of the model at any state $s$ of a trajectory which may not be the first one? Does the modified policy take into account that the trajectory may pass through state $s_e$ again later? What would happen at that point? \n\n* Could the authors clarify what they mean by \"earlier policies\"? Does this assume access to the model during training to collect observations? This is an important consideration for interpretability because usually we assume being given a fixed predictive model after training is completed. If FastSVERL requires access to trajectories during training, it should be made explicit and its implications discussed.\n\n* Could the authors expand on how ground-truth for MSE evaluation is obtained? Does \"exact values\" refer to exact Shapley values? Wouldn't these require expectation over an unknown distribution to mask the inputs?\n\n* Lines 325-326: I am missing how it can be that the models start with low error if the policy is random. If the policy is random, then there is \"nothing to explain\", so what are the models learning?\n\n* Does the sampling strategy apply to performance and value estimation as well?\n\n* It might be interesting to expand on the sampling strategy, since it seems to outperform the models. After reading the last paragraph in Sec. 5, it was unclear to me whether the message of the submission is that sampling is better than training models, which can be expensive and difficult to do in practice.\n\n---\n\n**Minor comments**\n\n* Abstract: \"real-time\" estimation feels outside the scope of the submission. Experiments do not discuss latency or role of real-time estimation for downstream tasks.\n* The difference between behavior, performance, and value could be spelled out more clearly for readers coming from the interpretability community who may not be familiar with reinforcement learning.\n* Lines 40-43 are too technical and hard to understand at this point in the paper.\n* Line 78: throughout the paper, the cardinality of the state space $\\lvert \\mathcal{X} \\rvert$ is mentioned several times. Is the state space assumed to be finite in this work? In several applications, this may not be finite. Same question about $\\mathcal{S}$\n* Line 82: it is unclear what \"state-of-the-art\" means in this context.\n* Line 86: can $p(C)$ be any distribution or should it be a uniform to retrieve the conditional expectation?\n* Eq. (8) typo in conditioning, should be $S_t = s^C$? Also $G_t$ is undefined\n* Line 213: reference for \"DQN-style loss\"?\n* Eq. (14): $V$ and $V'$ are also undefined\n* Line 337: it is unclear what \"Shapley models\" refer to\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nShapley values are widely used to provide interpretability to RL behaviors, however, their exponential computation cost makes them difficult to compute in many real problems. \nThe manuscript introduces FastSVERL, a scallable method for approximating Shapley values in the context of RL. It takes ideas from FastSHAP and extends them to the RL context. It particularly deals with sequential, policy-based RL environments, handling temporal dependencies and off-policy learning.\n\nSTRENGTHS AND WEAKNESSES:\nSTRENGTHS:\n- The proposal is very well motivated (need for practical scalable ways of providing interpretability in RL).\n- Specifically deals with RL challenges such as temporal dependencies, evolving behaviours, and learning from off-policy data.\n- The performed experimentation (in the supplementary material) is very solid, analyzing accuracy, efficiency and scalability.\n\nWEAKNESSES:\n- While the goal is to provide explainability to complex RL scenarios, the given experiments are only on moderate-complexity tractable problems.\n- As reported by the authors, the manuscript establishes a foundation for scalable Shapley-based explanation in RL, but the practical deployment is yet to be fully done.\n- The proposal relies on parametric models, whose generalization capabilities can be limited.\n\nQUESTIONS:\n- How does the method perform in large-scale, high-dimensional RL benchmarks beyond the toy domains used?\n- What is the overhead introduced by calculating the parametric model? What is its O shape?\n- Can FastSVERL be applied to any RL problem? Are there any requirements in terms of discrete/continuous domains? Continuous or episodic? Fully observable or partially observable?...\n- Can parametric models be computed in all cases? What amount of data is needed?\n\nLIMITATIONS:\nAt the end of the paper, the authors state the main limitation of this work: its practical deployment in real-world to test its robustness is missing. \nThere is no potential negative societal impact of this work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a framework for computing approximate shapley values to explain reinforcement learning policies. In particular, this work uses parametric models to amortize the approximation cost of shapley values across states and features which is compatible with both on-policy and off-policy methods. Explanations are generated to explain an agenet's behavior (policy), performance (reward), and value (future reward).\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* Fast and efficient shapley-based explanations can be useful for a wide-variety of tasks, particularly given the heavy application of RL to many real-world problems.\n\n* The proposed method supports both off-policy and on-policy learning, the latter of which is particularly useful as it supports explanations for non-stationary policies. Explaining how policies change during the course of training could be very useful.\n\n\nWeaknesses:\n* Experiments are only performed over very simple environments (a 2x4 Gridworld in the Appendix, and the Mastermind code-breaking game with codes of up to length 3). While I understand that the main results in the paper used small environments so as to support exact value calculation, this means that the *scalability* of the method is not tested. This is particularly relevant given that the paper claims that FastSVERL is suitable for real-time and real-world problems, yet the evaluation is only conducted over small toy problems that are not at all representative of real-world ones. Does this method scale to more complex environments/policies with respect to both computation time and approximation accuracy?\n\n* No mention is given to bias or fairness of the approximations. Are the estimators used in this paper unbiased?\n\n* There is little analysis on the single-sample approximation of characteristic values. I would think that this would increase variance given that these are Monte Carlo samples. Is this the case? If not, why?\n\nQUESTIONS:\n1. Are the estimators used in this paper unbiased?\n\n2. There is little analysis on the single-sample approximation of characteristic values. I would think that this would increase variance given that these are Monte Carlo samples. Is this the case? If not, why?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}