Approximating Shapley Explanations in Reinforcement Learning
Authors: Daniel Beechey
Özgür Şimşek
Venue: NeurIPS 2025 poster ()
Keywords: Reinforcement learning
Shapley values
explainable artificial intelligence
explainable reinforcement learning
feature-based explanations
Primary area: reinforcement_learning
TL;DR: FastSVERL provides a practical and scalable approach for principled, rigourous interpretability in reinforcement learning.

Abstract: Reinforcement learning has achieved remarkable success in complex decision-making environments, yet its lack of transparency limits its deployment in practice, especially in safety-critical settings. Shapley values from cooperative game theory provide a principled framework for explaining reinforcement learning; however, the computational cost of Shapley explanations is an obstacle for their use. We introduce FastSVERL, a scalable method for explaining reinforcement learning by approximating Shapley values. FastSVERL is designed to handle the unique challenges of reinforcement learning, including temporal dependencies across multi-step trajectories, learning from off-policy data, and adapting to evolving agent behaviours in real time. FastSVERL introduces a practical, scalable approach for principled and rigourous interpretability in reinforcement learning.

Approximating Shapley Explanations in Reinforcement Learning
Daniel Beechey University of Bath United Kingdom j @jeb20@bath. ac. uk
Özgür Şimşek University of Bath United Kingdom
s simsekkbath. ac uk
Abstract
Reinforcement learning has achieved remarkable success in complex decision- making environments, yet its lack of transparency limits its deployment in practice, especially in safety-critic settings. Shapley values from cooperative game theory provide principled framework for explaining reinforcement learning; however, the computational of Shapley explanations introduce FastSVERI scalable method for explaining reinforcement learning by approximating Shapley values. FastSVERL is designed handle the unique challenges of reinforcement learning, including temporal dependencies across multi-step trajectories, learning from off-policy data, and adapting to evolving agent behaviours in real time. FastSVERL introduces practical, scalable approach for principled and rigourous interpretability in reinforcement learning.
We obstacle for their
Introduction
Reinforcement learning has achieved remarkable success in complex decision-makee environments [,255, but the lack of transparency in an agent's decisions limits its deployment in practice especially safety-critical settings. While various interpretability methods have been proposed 9 19], they often lack theoretical guarantees. A principled and promising approach [ 2 is the use of Shapley values 26 attribute the influence of observes in its environment-oon on the agent's 's behaviour, outcomes, and predictions. Grounding these attributions in Shapley values provides a principled method for fair, transparent, and consistent credit assignment. One significant difficulty is that the computational cost of the Shapley approach scales exponentially with the number of features, making its application impractical for most real-world problems. There is therefore strong need to develop scalable approximation methods that enable practical Shapley- interpretability in reinforcement learning.
values that describe what an agent
In supervised learning, scalable approximations of Shapley values are achieved through two primary methods: sampling over feature subsets (29
and parametric models that amortise com- putation across inputs S 12 7 These methods are designed primarily for single-step predictions, whereas reinforcement learning introduces temporal dependencies across sequences of decisions. For example, explaining an agent's 's outcomes requires attributing influence over its expected return, which accumulates across possibly an infinite number of decisions. Furthermore, (1) as an agent's behaviour evolves, the explanations must evolve with it, and (2) when environment interaction is limited, explanations must be approximated from off off-policy data aata-gathered, by agents acting differently from the agent being explained. Addressing these challenges requires approximation methods tailored to the multi stule interactive, and evolving nature reinforcement lcarning. We develop such an approach here, which we call FastSVERL
FastS VERL code is available at:
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

We introduce a scalable parametric learning method for approximating Shapley values in reinforce- ment learning. The proposed upproach is designed to handle the temporal dependencies inherent to reinforcement learning, amortising Shapley value estimation across multi-step trajectories to explain agent behaviour, outcomes, and predictions efficiently. We address the practical constraints introduced in real-world reinforcement learning by considering how to explain policies from off-policy data whilst adapting them to evolving agent behaviours. The comprchensive, model-based framework we introduce provides a solid foundation that the research community can build upon. We present one such promising extension in Section [] which preliminary findings suggest halves computational costs, improves estimation accuracy, and extends naturally to supervised learning.
These contributions position FastSVERL as a principled, rigourous, and scalable solution for inter- pretability in the practice of reinforcement learning.
2 Background
Reinforcement learning [30] models an agent interacting with its environment to achieve desired outcomes. It is commonly formalised as a Markov Decision Process (MDP), defined by the 5-tuple 7.)). The environment is initialised in state So 8, following initial state distribution d(s) = Pr(Sg = s). At each decision stage 2 0, the agent observes the environment's current state S, E 8, which we assume can be decomposed into features (S.,..., SF indexed by F = (1.....m): the agent selects an action A, E AA: the environment then transitions to state S E 8 according to the transition kernel p(s' s, a) == Pr(S,+1 = S. = 8, A, a). and emits a scalar reward Re+1 E E with expected value r(s, a) = B[ S; = s, = a]. A policy : + (AA) maps each state distribution over actions. The agent's objective is to learn policy that maximises the discounted sum of future rewards, known as expected return, which is captured by the state value function:
(s) = [G = E,
x Re+k+1 St =
(I)
where y E [0, 1] is a discount factor that weights future rewards. An optimal policy is a policy that maximises v (s) for all E S. In practice, reinforcement learning agents use algorithms such as Deep Q-Networks (DQN) [17] and Proximal Policy Optimisation (PPO) [24] to learn near-optimal policies and value functions using neural networks. However, such methods generally do not explain the process behind an agent's behaviour, even though such explanations are often needed in practice.
Explaining supervised learning with Shapley values. One approach to interpreting supervised learning models attribute their predictions to the influence of individual input features [28 15
Consider model f X + R and an input = (x ,x',x" ......"......................... xx")
E X defined over set of features F [1, (1...., n... To quantify how features contribute to the prediction f'xx), we can consider how the model's output changes when some features are unknown. A characteristic function [=(C) (28
represents the model's expected prediction when only the features in subset e S I are known:
(2(e) E [S(X)
x --===0 =1],
where d denotes a subset of values far : E e). In particular, the difference fz(3) fz (0) quantifies the total change in prediction when all features are known versus when none are known. A principled way to distribute this quantity among the features is via Shapley values [26]. which assign credit to each feature based on its mean marginal contribution across all possible subsets of features:
lel! ((3) |el - 1)! (Su fc()))
('sa))
Σ ECFVA
F
Shapley values give the unique solution that satisfies four axioms formalising the notion of fairly attributing a given prediction among features. Whilst they have strong theoretical guarantees, the cost of computing Shapley values grows exponentially with the number of features. Each characteristic value is expectation over the input space X, with complexity [[[]).. Shapley values sum over 2" such values, one for each subset of features, giving a total cost of 02" -I])) per input. As a result, the characteristic values in Equation and the Shapley value sum in Equation must be approximated in high-dimensions domains.
Approximating characteristic values. One common approach to approximating the characteristic
function (e) is to learn parametric model f(z e; B8) that maps an input æ, with features not in e

---

replaced by value outside the support of X, to an approximate characteristic value [8 The model is trained minimise the expected squared error:
p(a) -(z) e
( = E
(4)
where p(æ) is the data distribution and p(e) any distribution defined over all feature subsets. This loss cannot reach zero: for sampled C, different inputs x sharing masked representations 2 may correspond to different f(z) values. The model cannot recover every target with only the features in e, and instead learns to predict their mean-name namely, the characteristic value f=(
Approximating the Shapley value summation. We now describe how approximate the Shapley summation in Equation assuming access to (possibly approximate) characteristic function (e)).. One approach is learn parametric model ő(2;; 0) X IR that predicts the Shapley values for all features of an input x. This is the approach taken by FastSHAP [2], which trains model ő(æ; 0) using an equivalent characterisation of the Shapley values for fixed input as the solution to constrained least squares problem:
(fa))ves = arg min E fa(e)-fuu0)) -f0c)
fz(0)
- of
s.t. Σ Φ = fa() - (2)) (5)
hes Cxp(C)
iE
Elliciency constraint
Here, the distribution p(C) samples subsets in proportion to the combinatorial weights used in the Shapley value formula:
n
pl
(6)
((1)
[el (n lel"
for e C I where 0 < |e| < The efficiency constraint reflects the requirement that the total contribution across features must equal the change in prediction from observing all features versus none. FastSHAP implements this characterisation by training the model (0x; 0) to minimise the expected loss of the least squares objective the data distribution p(x)):
0))
E p(z) P(C)
(L(e) - (0)
Σ
(z;
ZE
Since the model is unconstrained, correction term is added post hoc to enforce the efficiency constraint in Equation 5
d'(f)) x(;))))))))))))))))))))))))))))) - - Σ
-
JEB
Explaining reinforcement learning with Shapley values. Unlike supervised learning, which typ- ically focuses single predictions, reinforcement learning concerns how agent sequentially interacts with its environment to achieve desired outcomes. To understand these long-term interac- tions, it is helpful distinguish between three explanatory targets (2]: behaviour (how agent acts), outcome (the consequences of those actions), and prediction (estimates of those outcomes). SVERL (Shapley Values for Explaining Reinforcement Learning) [ 2 formalises these elements by attribut- ing them to state features. While outcomes can refer to many consequences of an agent's 's behaviour, SVERI focuses expected return, standard formalisation of outcome in reinforcement learning. For each element, SVERI defines a characteristic function over subsets of features e C F, which measures how the agent's action, expected return, or prediction of expected return changes when only the features in are observed. By evaluating how these measurements change across subsets of features, SVERL computes Shapley values to attribute each explanatory element to individual features. We now describe each characteristic function and explanation, beginning with behaviour.
SVERL explains behaviour by measuring how each feature influences the agent's 's action choice. The behaviour characteristic function # (C) is defined as the expected probability of selecting action a in state when only the features in e are known:
za (e)
T (S. a) ge = Σ p
s ) (s, (
(7)
SES+
where 8+ is the set of non-terminal states. The distribution y=(S 5e = e) is the conditional steady-state distribution: the probability that an agent following policy π in state s, given that
3

---

observes s². Shapley values (Equation are computed over the behaviour characteristic function to attribute the change in action probability when all features are known versus none, to the individual features of state
n # (5( (3) T.. (0).
SVERI explains outcomes by measuring how each feature contributes to the agent's expected return. The outcome characteristic function ū. (e) is defined as the expected return received from state when the agent's 's policy has access only to the features s
(e) = B,, [G+ St s]
(8)
where j is modified policy that selects actions using the behaviour characteristic function (C) when features are unknown in state and follows the original policy т elsewhere. Shapley values computed over the outcome characteristic function attribute the change in expected return across all features, 05 (5) 0. (0), to the individual features of state s.
SVERL explains prediction by measuring how each feature contributes the agent's, s, an observer's, s, prediction of the agent's is expected return 1" (s), which estimates the true expected return v" (s). The prediction characteristic function 65 (C) function is defined the predicted expected return from state s when only the features in e are known:
0- (e) = E i (S) se = = = p [
(9)
SES+
Shapley values computed over this function attribute the change in predicted expected return across all features, (3) 6- (0). to the individual features of state s.
3 Approximating Shapley values in reinforcement learning
SVERL provides rigorous and comprehensive framework for explaining reinforcement learning agents, but exact computation is impractical for most real-world problems. Each characteristic value an expectation over the state space 8, and Shapley values sum these values over all possible combinations of features F, resulting in total cost of 0 [8]) per explanation. In high- dimensional settings, exact computation is infeasible, necessitating scalable approximation methods for both (1) the Shapley value and (2) the characteristic functions that underpin each explanation.
3.1 Approximating the Shapley value summation in reinforcement learning
We begin by considering how to approximate the Shapley value sum (Equation 3). Given a character- istic function for each type of explanation, the computation proceeds identically, allowing a single approximation method to be used across all three explanation types.
In supervised learning, a common approach to estimate the Shapley value sum is via Monte Carlo sampling, averaging marginal contributions over randomly selected subsets C C 3 (2 15]. However, such sampling estimates would not generalise across states and would need to be recomputed from scratch whenever the agent's 's policy changes. As a result, this approach would be inefficient for repeated explanations of evolving policies. We therefore do not pursue it here. Instead, learning a parametric model that estimates the Shapley value contributions for all states. Our approach amortises the approximation cost across states and enables continual refinement of explanations under changing policies. We illustrate the method using behaviour explanations; we provide analogous loss functions for outcome and prediction explanations in Appendix A
propose
Specifically, we learn parametric model ő0ss, a: 0) : 8 x A |R (9 with parameters 0, estimate the Shapley contributions of all features of state 8 to the probability of selecting action a. We refer to models that predict Shapley values as Shapley models. Following FastSHAP [12), we adopt the characterisation of Shapley values in Equation 5 as the solution to a weighted least-squares problem, and train the Shapley model by minimising this loss:
[(0)= = E E
# (C) (2 - Σ (s. az (e) Unif( p(C)
E
(10)
TER
Here, p²((s) denotes the steady-state distribution of the policy that the characteristic function (C) is defined over. The distribution Unif(a) represents a uniform distribution over actions, and subsets are drawn from the distribution Equation 6 Because pd (s) is not directly accessible, we
approximate the expectation by sampling from states encountered while following æ. Since the model

---

is unconstrained, we add a post hoc correction to enforce the efficiency constraint in Equation
x*(a)) = ('( a: 0) + F (T(s - -;)
Σ (s.
(11)
JEB
In Appendix A we prove that with a sufficiently expressive model class, the corrected output of the global optimum 0(s, a;; )*) recovers exact and unbiased Shapley values for all state-action pairs (s, a) such that y" (s) > and a E A. By substituting the appropriate characteristic function into the loss in Equation I0, the same model architecture and training procedure can be used across all SVERL explanations of behaviour, outcome, and prediction.
3.2 Approximating characteristic functions in reinforcement learning
SVERL explanations rely on a characteristic function defined as an expectation over the state space, which is infeasible to compute exactly in high-dimensional domains. We begin by considering how to approximate these functions for explaining behaviour and prediction, which share the same structure, before turning to explaining outcomes, which requires different approach.
The behaviour characteristic function (C) in Equation 7 is the expected probability of taking action a in state given only the features in subset C. One possible approach is to estimate it via Monte Carlo sampling, drawing samples from the conditional steady-state distribution p" s²) [8] However, such estimates would not generalise across states or feature subsets, making them inefficient for repeated explanations. We instead, we propose training a parametric model (s. e, 8), with parameters B, to estimate the characteristic function, replacing features of not in C. with a value outside the support of 8, minimising the expected squared error:
((3) E
E (T. #(s a e: 3) (e) Uni((a) P(C)
E
(12)
This approach amortises the approximation cost across multiple states and feature subsets. The model cannot recover the exact target (ss. a) from partial input because different states can share the same values on subset e. It instead learns to predict their mean: the characteristic function T² (e). In Appendix B we prove that with a sufficiently expressive model class, the global optimum f(ss a e; (*)) recovers the exact and unbiased characteristic values for all triples (s, a, e) such that p" (s) > 0, a E A, and p(e)= > O.
The prediction characteristic function e (e) shares the same conditional expectation structure but uses (s) as the prediction target. We propose approximating using the same parametric modelling approach, minimising the expected squared error:
((3) = E E
1 (s) - i(s e; 2
(13)
p" P())
For the outcome characteristic function 05 (C), defined as the expected return from state 8, when the agent follows a modified policy μ that selects actions using the behaviour characteristic (C) at Se and the original policy и at all other states, can use the parametric approximation of # (C) from earlier, which reduces the challenge to estimating expected returns under the policy fu across many (sc, e) pairs.
The outcome characteristic ag (e) highlights the unique challenges posed by the sequential dynamics of reinforcement learning. For a fixed state s, and feature subset C, the characteristic can be estimated using standard reinforcement learning techniques. However, estimating this quantity for all (se: C) pairs amounts to solving 21 x [8] separate optimisation problems. To address this, we define single conditioned policy i( se, C) that behaves according to (C) when the agent's current state se, and otherwise follows the original policy æ. This conditioned policy enables us to parametrise the agent's 's behaviour across all states s and actions for any (se C) pair. We then introduce a parametric value function V(s se, C; 3) to estimate expected returns under the conditioned policy, amortising estimation of ह5. (C) across all (se, e) pairs.
matches the nate-to-be-explaine
To estimate V(s sg. e: 3). we consider the strengths and limitations of two general strategies from reinforcement learning. Offf-poicy methods can reuse data from earlier policies but may include few or no transitions under the policy aa 8; Se, e); on-policy methods avoid this difficulty by collecting new data from # at the cost of additional interaction with the environment. Both strategies have advantages; we present one representative approach for each. For clarity, we use simple DQN-style losses [17, but more advanced value-based methods can also be applied.
5

---

0.3
Or-policy, approx behavinur
Apprix autrome, apprax behaciour
Orepolicy, soact behaviour
Apprix outcome,, scaat behaviour Exact outcome, exact behaviour
0.10
Off-pelicy, approx behaviour Off policy, exact behavisur
MSE
0.2
0.05
560 1000 2500
50000
100000
150000
200000
2000
4000
Training updates
Training updates
Training updates
Behaviour characteristic
(b) Outcome characteristic
(c) Outcome Shapley values
How approximation accuracy improves with training updates in Mastermind- 222. Shaded regions, which are negligible, indicate standard error over 20 runs. As we progress from plot (a) to (b) to (c), downstream models use exact or approximate upstream models from earlier plots.
Figure I:
For on-policy learning, we minimise the difference between the current state-value estimate ((s Se: () and target computed from the bootstrapped return: (e,C; (ss|s.,e;3)
(13)
(14)
where B contains transitions sampled from the conditioned policy, and V is target network with periodically updated parameters. To populate B, consider collecting a single transition at decision stage t: the agent is state 8t, and separately, a state-to-be-explaine s, and feature subset e are sampled (e.g. from a replay buffer and uniform distribution, respectively). Conditioning on the sampled se and e collapses the single conditioned policy (a s; Se. e) into a standard policy for this step. The agent takes an action based on this policy: if its current state S, happens to be the state being explained, $e, acts according the behaviour characteristic model (Equation [2]: otherwise, it acts according its original policy This action produces a single transition (st, dr, dt,41+1, 8t++)), which is stored in buffer B.
For oof-policy learning, we instead learn a state-action value function ( 8e, C; () to hootstrap using actions sampled from the conditioned policy t(a s; e) by optimising the following objective:
L(B)
E (s.ar:")=B'''-
E E
E 10''s'''''' se, e Q(s, se, e) (15) (sc) p(e)
Here, the buffer contains transitions sampled from some other policy. The corresponding outcome characteristic is then recovered by:
(C)
Σ #(s a se, C) Q(s Se, C;
(16)
Together, these approximation techniques form the basis of a scalable framework for generating Shapley-based explanations in reinforcement learning. We call this framework FastSVERL.
3.3 Empirical illustration
We illustrate the use of FastSVERL in multiple domains, guided by three questions on accuracy, efficiency, and scalability: (1) How well can the proposed models learn approximate characteristic functions and Shapley values? (2) How many training updates are required to reach given level of approximation error? (3) How does the computational cost of the approximation scale with the number of states and features in an environment?
We start by focusing on outcome explanations, natural choice because they depend on three components- behaviour characteristic, the outcome characteristic, and Shapley values that together span the key parametric models and loss functions introduced in Sections 3.1 and 3.2 We approximate all three components for a DQN agent in the Mastermind- 2222 domain used by Beechey et al. 22 In the main paper, present experiments focusing on Mastermind- 222, with eight features and 53 states, due to its tractability for exact Shapley value computation. In the appendix, analogous results for all explanation types, additional domains, and complete domain descriptions are provided, including experiments in larger domains where exact computation remains feasible only for behaviour and prediction explanations.
subset of explanation types and domains,
Figure T shows the mean squared error (MSE) between predicted and exact values for (a) the behaviour characteristic, (b) outcome characteristic, and (c) outcome Shapley values, averaged over

---

all states and features. plotted against training updates. For the outcome characteristic, we include both on-policy and off-poliyy training regimes. For all downstream models, we present results using exact or approximate upstream components (e.g. the outcome characteristic trained using the exact or approximate behaviour characteristic), to reveal how emrors propagate through model approximations.
All three models converge to low approximation error (< 0.01). As expected, downstream mod- els that rely on approximated characteristics- rather than exact ones converge to less ac- curate solutions, illustrating how errors in up- stream components degrade downstream per- formance. For the outcome characteristic, off- policy training converges faster than on-policy training, which is expected given that it reuses the original agent's training experience rather than collecting new transitions from the con-
10
Dimension
10a
Training updates to target loss (log scale)
10a
101
Dimension
10'
10a
ditioned policy. Whilst the improvement is substantial, off-policy learning is effective only when the stored experience sufficienlly covers states relevant to the conditioned policy.
101
Dimensian
Length
Longth
10²
Length
Length
Length
We now examine how FastS approxima- tions scale with domain size in Hyper cube, an n-dimensional gridworld where the cube's di- mension n and side length control the number
10a
10*
Characteristic
Shapley
10a
of states ((")) and features (n). Figure 2 shows
how the training updates required to reach tar-
10a 109
get loss of 0.01 for the behaviour characteristic, and corresponding Shapley values, scales with the number of states and features. To isolate how the Shapley model scales without errors propagating from the behaviour characteristic, train using exact characteristic values.
10
10a
10a
104
Number of states (log scale)
Figure 2: Training updates (mean + standard error over 20 runs) needed to reach a fixed target loss (0.01) when approximating behaviour explanations in Hypercube. Each subplot fixes the number of features (i.e. dimensions). Bar colour indicates cube side length l.
For a fixed number of features (i.e. cube di- mension), the training cost increases roughly linearly on the log-log scale, indicating approx- imate polynomial growth as the number of states increases. In contrast, given fixed number of states, increasing the number of features has little effect on training cost, suggesting that states-no not features-ad the dominant driver of computational cost. Finally, the behaviour characteristic requires more updates than the Shapley model, possibly because must predict values for all feature subsets at each state (88) 2²). In contrast, the Shapley model predicts values per feature and state ( x n).
Table 1: Convergence of behaviour models in large-scale Mastermind domains over 10 runs.
Updates to Converge Final Loss
Domain
Model
(Mean + Std. Err.)
(Mean + Std. Err)
Mastermind 443 (24 features, 2 4.. 10? states)
Characteristic
(1.10 + 0.11) x 10
(3.83 + 0.02) x 10
Shapley
(7.31 + 0.68) 10
(1.30 + 0.04) 10 3
Mastermind -453
Characteristic
(1.29 + 0.11)
10
(3.60 0.02) 10
(30 features, 2 3.5 x 10 states)
Shapley
(7.84 + 0.49) x 10
(0.96 + 0.03) x 10-3
Mastermind-46e -463
Characteristic
(1.18 1: 0.12) x 10
(3.70 4 0.01) x 10
(36 features, 2 2.8 x 10t states)
Shapley
(7.12 + 0.51) x 10
(1.88 + 0.04) x 10
While accuracy validation requires small domains with computable ground truths, it is also important to investigate FastSVERL's approximations at scale. We now examine three key properties in substantially larger Mastermind domains: (1) if the models converge, (2) the stability of that convergence, and (3) if the computational cost remains manageable. Table shows that both the characteristic and Shapley models converge and that this convergence is stable and reliable across runs. Most importantly, the number of training updates required remains consistent even as number of states and features grows. Note that this training loss is not measure of ground-truth
7

---

policy
0.020
1:1
0.020
Exact
M5F
0.02
On- On-policy
Off policy (with IS;
10:1
0.015
Sample
Model
0.010
0,010
0.01
0,005
0.00
0.000
0.000
100000 200000 300000
5000
10000
2000
4000
6000
Training updates
Training updates
Training updates
(a) Behaviour characteristic (off-policy training)
(b) Prediction Shapley
(c) Behaviour Shapley (sampled characteristic)
(parallel training)
Figure 3: Approximation accuracy over training updates in Mastermind 222. Each line shows the mean squared error (MSE) between predicted and cxact values, averaged over all states and features. Shaded regions indicate standard error over 20 runs, corrected for variance in agent training [16).
accuracy. For the characteristic model, the loss is designed to converge to non-zero value as it learns an expectation over unknown features. For the Shapley model, convergence zero indicates has learned to explain the approximations from the characteristic model, not necessarily the true values.
While ground-truth accuracy cannot be validated, demonstrate example behavioural insights the framework can produce by visualising an explanation for a trained Mastermind-4e agent in Table 2 In this version of the code-breaking game, an agent must guess hidden 4-letter code, drawn from 3-letter alphabet. Each guess receives clues for the number of correct letters in the correct position (Clue 2) and wrong position (Clue I, full details in Appendix G). The table shows board state where darker blue indicates feature's 's positive contribution to the probability of the agent's next action (in green).
Taking care not to over-interpret [ the expla- nation appears align with logical, high-level strategy for the game. First, the most influen- tial features (blue cells) are from recent guesses (2-4), which are sufficient to deduce that the code is a permutation of (B, C, C), while the now-redundant first guess has neutral influ- ence. Secondly, the unused guess slots are cor- rectly assigned a neutral influence. This reveals an important insight about the agent's behaviour: the optimal policy does not require these slots.
Table 2:
Behaviour Shapley values in one sample state of Mastermind-463.
Guess Clue
Pos
Pos
Pos 3 Pos
Clue 2
5
C
C
C
B
4
2
B
C
C
C
2
3
2
B
C
2
2
0
C
C
3
Importantly, it also suggests the approximation satisfies the nullity axiom of Shapley values by assigning zero contribution irrelevant features.
1
0
A
A
C
A
1
4 Applying FastSVERL to common reinforcement learning settings
We now discuss approximating Shapley explanations under practical constraints.
Learning to explain off-policy. VERL trains characteristic and Shapley models using samples from the steady-state distribution of the policy being explained. This distribution is approximated using states encountered by the policy. What if the agent cannot interact with the environment to collect this data, or if the interaction is costly? We then have more constrained off-policy problem, distinct from the setting previously discussed for the performance characteristic, which addressed sample-efficient data reuse when new interactions are possible. Here, we propose learning passively by drawing states from the agent's history of interaction and applying importance sampling (18) to correct the distributional mismatch. For any loss of the form L(8) Eg-p" [e(s; 8)], we estimate:
( ae)
L(3)
E(s
(17)
(x()),a))
where n; denotes the policy used at decision stage to generate sample (st, The reweighting in Equation estimates the loss that would have been observed had the data been collected under
policy æ. The usefulness of this estimate depends on factors such as whether the buffer contains
states visited under We provide a full derivation and discussion of Equation 17 in Appendix D

---

We illustrate the impact of importance sampling by training the behaviour characteristic model in Mastermind-222 under three conditions: (1) using on-policy data from the final policy. (2) using off-policy data from the agent's training buffer without importance sampling (IS), and (3) using the same buffer with importance sampling. The results are shown in Figure Bal Importance sampling lowers approximation error when using the agent's 's training buffer, as expected, but does not match the accuracy of the on-policy baseline.
Continuously learning to explain. FastSVERL explains an agent's behaviour, outcomes, and prediction under given policy. Yet, in many settings, such as continual learning, we will want to explain an agent as its policy changes over time. Rather than retraining FastSVERLS models when the policy changes, we ask whether jointly updating them with the agent's ' policy can keep explanations aligned to the policy throughout learning. FastSVERL: approximations can be continuously updated because they use parametric models; contrast, sampling-based methods would require recomputing explanations from scratch whenever the policy changes. To this end, we propose a training regime in which the interaction data used to update the agent's policy is also used learn and continuously update the Shapley explanation models.
Once again, we find ourselves in an off-policy context because earlier behaviour policies will have typically generated the data. We therefore apply the importance sampling technique from Equation 17 to account for off-policy samples. When policy changes are small, as is typical when using algorithms like PPO (24), we expect the explanation models from the previous decision stage to remain closely aligned with the agent's newly updated policy. As a result, jointly updating the explanation models alongside the agent's policy may restore complete alignment.
We illustrate the proposed approach by jointly training DQN agent, a characteristic model, and Shapley model for prediction in Wastermind-222 To control how quickly the explanations adapt to the changing policy, we vary the number of gradient descent updates applied to the explanation models, per policy update, using the ratios 1:1, 2:1, 10:1, and 50:1. Figure3b shoss the approximation error in the Shapley model throughout training for the various update ratios tested. All configurations start with a low eтror for the agent's 's initial (random) policy because actions and value estimates are independent of features; the ground truth contributions already match the near wear-zero predictions of the randomly initialised Shapley model. As the agent's policy improves, the approximation error spikes for low update ratios (1:1 and 2:1). These spikes coincide with a sharp rise in expected return observed during training (see Appendix [)) reflecting large policy updates and suggesting that explanation models aligned with the agent's s previous policy become misaligned when the policy shifts significantly. However, increasing the update ratio mitigates these spikes: the 10:1 and 50: I configurations maintain relatively low approximation error throughout. These results suggest that jointly training FastSVERL's models with the agent may allow explanations to remain aligned with changing policy, provided the update rate sufficient to track larger shifts.
5 Approximating Shapley values without characteristic models
We introduced FastSVERL, model-based framework to approximate Shapley values in reinforcement learning, providing a solid foundation upon which the community can build. Here, we present one such promising extension. FastSVERL uses parametric approximations of characteristic functions to train Shapley models. Training these characteristic models is major computational bottleneck, with the Hypercube experiment in Figure suggesting they require as much computation as training the Shapley models themselves. This motivates the search for alternative approximation methods. Monte Carlo sampling might be an appealing alternative because it removes the need for parametric models entirely. However, this merely shifts the computational burden from model training to the high cost of repeated sampling. What if we could preserve the model-free benefits of sampling without incurring its computational cost? We propose integrating single-sample approximations of characteristic values (with low computational cost) directly into the Shapley model loss, amortising characteristic estimation within Shapley training itself. We illustrate this idea using behaviour explanations.
We replace the behaviour characteristic 75. (C) in the Shapley model loss (Equation 10) with a single- sample approximation. At each loss evaluation, we sample a state s' ~ p- sC) and use the action probability (s',, a) in place of querying characteristic model:
E T a) (0) -00-X - (s
(0)
E
E
E
(18)
p (a) Unif(a) p(e) s' s/mpp (-)
iEe

---

In Appendix[F] we prove that optimising this new loss recovers the exact and unbiased Shapley values, with the same per-update cost as the original loss in Equation [0] This approach presents a favourable trade-off: we accept higher variance in individual gradients to eliminate the cost of pre-training characteristic model and prevent its approximation errors from propagating into the Shapley model.
We illustrate the proposed approach by training a behaviour Shapley model in Mastermind-22 with three characteristic approximations: (1) single-samples, (2) model-based, and (3) exact values. Figure 3c shows the Shapley model's loss over cumulative training updates, reflecting characteristic and Shapley model training. The model-based curve starts later because its initial updates are dedicated to learning the characteristic model (indicated by the dashed line), and it converges to higher loss, reflecting approximation errors in the characteristic. In contrast, sampling converges before the model- based setup begins its Shapley updates, matching the performance of using exact charucteristic values. Interestingly, the model trained with exact values converges slightly slower. As observed in prior work 77. the noise from the single-sample approximations may act as stochastic regulariser, encouraging more efficient learning by preventing the model from overfitting to single batch. These results show how replacing characteristic models with sampling can improve Shapley model efficiency and accuracy: halving total training time and eliminating error propagation.
Related work
31 and prediction 34 23 35 in reinforcement learning. Beechey et al. 2 studied the theoretical validity of Shapley missing explanatory element, unifying behaviour, outcomes, and prediction under a single theoretical framework, SVERL. We build on this earlier foundation develop scalable approximation techniques, which make it possible to use SVERL in practical settings.
Shapley values have been applied to explain behaviour 21
10 32
14
20
values in reinforcement learning. Their analysis revealed outcomes
In supervised learning. Shapley values were initially approximated through sampling [29 27 28 15
More recent approaches improve efficiency by amortising computation across inputs with parametric models X [2] that incorporate noisy Shapley targets during training 77. Our work incorporates the use of these amortised methods into reinforcement learning, addressing practical constraints such as off-polia off-policy data and continual learning. In addition, we introduce single-sample noisy target that can eliminate the need for characteristic models, substantially improving training efficiency.
7 Discussion
We introduced FastSVERL, a scalable parametric framework for Shapley-based explanations of reinforcement learning. FastSVERL estimates Shapley values in single forward pass, without relying on costly Monte Carlo samples. We addressed two practical challenges: (i) off-policy learning, to enable explanations with minimal environment interaction, and (ii) continual learning, to allow explanations of non-stationary policies in real time. We showed that replacing characteristic models with single-sampled approximations can substantially improve efficiency and explanation quality; this approach also directly extends to supervised learning. These contributions position FastSVERL as a scalable solution for interpretability in practical reinforcement learning problems.
The proposed methods are broadly applicable to episodic and continuous tasks, well partially observable settings. Although this work focuses on discrete action spaces, the underlying theoretical framework supports continuous actions, and FEstSVERL's parametric approach readily adapts. Ap- plying the framework to continuous state spaces, however, highlights a key challenge: approximating the steady-state distribution. In high-dimensional settings, experience buffers provide only a sparse approximation of the true distribution. One promising direction is to learn a parametric model of this distribution
g which could generalise across sparsely sampled regions.
Our work focused on the significant computational challenges of approximating Shapley values. Formal user studies and practical deployment in real-world systems are essential to evaluate how these explanations aid human understanding and decision-making. While our qualitative results suggest the framework can produce plausibly interpretable insights, user studies can rigorously evaluate robustness, identify use cases, and reveal the conditions under which the approximations become unreliable. This next step critical for establishing best practices and strengthening V capacity for scalable, real-time interpretability in complex reinforcement learning environments.
10

---

Acknowledgments and Disclosure of Funding
This work was supported by the Engineering and Physical Sciences Research Council (EPSRC) [grant number EP/X025470/11 and the UKRI Centre for Doctoral Training in Accountable, Responsible and Transparent AI (ART-AI) [EP/S023437/1). We thank our reviewers for a constructive process.
References
[1] Daniel Beechey, Thomas MS Smith, and Özgür Şimşek. Explaining reinforcement learning with Shapley values. In International Conference on Machine Learning, pages 2003-2014, PMLR, 2023.
[2] Daniel Beechey, Thomas M. S. Smith, and Özgür Şimşek. theoretical framework for explaining reinforcement learning with Shapley values. arXiv preprint arXiv:2505.07797, 2025.
[3] Marc G Bellemare, Salvatore Candido, Pablo Samuel Castro, Jun Gong, Marlos C Machado, Subhodeep Moitra, Sameera $ Ponda, and Ziyu Wang. Autonomous navigation of stratospheric balloons using reinforcement learning. Nature, 588(7836):77-82, 2020.
[4] Nicolas Blystad Carbone. Explainable Al for path following with model trees. Master's thesis, NTNU, 2020.
[5] A Charnes, B Golany, M Keane, and J Rousseau. Extremal principle solutions of games in characteristic function form: core, chebychev and shapley value generalizations. Econometrics of planning and efficiency, pages 123-133, 1988.
[6] Ian Covert, Scott Lundberg, and Su-In Lee. Explaining by removing: A unified framework for model explanation. Journal of Machine Learning Research, 22(209):1-90, 2021.
[7] Covert, Chanwoo Kim, Su-In Lee, James Zou, and Tatsunori Hashimoto. Stochastic amortization: A unified approach to accelerate feature and data attribution. arXiv preprint arXiv:2401.15566, 2024.
[8] Christopher Frye, Damien de Mijolla, Tom Begley, Laurence Cowton, Megan Stanley, and Ilya Feige. Shapley explainability on the data manifold. In International Conference on Learning Representations, 2020.
[9] Samuel Greydanus, Anurag Koul, Jonathan Dodge, and Alan Fern. Visualizing and understand- ing Atari agents. In International Conference Machine Learning, pages 1792-1801. PMLR, 2018.
[10] Lei He. Nabil Aouf, and Bifeng Song. Explainable deep reinforcement learning for UAV autonomous path planning. Aerospace Science and Technology, 118:107052. 2021.
[11] Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and João G.M. Araújo. CleanRL: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23(274):1-18, 2022. URL /RL /tttp::/jjmiie http://wmr.oorrraaeeeep wttg:pappess//2//222 1342. html
[12] Neil Jethani, Mukund Sudarshan, Ian Connick Covert, Su-In Lee, and Rajesh Ranganath. FastSHAP: Real-time Shapley value estimation. In International Conference on Learning Representations, 2021.
[13] Roman Liessner, Jan Dohmen, and Marco A Wiering. Explainable reinforcement learning for longitudinal control. In International Conference on Agents and Artificial Intelligence (2), pages 874-881, 2021.
[14] Jakob Lover, Vilde B Gjærum, and Anastasios M Lekkas. Explainable Al methods on deep reinforcement learning agent for automatic docking. IPAC-PPpersOOnL.ive 54(16))146-152. 2021.
[15] Scott M Lundberg and Su-In Lee. unified approach to interpreting model predictions. Advances in Neural Information Processing Systems, 30, 2017.

---

[16] Michael EJ Masson and Geoffrey R Loftus. Using confidence intervals for graphically based data interpretation. Canadian Journal of Experimental PsychologyRevue Canadienne de Psychologie Expérimentale, 5733):033, 2003.
[17] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing Atari with deep reinforcement learning. arXiv preprini 1rXiv:1312.56// 2013.
[18] Doina Precup, Richard Sutton, and Satinder Singh. Eligibility traces for off-policy policy evaluation. In ICML, volume 2000, pages 759-766. Citeseer, 2000.
[19] Nikaash Puri. Sukriti Verma, Piyush Gupta, Dhruv Kayastha, Shripad Deshmukh, Balaji Krish- namurthy, and Sameer Singh. Explain your move: Understanding agent actions using specific and relevant feature attribution. In International Conference on Learning Representations, 2020.
[20] Sindre Benjamin Remman, Inga Strümke, and Anastasios M Lekkas. Causal versus marginal Shapley values for robotic lever manipulation controlled using deep reinforcement learning. In 2022 American Control Conference (ACC), pages 2683-2690. IEEE, 2022.
[21] Stefano Giovanni Rizzo, Giovanna Vantini, and Sanjay Chawla. Reinforcement learning with explainability for traffic signal control. 2019 IEEE Intelligent Transportation Systems Conference (ITSC), pages 3567-3572. IEEE, 2019.
[22] Luis M Ruiz, Federico Valenciano, and Jose M Zarzuelo. The family of least square values for transferable utility games. Games and Economic Behavior, 24(1-2):109-13)) 1998.
[23] Lincoln Schreiber, Gabriel Ramos, and Ana Lucia Cetertich Bazzan. Towards explainable deep reinforcement learning for traffic signal control. In Proceedings of LatinX in Al Workshop at ICML 2021, LXAI. LXIA, 2021.
[24] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1rXiv:1707.06347 2017.
[25] Jaemin Seo, SangKyeun Kim, Azarakhsh Jalalvand, Rory Conlin, Andrew Rothstein, Joseph Abbate, Keith Erickson, Josiah Wai, Ricardo Shousha, and Egemen Kolemen. Avoiding fusion plasma tearing instability with deep reinforcement learning. Nature, 626(8000):746-755) 2024.
[26] Lloyd S Shapley. A value for n-person games. Contributions to the Theory of Games, 2(28): 307-317, 1953.
[27] Erik Strumbelj and Igor Kononenko. An efficient explanation individual classifications using game theory. The Journal of Machine Learning Research, 11:1-18, 2010.
[28] Erik Štrumbelj and Igor Kononenko. Explaining prediction models and individual predictions with feature contributions. Knowledge and Information Systems, 41:647-665, 2014.
[29] Erik Štrumbelj, Igor Kononenko, and Robnik Šikonja. Explaining instance classifications with interactions of subsets of feature values. Data & Knowledge Engineering, 68(10):886-90) 2009.
[30] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[31] Philipp Theumer, Florian Edenhofner, Roland Zimmermann, and Alexander Zipfel. Explainable deep reinforcement learning for production control. In Proceedings of the Conference on Production Systems and Logistics: CPSL 2022, pages 809-818. Hannover: publish-Ing.. 2022. 202.2
[32] Yuyao Wang, Masayoshi Mase, and Masashi Egi. Attribution-based salience method towards interpretable reinforcement learning. AA Spring Symposium: Combining Machine Learning with Knowledge Engineering (!), 2020.
[33] Christopher JCH Watkins and Peter Dayan. Q-learning. Machine Learning, 8(3):279-292. 1992.
12

---

[34] Ke Zhang, Peidong Xu, and Jun Zhang. Explainable Al in deep reinforcement learning models: SHAP method applied in power system emergency control. In 2020 IEEE 4th Conference on Energy Internel and Energy System Integration (EI2), pages 711-716. IEEE, 2020.
[35] Ke Zhang, Jun Zhang, Pei-Dong Xu, Tianlu Gao, and David Wenzhong Gao. Explainable AI in deep reinforcement learning models for power system emergency control. IEEE Transactions on Computational Social Systems, 9(2):419 427, 2021.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our experiments are intended to illustrate the potential of the proposed frame- work, not to provide absolute guarantees, aligning with the claims made in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Due to space constraints, we do not have a dedicated section, but we discuss limitations throughout.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
14

---

Answer: [Yes].
Justification: Provided in the appendices.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Full details provided in the appendicies.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
15

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Code is publicly available on GitHub.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //ipss.cc/pill wttpcccpppppppppeeeeeeeeeeeeeeee http:///e/.ieee/eteSuuuiiiissss (Codeooomiissssseeeeeeee swiiiissssioicyyyccc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Provided in the appendices.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Where standard error is not reported visible, it is because it was negligible. Full details on experiments are provided in the appendices.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
16

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: This was not recorded when the experiments were run. The full research project required more compute than the experiments reported in the paper because of preliminary and failed experiments.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htt.g///t/////aar./o///////////a
Answer: [Yes]
Justification:
We have read the NeurIPS code of ethics and confirm that our work conforms.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discusses positive societal impacts; foresee no negative societal impacts of this work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
17

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: None of our publicly available code has a high risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: No existing assets used.
Guidelines:
The answer NA means that the paper does use existing assets. The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
18

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[NA]
Justification:
No existing assets used.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
19

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: This work's 's core method does not involve LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy httt:p:://aarrrps..cc///offeeeee for what should or should not be described.
20

---

A
Shapley models
This section presents the learning objectives used to train the Shapley models for all three explanation types: behaviour, outcomes, and prediction. For each case, adopt the characterisation of Shapley values as the solution to a weighted least squares problem (see Equation 5 and construct a corre- sponding loss function. A general convergence result applicable to all three provided at the end of the section.
Behaviour: We train a parametric model ( a; 0) x A + contributions of each feature of state to the agent's probability of selecting action The model is trained to minimise the following loss:
R to predict the Shapley
[())
= E
E
E
- - Σ (s (s a;
(19)
Pr (a) Unif(e) p())
iEC
After training, the model output corrected to enforce the efficiency constraint:
~ (s; + [F] m(s -2)
1
Σ ② (s, 0),
(20)
With sufficiently expressive model class and exact optimisation, the corrected output of the global optimum o(s, a; 0*) recovers exact Shapley values for all state-action pairs (s, a) such that p=(ss and a E A.
Outcome. We train a parametric model ((s; 0) : 8 - R19 to predict the Shapley contributions of each feature of state to the agent's 's expected return e (s). The model is trained minimise the following loss:
( = (*(s) p(e)
(2) - 05 (0)
Σ (s; 0)
(21)
After training, the model output is corrected enforce the efficiency constraint:
(' = (s; +
[F] 2d (s) (0)
Σ き (s:
(22)
jEY
With sufficiently expressive model class, the corrected output of the global optimum o(s; 0*) recovers exact Shapley values for all states s such that pd (s) >
Prediction. We train a parametric model ((s; 0) $ - | to estimate the Shapley contributions of each feature of state s prediction of the agent's s expected return 1 (s). The model trained to minimise the following loss:
(s; 0)
(())
E E (a)
(e)
(0)
(23)
p(e)
sec
After training, the model output is corrected to enforce the efficiency constraint:
(6 & [F
1
0 (3(0)
Σ (s:)))
(24)
jES
With sufficiently expressive model class, the corrected output of the global optimum Ø(s; recovers exact Shapley values for all states such that pa 0.
0*)
Convergence proof. We now prove that the learning objective in Equation 19 recovers exact Shapley values at the global optimum. This result generalises the prediction and outcome objectives because they share the same structure.
We make the following assumptions:
The model dss o; 0) is selected from function class expressive enough to represent the true Shapley value function @(77) for all state-action pairs (s, a) such that p" (s) > 0.
21

---

2. The global minimum of the loss (0) exists and is attained.
3. States sampled from the steady-state distribution p" (s), and actions sampled uniformly from A.
Fix a state-action pair (s, a) such that p" p=((s) > The learning objective for this pair reduces to the following expected loss:
E
(C(C) (0)
(25)
p(C)
where o E | denotes the predicted attribution vector produced by the model for this fixed (s, a).
If the following efficiency constraint is imposed:
Σ
(d)= -
(26)
then this becomes constrained weighted least-squares problem with unique global minimiser given by the Shapley values ((*g) [S].
This constraint can be satisfied by applying additive correction to the model output:
o' ( :- (s a: + [3] x(ssa) #. (0)
Σ (3(..a:0)
(27)
JES
which adjusts the model output by constant shift that distributes the residual error uniformly across features [22]. This correction preserves the minimiser of the original unconstrained loss because the transformation is linear and orthogonal to the residual, and ensures the efficiency constraint is satisfied.
Since the model class contains the exact solution and the loss minimised exactly, the globally optimal parameters 8* yield a model Ō(s, a; 8*) that recovers the Shapley values for all (s, a) such that pa (s) and E A.
Remark. All of the Shapley models trained solve weighted least-squares problem whose unique solution is the true Shapley value vector for the given characteristic function. Therefore, these estimators asymptotically unbiased.
B Characteristic models
This section presents the learning objectives and convergence results for training the characteristic models for behaviour, prediction, and outcome explanations.
Behaviour. We train a parametric model #(s, C; 3) to approximate the characteristic function (C): the expected probonnitbbbbittty π ((s a) when only the features in e are known. The model receives input state-action pair (8, a), where features of not in e are replaced by fixed masking value outside the support of S. is trained to minimise the following loss:
((3)
E E [T(s.)) - #(s, a | 2)
(28)
pr Umif(a) P(C)
With sufficiently expressive model class and exact optimisation, the global optimum fts, (; recovers the exact characteristic value for all triples (s,a, e) such that (=(s) 0, a E A and p(e) 0:
A(s, e; = E [T(S a) Se = = (C).
(29)
Prediction. We train a parametric model i(s (; to approximate the characteristic function (C): the predicted expected return 8h (s) when only the features in are known. The model receives as input state where features not in é are replaced by fixed masking value outside the support of It is trained to minimise the following loss:
L(3
E
E P(C)
((s)) i(s e; 2
(30)
p
22

---

With sufficiently expressive model class and exact optimisation. the global optimum 0(s e;3*) recovers the exact characteristic value for all pairs (s, e) such that p= and p(c) > 0:
o(s C; (8)) E ((S) s = = 0. (e).
(31)
Behaviour convergence proof. We now prove that the learning objective in Equation recovers exact characteristic values at the global optimum. This result generalises to the prediction objective in Equation 30 because they share the same structure.
We make the following assumptions:
1. The model f (s, a C: () is selected from function class expressive enough to represent # (C) for all masked inputs corresponding to (s", a) such that p=((s) 0 and ((C) >> 0.
2. The global minimum of the loss ((3) exists and is attained.
3. States s are sampled from the steady-state distribution p" (s), actions from the uniform distribution over A, and subsets from fixed distribution over all subsets of F.
We consider the contribution to the loss from a fixed subset C and action a. The full loss is an expectation over such terms. Under this conditioning, the loss reduces to:
(ss. - #(s a C; [=
(32)
p*(s)
where C and are fixed. Because the model receives masked inputs in which features outside C are replaced by fixed value, it produces the same output for all states & that share the same values d on C. As result, the loss decomposes into disjoint terms over equivalence classes of e:
p Er 22
E (s(())))
(s( - #ss e; e=
(33)
where pa (s) denotes the marginal distribution over observed feature subsets under p (s). Each term a squared-error regression problem in which the model output is constant across the equivalence class. Therefore, the unique minimiser of each such term is the conditional expectation:
A(s,a a e; (²)8")= E ((, se = = = (C),
(34)
for all s such that p=(s²)= 0.
arbitrary, and the result holds for all equivalence classes se with p" (s > 0, the global minimiser of the full loss recovers exact characteristic values for all (s, a, e) such that p" (s) 0 and p(e) 0.
Since both the subset e and action
The characteristic models for behaviour and prediction are trained to minimise an expected squared-error loss. The unique global minimiser of this objective is the true conditional expectation. An estimator that converges to this true mean is, by definition, asymptotically unbiased.
Remark.
Outcome characteristic convergence. We do not provide a convergence proof for the outcome characteristic model. This component trained as a value function using standard reinforcement learning techniques, including bootstrapping and lemporal-difference updates. In our case, the implementation is based on the DQN algorithm. While convergence guarantees exist for value- based learning in the tabular setting [33 convergence results are generally not known for deep reinforcement learning methods with function approximation. This remains long-standing open problem in the field. As such, the convergence behaviour of the outcome characteristic model cannot be formally established. Nonetheless, we empirically observe reliable learning across all domains this work considers.
C Empirical illustrations of FastSVERL's explanation models
In this section, we extend the empirical illustration of FastSVERL presented in Figures andd] by providing results for the complete set of explanation behaviour, outcomes, and prediction.
Accuracy. Figure 4 on the following page shows approximation accuracy function of batch updates of gradient descent (training updates) for characteristic and Shapley models applied a DQN agent across three domains: Mastermind-24 (8 features, 53 states), Mastermind-335 333 (15 features,
23

---

over 100,000 states), and Gridworld (2 features, 7 states). Complete domain descriptions are provided in Appendix G Notably, Mastermind represents the practical limit exact Shapley value computation for behaviour and prediction explanations, highlighting FastSVERL's ability to approximate these explanations in substantially larger state spaces.
Each experimental run is seeded for reproducibility, with variability primarily stemming from sources typical to PyTchh-based training, including weight initialisation and batch sampling. Characteristic models are trained using a single agent instance across all runs. Likewise, Shapley models are always trained with the same agent and characteristic models for each run, avoiding variability introduced by retraining upstream components.
Consistent with the findings in Section 33 all models converge to low approximation error and exhibit clear error propagation from characteristic models to Shapley values.
24

---

Behaviour characteristic
Behaviour characteristic
Behaviour characteristic
0.03
0.100
MSE
0.10
0.02
0.075
0.050
0.05
0.02
0.00
2500 3000
2000020000000000000000000 Training updates
1500 Training updates
1000
Training updates
Behaviour Shapley
Behaviour Shapley
Behaviour Shapley
0.020
Exact char as lerriskk
0.008
Exact cha beristk
0.015
Exact characteritic
App nox charactaristic
App characteriziic
Approx dharacterisic
0.015
0.006
MSC
0.010
0.000
0.004
0.005
0.055
0.000
0.000
0.000
10:00
2000
3000
5000
10000
15000
200
400
600
800
1000
1200
Training updates
Training updates
Training updates
Outcome characteristic
Outcome characteristic
On-palicy, spprox
Cr-paiit, apprex
Orepelicy, moect behaviour
Enspaiiy, seart behwiisr off-policy, apprex behaviour Off policy, exact behaviour
MSE
off approx behaviour
Off policy, sract behaviour
0.1
0
50000
100000 150000 200,c0 Training updates
20000 40000 60000 80000 100000 Training updates
Outcome Shapley
Outcome Shapley
0.06
Approx outcome; aporox behavoor
Apprex sutcome, approx behavtour
behaviour
0.4-
exact behavfour
MSE
0.04
outcome, exact behavlour
Exact outcome. exact behaviour
0.02
0.00
0.0
Training updates
Training updates
Prediction characteristic
Prediction characteristic
Prediction characteristic
0.15
MSE
0.10
30-
0.05
0.00
0.0
0
1000
2000
3000
4000
5000
4c000
80000
120000
500
1000
15:0
Training updates
Training updates
Training updates
Prediction Shapley
Prediction Shapley
Prediction Shapley
0.020
Exatt characterisic:
charactenttic
Exact chanactersticc
0.015
Approx characteristic
Approx characteristc
Approx characteristic
MSE
0.02
0.05
0.000
0.00
3000
10000
20000
30000
400
1200
Training updates
Training updates
Training updates
(a) Mastermind-222
(b) Mastermind-39
(c) Gridworld
Figure 4: How approximation accuracy improves training updates for FERtSV explanation models across Mastermind-222 (left), Mastermind-333 (middle), and Gridworld (right). Each line shows the mean squared error (MSE) between predicted and exact values, averaged over all states and features. Shaded regions, which are negligible, indicate standard error over 20 runs. As you move down through the rows, downstream models (e.g. the behaviour Shapley model) use exact or approximate upstream models from earlier plots (e.g. the behaviour characteristie). Empty slots correspond to the outcome explanations that cannot feasibly be computed exactly in Mastermind-333.
25

---

Dimension
10a
Dimension
10a
10a
Training updates to target loss (log scale)
Training updates to target loss (log scale)
10
10
10
Dimension
10a
10a
10a
10a
10a
Dimension
Length
101
Dimension
Length
Length 3
Length
10²
Length
Length
Length
10'
Length
Length 6
Length
10a
10"
10a
Dimension
Characteristic Shapley value
10*
Dimension
Shapley value
Characteristic
10a
10a
10a
102
10"
10
10a
10a
10*
10"
10
10a
10a
10*
Number of states (log scale)
Number of states (log scale)
(a) Behaviour
(b) Prediction
Figure 5: Training batches required to reach fixed target loss (0.01) when approximating charac- teristic and Shapley values for behaviour and prediction in Hypercube; standard error over 20 runs. Each subplot fixes the number of features (i.e. dimensions). Bar colour indicates cube side length
Scalability. extend the illustration of FasISVERL's scalability presented in Figure applying it prediction explanations in the Hypercube domain. Outcome explanations were not included because exact Shapley values were infeasible to compute for the larger cubes. Variability across runs is limited to randomness in training, with all experimental conditions fixed and the entire process fully seeded for controlled reproducibility. Figure 5 shows the training updates required to reach a fixed target loss (0.01) for both behaviour and prediction explanations. Consistent with the findings from behaviour explanations, the number of training updates scales approximately polynomially with the number of states, while remaining relatively insensitive the number of features.
Large-scale convergence results. We now extend the large-scale domain analysis from Table to the prediction and performance explanation models. As with the behaviour explanations, direct accu- racy validation is infeasible in these domains. We therefore examine the same three key properties: (1) if the models converge, (2) the stability of that convergence, and (3) if the computational cost remains manageable as domain complexity grows.
Table 3 presents the convergence results for the prediction models. Consistent with the findings for the behaviour models, both the characteristic and Shapley models converge to stable loss, and the number of training updates required remains consistent across the domains.
Next, Table presents the results for the performance models. For the performance explanation, we present results only for the Shapley model. The associated characteristic model uses bootstrapped reinforcement learning objective that does not converge to a fixed value and was instead trained for fixed number of updates- the same number of updates required to train the agent. The Shapley models were trained using only the off-policy performance characteristic. The results again show stable and efficient convergence, consistent with the findings in the main body.
To complement the results in the tables above, Figure visualises the full training curves for each
model, illustrating the rates of convergence.
26

---

Table 3: Convergence of prediction models in large-scale Mastermind domains over 10 runs.
Updates to Converge (Mean + Std. Err.)
Final Loss
Domain
Model
(Mean + Std. Err.)
Characteristic
(1.08 + 0.06) x 106
(2.54 + 0.04) x 10-1
Mastermind-443
Shapley
(6.01 0.51)
x 105
(1.57 0.07) x 10-1
Characteristic
(1.07 = 0.11)
x 10
(2.74 0.04) x 10
Mastermind-453
Shapley
(5.93 + 0.46)
x 105
(1.77= x 10-1
Characteristic
(9.88 + 0.34)
x 10
(3.27 x 10-2
Mastermind-46
Shapley
(9.18 + 0.63)
x 10
(1.70 + 0.06) x 10-1
Table 4: Convergence of performance Shapley models in large-scale Mas termind domains over 10 runs.
Updates to Converge
Final Loss
Domain
Model
(Mean + Std. Err.)
(Mean Std. Err.)
Mastermind-443
Shapley
(6.79 + 0.63) x 10
(1.58 + 0.07) x 10
Mastermind-455
Shapley
(5.70 + 0.44) 10
(6.33 + 0.30) x 10
Mastermind-46
Shapley (5.56 + 0.34) 10
(7.57 + 0.26) x 10-1
Full qualitative results. We extend the qualitative illustration of the framework's 's behaviour insights in Table by providing the full set of visual explanations in our project's code repository This extends the single illustrative example by presenting the behaviour, prediction, and performance explanations for all states encountered by an optimal policy in each of the large-scale Mastermind domains.
D Off-policy learning for explanation models
This section presents the full derivation of the off-policy importance sampling approach introduced in Section 4 along with complete set of empirical illustrations.
D.1 Theory and derivation
FastSVERL trains characteristic and Shapley models using samples from the steady-state distribution (s((s) of the policy being explained. But what if the agent cannot interact with the environment to collect this data, for example, when further interaction is costly? In such settings, we propose sampling from replay buffer 'B containing transitions gathered during training.
Because the buffer B aggregates data from sequence of past policies {mr)² its marginal state distribution (s) differs from the target distribution (s). To correct the distributional mismatch, we apply importance sampling [18), reweighting each sample drawn from B. For any loss of the form
L(3)
E [((s;))]
(35)
s
we rewrite the expectation under (s() using samples from pa (s):
p" p (s)
C(B) E
E(s; 8)
(36)
This yields unbiased estimate of the original loss. However, neither p*((s) nor (s) is known explicitly, so we approximate the density ratio using action probabilities. Since each transition (st, az) B was generated by known behaviour policy ni, we propose estimating the importance
"The complete results available at: https://gittub.co htaps/uubbbcmmm/essb///ttstterrr
27

---

Characteristic
Characteristic
Training Loss
Wasterinn 443
Mastermind 443
Masterming-45 453
Mastermind--S
0.008
8.006
0.004
10000 15000 25000
5000
10000
15000
20000
Training updates
Training updates
Shapley
Shapley
Shapley
0.08
Training Loss
Masteminns-411
Masterminz-443
Mustemins-641
Masteriins-433
Mastermine-45 2
Mastermind-453
Mastemind-463
Mastemiind4463
Mastermind-463
0.4-
0.02
0.2
0.00
200000 400000 6C0C00 80000
0.0
0.2
0.4
0.6
0.8
1.0
0
2000CO
400000
600C00
Training updates
Training updates
Training updates
(a) Behaviour
(b) Prediction
(c) Outcome
Figure 6: Training loss curves for the characteristic (top row) and Shapley (bottom row) models in the large-scale Mastermind domains. Each line represents the mean loss over training updates, aver- aged across all three domain sizes (Mastermind) md-443, Mastermind-455 and Mastermind-4633, Shaded regions indicate standard error over 10 runs. The empty slot corresponds to the performance characteristic model.
weight as:
( ar)
L(B) R3
{(sr;
B)
(37)
(T(s ( ac)
The reweighting in Equation 37 estimates the loss by weighting each sample in proportion to its relevance under æ.
Two practical factors may affect the stability and accuracy of the importance-weighhe Equation 37 The first the similarity between the target policy æ and the past policies { that generated the buffer. The more these policies differ, the higher the variance of the importance weights, increasing the variance distribution p" (s). If states commonly visited by π are underrepresented, their corresponding loss terms may poorly estimated. In practice, transitions from later-stage policies are more likely to resemble the final policy so additional interaction near the end of training may improve both policy similarity and state coverage in the buffer.
loss in
loss estimate. The second is the buffer's coverage of the state
When importance weights exhibit high variance, techniques such as weight clipping (24] or adaptive weighting 30 can help stabilise training. We illustrate these strategies empirically the next section.
D.2 Empirical illustrations
We present three empirical illustrations of off off-policy training in FastSVERL. First, extend the ex- periment from Figure 3a by training prediction characteristic models off-policy in Mastermind- 2222 We then investigate the impact of two variance-control strategies: adaptive weighting [30] and weight clipping [24]. All figures report approximation accuracy over training updates, measured by mean squared emror (MSE) between predicted and exact values, averaged across states and features. Shaded regions indicate the standard error over 20 runs. Variability across runs is limited to randomness in the training pipeline, with all experimental conditions fixed and the entire process fully seeded for controlled reproducibility. Standard errors are corrected to account for this variability [16]
Learning characteristics off-policy Mastermind- -22.. Figure 7 extends the illustration of off- policy training from Figure Balto prediction. Consistent with the findings in Section 4 (1) importance sampling reduces approximation error relative to using the training buffer without reweighting, suggesting that correcting for distributional mismatch improves performance; and (2) approximation error remains higher than under on-policy training, indicating that even with reweighting, off-policy data is an imperfect substitute.
28

---

0.03
Ort-palicy (without Oft-palicy (with I5) Cn-poliicy
Oftiolicy (without 15)
0.04
Ort-alicy (with 15) On-policy
0.02
MSE
0.01
0.02
0.00
0.00
100000
200000 300000
100000
200000 300000
Training updates
Training updates
(a) Behaviour characteristic
(b) Prediction characteristic
Figure 7: Approximation accuracy of off-policy training in Mastermind- 222 with models trained using either on-poliey data, or off-policy data with two configurations: (1) without importance sampling (IS) and (2) with IS.
0.100
Unnormalised IS Without Normalised On-policy
Unnormalised Without IS Normalised
MSE
0.050
On-pelicy
2000
4000
0
2000
4000
6000
8000
Training updates
Training updates
(a) Behaviour characteristic
(b) Prediction characteristic
Figure 8: Approximation accuracy of off-policy training in Gridworld with models trained using either on-policy data, or off-policy data with three configurations: (1) without importance sampling (IS), (2) with unnormalised IS, and (3) with normalised IS.
Normalising importance weights. A common strategy reduce variance in importance sampling to normalise weights within each batch by dividing them by their sum [30)- We illustrate this strategy by training behaviour and prediction characteristic models in Gridworld under three off- policy conditions: (1) without importance sampling, (2) with unnormalised importance sampling, and (3) with normalised importance sampling. An on-policy configuration is included as a baseline. The results, shown in Figure [8] suggest that normalising the importance weights is crucial: using unnormalised weights leads to worse approximation accuracy than no reweighting all, while normalised weights yield the best results. We therefore apply weight normalisation in all experiments using off- off-colicy sampling.
Clipping importance weights. Another common strategy to reduce variance in importance sam- pling is clip the weights to fixed range [24]. We illustrate this strategy by training behaviour and prediction characteristic models in Gridworld under several off-policy conditions: (1) without importance sampling, (2) with unclipped importance sampling, and (3) with importance weights clipped symmetrically around to the range [1- c, c.. We consider thresholds of = 0.99, 0.995, and 0.998, selected yield spread of performance curves. An on-policy configuration included as baseline. The results, shown in Figure 9
indicate that reducing variance through clipping may impair performance: unclipped importance sampling achieves the highest accuracy, while tighter clipping progressively harms approximation. The extreme case of c = 0, equivalent to no importance sampling. performs the worst. These results suggest blas-variacc trade-off in importance sampling. where retaining the full range of weights avoids the bias introduced by clipping and may be preferred to mitigating variance in this setting.
E Empirical illustrations of continuously learning to explain
We extend the illustration of jointly training a DQN agent and explanation models presented in Figure 3b) to all explanation types in the domains Mastermind-222 and Gridworld. Variability across runs is limited to randomness the training pipeline, with experimental conditions fixed and the entire process fully seeded for controlled reproducibility.
29

---

0.08
Without
With 0.99)
With (clip 0.99)
MSE
With 0.995)
9.3)
0.04
With (clip 0.998)
With (clip 0. 998)
clipping]
With (no clipping)
0.02
On-palicy
On-policy
1000
2000 3000 4000 5000
2000
4000
6000
8000 10000
Training updates
Training updates
(a) Behaviour characteristic
(b) Prediction characteristic
Figure 9: Approximation accuracy of off-policy training in Gridworld with models trained using either on-policy data, or off- off-policy data with three configurations: (1) without importance sampling (IS), (2) with unclipped IS, and (3) with IS clipped at thresholds of 0.99, 0.995, and 0.998.
Figure 10 presents the approximation accuracy for the Shapley models across varying update ratios (I:1, 2:1, 10:1, and 50:1), alongside the agent's expected return and DQN loss to track the impact of policy changes during training. Consistent with the findings Section increasing the update ratio mitigates error spikes during rapid policy shifts, reflecting better alignment between the agent and the explanation models. That is, apart from the behaviour explanation for Gridworld, where varying the update rate has minimal effect error reduction. This may be due the already low error magnitudes, suggesting the bottleneck could instead be linked to another source of error, such the off-policy nature of the data.
F Sampling-based approximation of characteristic functions
Training characteristic models is a major computational bottleneck in FastSVERL. In Section we proposed removing this cost by integrating cheap single-sample approximations of characteristic values into the Shapley model loss, demonstrating this for the behaviour characteristic. The same approach can be applied the prediction characteristic, which also admits natural sampling-based approximation.
In this section, we complete the proposal by applying it to all explanation models that rely either behaviour or prediction characteristie. We first consider their use training Shapley models, then turn to sampling the behaviour characteristic in training the outcome characteristic. We conclude with the full set of empirical illustrations of the proposals.
Behaviour Shapley. We train parametric model d(s, a; 0) to predict the Shapley contributions of each feature of state the agent's probability of selecting action a. Instead of querying characteristic model, we replace the characteristic function (C) in the Shapley model loss with a sangle-sample approximation. At each loss evaluation, a state s' ~ p=(. s ) is sampled from the conditional steady-state distribution, and the characteristic value is approximated using iTs', a). The model trained to minimise the following loss:
(() = E
E
E
E s'mpp" (-1ag)
a) - Σ
(38)
(8)
Unift P(C)
E
After training, the Shapley model's output corrected satisfy the efficiency constraint:
Σ 2 (8, (s,a;0)) )
(' ~ S 03 + [F ((8,a) -g
(39)
JEZ
With a sufficiently expressive model class and exact optimisation, the corrected output of the global optimum ò(s, a; 0*) recovers exact Shapley values for all state-action pairs (s, a) such that (=(ss ()) > 0 and A.
In practice, we approximate the conditional steady-state distribution (s sc) by sampling from a
replay buffer containing transitions collected under uniformly selecting states s' such that s ' e
30

---

Expected return
Expected retur
Return
-20
1-
-40
7500 Training updates
2500
5000
10000
2000
4000
6000
8000
10600
Training updates
DON loss
DQN loss
1.0
Loss
0.9
0.8
0.7-
0.6
2000
4000
6000
8000 10000
0
2000
4000
6000
8000
10000
Training updates
Training updates
Behaviour Shapley
Behaviour Shapley
10:1
1:1
MSE
0.02
2:1
50:1
2:1
10:1
50:1
0.01
0.001
0.000
10000
4000
6000 8000
10000
Training updates
Training updates
Outcome Shapley
Outcome Shapley
0.08
1:1
10:1
2:1
50:1
2:1
MSE
10:1
0.04
50:1
0.02
2000
4000
6000
8000
2000
4000
6000
8000
Training updates
Training updates
Prediction Shapley
Prediction Shapley
0.020
1:1
1.0
1:1
2:1
MSE
0.015
10:1
10:1
0.005
0.000
5000
2000
6000
10000
Training updates
Training updates
(a) Mastermind -222
(b) Gridworld
Figure 10: Approximation accuracy of Shapley values trained in parallel with the agent under explanation-to--aenn update ratios of 1:1, 2:1, 10:1, and 50:1 in Mastermind-201 222 (left) and Gridworld (right). Each line represents the mean squared error (MSE) between predicted and exact values, averaged over all states and features. Shaded regions indicate the standard error over 20 runs. The agent's expected return and DQN loss also shown to illustrate learning dynamics during training.
Alternatively, could train parametric model of this distribution (8) which may generalise better to sparsely sampled regions but reintroduces the cost learning an additional model.
The null characteristic value #3 (0) is similarly estimated via Monte Carlo sampling from the steady- state distribution:
# (0) = E [s))
(40)
s>)p((s)
This estimate is computationally efficient: it can be computed once by passing the entire replay buffer through the policy network in parallel and averaging the results, then reused across all loss evaluations.
31

---

Prediction Shapley. We train parametric model 0(s;0) : + I 3 to predict the Shapley contributions of each feature of state the agent's value estimate 8 (s). We replace the prediction characteristic igg (C) in the Shapley model loss with a single-sample approximation. At each loss evaluation, a state ~ p= [+|s5 I is sampled from the conditional steady-state distribution, and the characteristic value is approximated using 6T (s'). The model trained to minimise the following
loss:
(00)
E (8)
E p(e)
E
(*((')
(0)
(s;
(41)
of
(1(e))
iE
After training, the model output is corrected to satisfy the efficiency constraint:
1 o'(0") = (s; + |3]
6" (s) (0)
Σ
Oe (s:
(42)
JET
With a sufficiently expressive model class and exact optimisation, the corrected output of the global optimum o(s;0") recovers the exact Shapley values for all (s, e) such that p" (s) 0 and p(C) 0.
Behaviour convergence proof. We now prove that the learning objective in Equation 38 recovers exact Shapley values at the global optimum. This result generalises to the prediction objective in Equation 41 because they share the same structure.
We make the following assumptions:
1. The model 0(s o; 0) is selected from a function class expressive enough to represent the true Shapley value function (( for all state-action pairs (s, a) such that p" (s) > O.
2. The global minimum of the loss (() exists and is attained.
3. States s are sampled from the steady-state distribution (s). actions from the uniform distribution over A, subsets C from fixed distribution over all subsets of F, and samples s' ~ ~ppp(( sC) from the conditional steady-state distribution.
For a fixed state-action pair (s, a) such that p² p=((s > 0, the sampling-han based loss in Equation 38 reduces to:
E ((C) 8' 2p
T(d.a)) - के (0) Σ (s. az;
(43)
iEe
Since (s) is the only term that depends on s', we may equivalently write:
E
E (-pae
(8', # (0) Σ
0)
(44)
P(C)
iEe
By the definition of the behaviour characteristic function, we have:
E C())
72 (C) (0) Σ (s,
(45)
This expression is identical to the reduced Shapley model loss in Equation 25 bringing to the same point as in the proof presented Appendix The remainder the proof proceeds identically, By enforcing the efficiency constraint via an additive correction [22].
('( (',s a; a) + [5] x(s.a) #
Σ (3(s.a;8))
(46)
JES
the unique minimiser of the loss is given by the Shapley values o (#
5 Therefore, the globally optimal parameters 0" recover exact Shapley values for all (s, a) such that p" (*( > 0.
32

---

Remark, The sampling- caupling-based based approximations of the behaviour and prediction Shapley values are unbiased because their objectives recover the original asymptotically unbiased model-based losses in expectation.
Outcome characteristic. return received from state 8 when the agent's 's policy has access only to the features in C:
The outcome characteristic function 05 (C) is defined as the expected
T (e) = E, [G, S s],
(47)
where ju is modified policy that selects actions using the behaviour characteristic function (C)) when features are unknown in state 8, and follows the original policy и elsewhere.
To estimate 1 (e), FasISVERL defines conditioned policy #(a 5; Se, C)) that follows the behaviour characteristic model (ss a e; when 8 Se, and follows the original policy и elsewhere. A parametric value function V(s Se: C; is then trained to predict the expected return under this conditioned policy for each (se, C) pair.
Training this outcome characteristic model requires access to a pre-trained behaviour characteristic model, introducing additional computational cost. To avoid this, we instead bypass the behaviour characteristic model entirely by sampling a state s' ~ p" ( and using x (5', a) as the action probability whenever the conditioned policy ft 8; Ses e) would otherwise act according to the behaviour characteristic model that is, when = Se.
The proposed sampling approach applies only the on-policy formulation of the outcome charac- teristic. In this setting, the value function V(s Se, 3) is trained using a standard bootstrapped DQN-style loss:
( (,,r,",,e,,))-- |r VV'(' Se, C,
V(s 8es 3
(48)
where the buffer ' contains transitions collected from the conditioned policy when sampling is used to approximate the behaviour characteristic at 8p.
An off-policy variant was also considered in Section 3.2 where a parametric state-action value function Q(s, Se, C; B) is trained and the outcome characteristic recovered as
1 (e) RC Σ #(s,a a 82, C) (s, a se, C;
(49)
aEA
Since this recovery step requires querying the behaviour characteristic model, which is no longer available under the sampling-brs based approximation, the method cannot be applied in this setting.
Empirical illustrations. We extend the analysis of replacing characteristic models with sampling, as presented in Figure 3c to prediction Shapley models and on-policy outcome characteristic models across Mastermind-22 2222, Mastermind-332 and Gr Gridworld. Variability across runs is limited to randomness in the training process, with all experimental conditions fixed and the entire pipeline fully seeded. Explanation models are trained using single agent instance across all runs.
Figure I shows the explanation model's is losses over cumulative training updates. Consistent with the findings in Section sampling-based approximations converge faster than their model-based counter- parts and eliminate error propagation from characteristic models. However, minimal computational gains are observed for the outcome characteristic, as the cost of training the behaviour characteristic relatively small compared the substantial cost of training the outcome characteristic itself. This suggests that while sampling effectively reduces error propagation, its impact on computational efficiency for outcome explanations limited.
G Domains
This section provides descriptions of the reinforcement learning domains used in the experiments.
Gridworld 2 a deterministic 2 x 4 environment where the agent's state is defined by its (x,
grid coordinates. The state space is:
8 = (1,, 1), 3), (1, (2), 42
(50)
Fach episode begins start state sampled uniformly from (1, and (2, 1). The agent can take actions North, East, South, and West; actions that would move the agent outside the grid or into
33

---

Behaviour Shapley
Behaviour Shapley
Behaviour Shapley
Exact
Sample
Sample
0.015
Exact Sample
MSE
Model
0.006
Model
0.020
0.004
Modal
0.005
0,005
0.000
0.000
0,000
2000
4000
60000
20000
40000
60000
80000
1000
2000
3000
4000
Training updates
Training updates
Training updates
Prediction Shapley
Prediction Shapley
0.04
Prediction Shapley
Exact
Exact
0.015
Sample
0.03
Sample
0.2
MSE
Model
Model
Sample
0.010
0.02
0.1
0.01-
0.00
0.0
2000
4000
6000
0
50000
100000
150000
1000
2000
3000
Training updates
Training updates
Training updates
Outcome characteristic
Outcome characteristic
Exact
Exact
Sample
Sample
M5F
Model
Model
0.2
0.4
0.6
0.8
20000 60000 80000 100000
Training updates
Training updates
(a) Mastermind-225
(b) Nastermind-335
(c) Gridworld
Figure 11: Approximation accuracy of Shapley and characteristic values trained with sampled, exact or model-based characteristics in Mastermind 222 (left), Mastermind-385 (middle), and Gridworld (right). The empty slot corresponds the outcome characteristic that cannot feasibly be computed exactly in Mastermind-333 Each line represents the mean squared error (MSE) between predicted and exact values, averaged over all states and features. Shaded regions indicate standard error over runs. The dashed lines mark the end of pre-training for the characteristic models the model-based approach.
the missing state (1, are treated as invalid, incurring a reward but leaving the agent's position unchanged.
The terminal states are (1,4) and (2, 4). The agent receives a reward of -1 per decision stage and additional +10 upon reaching terminal state, making this shortest-path problem. The optimal policy moves East in (1, and North in all other states to minimise the number of steps to termination.
Mastermind 2 is a reinforcement learning adaptation of the classic code- bade-breaking game. At the start of each episode, the environment randomly samples hidden code consisting of sequence of letters. The agent must identify the code within a fixed number of guesses. After each guess, the environment returns types of feedback:
Position clue: the number of letters that are both correct and in the correct position.
Misplaced clue: the number of correct letters that are the wrong position.
These values are computed sequentially: letters that are correct and in the correct position contribute to the position clue and are excluded when computing the misplaced clue, which counts only the remaining correct letters in the wrong position.
Each state encodes the agent's current game board: a sequence of previous guesses and the cor- responding feedback. States features: one per letter the guess, plus two features for the position and misplaced clues. Unused guesses are represented using dedicated empty value.
Ceature-based, with each guess represented by fixed number of
The environment can be configured by varying the code length, the number of guesses, and the size of the letter set (i.c. the alphabet). We consider five configurations:
34

---

Mastermind 222: Codes of length drawn from the alphabet B), yielding 4 possible codes. The agent allowed up 2 guesses. The state space contains 53 unique states, each represented by 8 features. Each code corresponds to unique action, giving 4 available actions.
Baatermind-3333 Codes of length 3 drawn from the alphabet (A, B, CJ, yielding 27 possible codes. The agent is allowed up to 3 guesses. The state space contains over 100,000 states, each with 15 features. There are 27 available actions.
Mastermind-443: Codes of length 4 drawn from the alphabet {A, C}, yielding 81 possible codes. The agent is allowed up to 4 guesses. The state space contains over 4.3 x 107 states, each with 24 features. There are 81 available actions.
Mastermind-4533 Codes of length 4 drawn from the alphabet (A, C), yielding 81 possible codes. The agent is allowed up to guesses. The state space contains over 3.5 x 10 states, each with 30 features. There are 81 available actions.
Mastermind-463; Codes of length 4 drawn from the alphabet {A, B, C}, yielding 81 possible codes. The agent allowed up to guesses. The state space contains over 2.8 x 10 states, each with 36 features. There are 81 available actions.
The reward function assigns -1 per guess and provides an additional reward equal to the maximum number of guesses if the agent correctly identifies the hidden code. Episodes terminate when the correct code is guessed or the guess limit is reached.
H Experimental setup and compute resources
All experimental configurations, including byperparameters, training settings, and environment details, are included in the project code's test scripts 3 These scripts are designed to produce fully reproducible and identical results for every experiment. The DQN agent used throughout the experiments is based on the implementation from CIeanRI. 11.
The hyperparameters for all agents, characteristic models, and Shapley models were pragmatically chosen without tuning, as the experiments are intended to illustrate FastSVERL's properties rather than benchmark against alternative methods. Initial values were selected, found to be sufficient for learning, and kept constant across experiments unless they were the specific subject of study or directly linked to design choices being evaluated. The only exception to this approach of the masking value used behaviour and prediction characteristic models to represent unknown features. Although the theoretical framework permits any value outside the support of 8, we found that large magnitude values hindered training stability, possibly due to amplified gradient magnitudes. In contrast, smaller magnitude values, closer the support of 8, resulted in smoother learning and were adopted for all experiments.
the choice
Standard errors in all experiments calculated using the standard error of the mean, corresponding to -ssigma error bars and shaded areas. To better isolate variability arising from the experimental conditions, and not noise introduced by unrelated stochastic factors, we sometimes apply the cor- rection method proposed by Masson and Loftus [16), which adjusts for run-specific differences that unrelated to the treatment effect. Specific sources of variability, such agent initialisation, detailed alongside each experimental setup in the appendix sections.
All experiments were conducted on local workstation equipped with the following specifications:
Processor: Intel i9- 14900K (24 cores, up to 6.0GHz)
GPU: NVIDIA RTX 4090 (24GB VRAM)
Memory: 96GB DDR5 RAM
Storage: 2x NVMe (Samsung 990 EVO) and 4TB SSD (Samsung 870 QVO)
The full research project required substantially more compute than the experiments reported here, including preliminary testing and exploratory experiments not included in the final results.
Fastt code available at: httpp::/ggkhhu...///////////////
35


Reviews:
[{"review_id": "4RUop2mTDP", "reviewer": "Reviewer_2mtF", "summary": "This paper introduces a framework for computing approximate shapley values to explain reinforcement learning policies. In particular, this work uses parametric models to amortize the approximation cost of shapley values across states and features which is compatible with both on-policy and off-policy methods. Explanations are generated to explain an agenet's behavior (policy), performance (reward), and value (future reward).", "questions": "1. Are the estimators used in this paper unbiased?\n\n2. There is little analysis on the single-sample approximation of characteristic values. I would think that this would increase variance given that these are Monte Carlo samples. Is this the case? If not, why?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n* Fast and efficient shapley-based explanations can be useful for a wide-variety of tasks, particularly given the heavy application of RL to many real-world problems.\n\n* The proposed method supports both off-policy and on-policy learning, the latter of which is particularly useful as it supports explanations for non-stationary policies. Explaining how policies change during the course of training could be very useful.\n\n\nWeaknesses:\n* Experiments are only performed over very simple environments (a 2x4 Gridworld in the Appendix, and the Mastermind code-breaking game with codes of up to length 3). While I understand that the main results in the paper used small environments so as to support exact value calculation, this means that the *scalability* of the method is not tested. This is particularly relevant given that the paper claims that FastSVERL is suitable for real-time and real-world problems, yet the evaluation is only conducted over small toy problems that are not at all representative of real-world ones. Does this method scale to more complex environments/policies with respect to both computation time and approximation accuracy?\n\n* No mention is given to bias or fairness of the approximations. Are the estimators used in this paper unbiased?\n\n* There is little analysis on the single-sample approximation of characteristic values. I would think that this would increase variance given that these are Monte Carlo samples. Is this the case? If not, why?", "was_revised": true, "final_justification": "The authors have clarified many of my concerns about the paper. Although I'm still not completely sold on Mastermind as the core evaluation benchmark, I think their claims of a computationally efficient method to calculate Shapley values are fairly well supported so I am updating my score from 3 to 4.", "rebuttal": "Thank you for your time and constructive feedback. We are glad you appreciate the importance of fast and efficient Shapley-based explanations and see the value in developing approximations tailored to RL-specific settings.\n\nWe address each of your points on scalability, estimator bias, and the variance of the single-sample approximation below. We hope our clarifications help to resolve your concerns.\n\n**On Scalability and Evaluation in \"Toy\" Environments (Weakness 1)**\n\nThis is an important point. We agree that the real-world applicability of a method is essential, and scalability is an important part of that. We should clarify that our use of \"real-world\" and \"real-time\" primarily refers to the framework's ability to handle the practical constraints of real-world reinforcement learning. For us, this means providing explanations for a continually evolving agent in real-time, which requires adapting explanations to non-stationary policies and learning from off-policy data. These are both central features of FastSVERL.\n\nRegarding our experimental design, you rightly point out that our deliberate use of smaller domains was a direct consequence of the need to compute exact Shapley values for validation. This is a crucial methodological point: any claims about approximation accuracy can be rigorously tested *only* in these verifiable settings. Therefore, our experiments were necessarily constrained to this regime to ensure scientific validity. \n\nIt is only within this verifiable regime that we can demonstrate how FastSVERL’s accuracy scales. Our Hypercube experiments were designed specifically for this purpose, and they showed that the computational cost of achieving accurate explanations scales well with both the number of states and features. This graceful scaling of accuracy, in the one region where it can be principally measured, suggests the method should continue to perform well in larger domains. Furthermore, the results on Mastermind-333 in our supplement—with over 100,000 states and 15 features ($2^{15}$ subsets)—already push the boundary of what can be considered a \"toy\" domain for Shapley value computation, providing further evidence of the framework's capabilities.\n\nThat said, while we cannot directly assess accuracy in larger domains, we agree that general scalability deserves greater focus. Motivated by your feedback and that of others, we have run new experiments on substantially larger versions of Mastermind. We continue to use Mastermind specifically to allow fine-grained control over complexity. We believe these new domains, the largest having 36 features and over 280 billion states, move beyond the \"toy\" setting, particularly from a Shapley approximation perspective where the summation is over $2^{36}$ feature subsets.\n\nWhile it is impossible to measure accuracy against an exact ground truth in these domains, our theoretical guarantees and the strong accuracy demonstrated in verifiable settings suggest that FastSVERL's models will converge towards the correct values. Therefore, our goal with these new experiments is to empirically test three key properties at scale: (1) if the models indeed converge, (2) the stability of that convergence across runs (measured by standard error), and (3) if the computational cost of convergence remains manageable.\n\nDue to space constraints, we present a representative subset of results below for the behaviour characteristic and Shapley models, reporting the mean training updates to convergence and the final loss on their training objectives over 10 runs. It is important to clarify that this loss is for the training objective and is not a measure of ground-truth accuracy. The characteristic model's loss is designed to converge to a non-zero value as it learns to predict an expectation over the unobserved features. In contrast, the Shapley model's loss is designed to converge to zero, but this indicates it has successfully learned to explain the values provided by the characteristic model, not necessarily the ground truth.\n\n**Table 1: Convergence and Stability of Behaviour Explanations**\n| Domain | Explanation Model | Updates to Converge (Mean ± Std. Err.) | Final Loss (Mean ± Std. Err.) |\n| :--- | :--- | :---: | :---: |\n| **Mastermind-443** | Behaviour Characteristic | $(1.13 \\pm 0.177) \\times 10^6$ | $0.0038 \\pm 0.000018$ |\n| (24 features, $\\ge 4.3 \\times 10^7$ states) | Behaviour Shapley | $(7.99 \\pm 0.902) \\times 10^5$ | $0.0013 \\pm 0.000037$ |\n| **Mastermind-453** | Behaviour Characteristic | $(1.30 \\pm 0.150) \\times 10^6$ | $0.0036 \\pm 0.000021$ |\n| (30 features, $\\ge 3.5 \\times 10^9$ states) | Behaviour Shapley | $(7.91 \\pm 0.503) \\times 10^5$ | $0.0010 \\pm 0.000032$ |\n| **Mastermind-463** | Behaviour Characteristic | $(1.25 \\pm 0.174) \\times 10^6$ | $0.0037 \\pm 0.000013$ |\n| (36 features, $\\ge 2.8 \\times 10^{11}$ states) | Behaviour Shapley | $(6.56 \\pm 0.816) \\times 10^5$ | $0.0019 \\pm 0.000044$ |\n\nThe results in Table 1 confirm that the models successfully converge in these large-scale domains. The standard error across multiple runs demonstrates that this convergence is stable and highly reliable. Most importantly, the number of training updates required for convergence remains similar even as the domain complexity grows.\n\nWe also demonstrate that the significant computational advantages of the sampling-based approximations we propose (Section 5) hold at scale. This method is a key contribution of our work, and Table 2 confirms its efficiency benefits in these large domains.\n\n**Table 2: Total Computational Cost for Behaviour Explanations**\n| Domain | Total Updates (Model-Based) (Mean ± Std. Err.) | Total Updates (Sampling-Based) (Mean ± Std. Err.) |\n| :--- | :---: | :---: |\n| **Mastermind-443** | $(1.84 \\pm 0.136) \\times 10^6$ | $(5.69 \\pm 0.498) \\times 10^5$ |\n| **Mastermind-453** | $(2.09 \\pm 0.095) \\times 10^6$ | $(7.25 \\pm 0.638) \\times 10^5$ |\n| **Mastermind-463** | $(1.91 \\pm 0.104) \\times 10^6$ | $(6.06 \\pm 0.584) \\times 10^5$ |\n\nBy eliminating the need to pre-train a characteristic model, the sampling-based approach reduces the total computational cost by over $2\\times$ across all large-scale domains tested. This suggests that the substantial efficiency gains are consistent.\n\nWe hope these clarifications and new results will help to address your concerns about scalability.\n\n**On the Bias of the Estimators (Weakness 2, Q1)**\n\nThank you for this question. To clarify, the estimators for the behaviour and value estimation characteristics are asymptotically unbiased, as are all of our Shapley estimators. These properties stem directly from their learning objectives, as proven in the supplementary material.\n\nFirst, our characteristic models for behaviour and value estimation are trained to minimise an expected squared-error loss. The unique global minimiser of this objective is the true conditional expectation (Appendix B). An estimator that converges to this true mean is, by definition, asymptotically unbiased.\n\nSecond, all of our Shapley models are trained to solve a weighted least-squares problem whose unique solution is the true Shapley value vector for the given characteristic function (Appendix A). Therefore, these estimators are also asymptotically unbiased. The same property holds for our sampling-based objective, as we prove in Appendix F and address in the following point.\n\n**On the Variance of the Single-Sample Approximation (Weakness 3, Q2)**\n\nThis is a great question. Your intuition is correct: using a single Monte Carlo sample at each training step does introduce variance into the gradient updates (though one could reduce this variance by increasing the number of samples per update).\n\nHowever, the key insight is that our parametric Shapley model is designed to handle this, effectively averaging out the variance over the entire training process. By optimising over many of these noisy but computationally cheap samples, the model learns to predict the expected value, guided by an objective that, as we prove in Appendix F, is unbiased in expectation and identical to the original model-based loss.\n\nThis approach presents a favourable trade-off: we accept higher variance in individual gradient updates to completely eliminate the propagation of approximation error from a pre-trained characteristic model. The empirical results strongly support this design choice. As shown in Figure 3c in the main paper and the results in Figure 7 of the supplement, the sampling-based approach converges significantly faster and to a lower final error than the model-based approach. This is precisely because it avoids compounding the approximation errors from the upstream characteristic model, leading to a more accurate and efficient final result.\n\nWe thank you again for your detailed and constructive feedback. We hope that our response and the new large-scale experiments will help to address your initial concerns. We will incorporate the full set of these new results, along with all the clarifications discussed, into the final version of the manuscript, and hope this new evidence will be reflected in your updated evaluation of our work."}, {"review_id": "lpZgDhYziz", "reviewer": "Reviewer_boW8", "summary": "Shapley values are widely used to provide interpretability to RL behaviors, however, their exponential computation cost makes them difficult to compute in many real problems. \nThe manuscript introduces FastSVERL, a scallable method for approximating Shapley values in the context of RL. It takes ideas from FastSHAP and extends them to the RL context. It particularly deals with sequential, policy-based RL environments, handling temporal dependencies and off-policy learning.", "questions": "- How does the method perform in large-scale, high-dimensional RL benchmarks beyond the toy domains used?\n- What is the overhead introduced by calculating the parametric model? What is its O shape?\n- Can FastSVERL be applied to any RL problem? Are there any requirements in terms of discrete/continuous domains? Continuous or episodic? Fully observable or partially observable?...\n- Can parametric models be computed in all cases? What amount of data is needed?", "limitations": "At the end of the paper, the authors state the main limitation of this work: its practical deployment in real-world to test its robustness is missing. \nThere is no potential negative societal impact of this work.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "STRENGTHS:\n- The proposal is very well motivated (need for practical scalable ways of providing interpretability in RL).\n- Specifically deals with RL challenges such as temporal dependencies, evolving behaviours, and learning from off-policy data.\n- The performed experimentation (in the supplementary material) is very solid, analyzing accuracy, efficiency and scalability.\n\nWEAKNESSES:\n- While the goal is to provide explainability to complex RL scenarios, the given experiments are only on moderate-complexity tractable problems.\n- As reported by the authors, the manuscript establishes a foundation for scalable Shapley-based explanation in RL, but the practical deployment is yet to be fully done.\n- The proposal relies on parametric models, whose generalization capabilities can be limited.", "was_revised": true, "final_justification": "Authors have successfully addressed the raised concerns and appropriately engage in insightful discussions, and thus, I reiterate my original ranking of acceptance.", "rebuttal": "Thank you for your supportive and positive review. We are grateful for your kind words on the paper's motivation, the handling of reinforcement learning challenges, and the solid experimentation. We are happy to address the points and clarify the questions you raised.\n\n**On Scalability and Practical Deployment (Weakness 1, 2 & Q1)**\n\nWe agree that demonstrating how the proposed methods scale to more complex, large-scale benchmarks is an important next step. Our use of smaller domains was a direct consequence of the need to compute exact Shapley values for validation. This is a crucial methodological point: any claims about approximation accuracy can be rigorously tested only in these verifiable settings. Therefore, our experiments were necessarily constrained to this regime to ensure scientific validity.\n\nWhile we cannot directly assess accuracy in larger domains, we were motivated by your feedback and that of others to provide further evidence of the framework's scalability. We have run new experiments on substantially larger versions of Mastermind. We believe these new domains, the largest having 36 features and over 280 billion states, move beyond the \"toy\" setting, particularly from a Shapley approximation perspective where the summation is over $2^{36}$ feature subsets.\n\nThe theoretical guarantees and strong accuracy demonstrated in verifiable settings suggest that FastSVERL's models will converge towards to the correct values. Therefore, our goal with these new experiments is to test three key properties at scale: (1) if the models indeed converge, (2) the stability of that convergence across runs (measured by standard error), and (3) if the computational cost of convergence remains manageable.\n\nDue to space constraints, we present a representative subset of results below for the behaviour characteristic and Shapley models, reporting the mean training updates to convergence and the final loss on their training objectives over 10 runs. It is important to clarify that this loss is not a measure of ground-truth accuracy. The characteristic model's loss is designed to converge to a non-zero value as it learns to predict an expectation over the unknown features. In contrast, the Shapley model's loss is designed to converge to zero, but this indicates it has successfully learned to explain the values provided by the characteristic model, not necessarily the ground truth.\n\n**Table 1: Convergence and Stability of Behaviour Explanations**\n| Domain | Explanation Model | Updates to Converge (Mean ± Std. Err.) | Final Loss (Mean ± Std. Err.) |\n| :--- | :--- | :---: | :---: |\n| **Mastermind-443** | Behaviour Characteristic | $(1.13 \\pm 0.177) \\times 10^6$ | $0.0038 \\pm 0.000018$ |\n| (24 features, $\\ge 4.3 \\times 10^7$ states) | Behaviour Shapley | $(7.99 \\pm 0.902) \\times 10^5$ | $0.0013 \\pm 0.000037$ |\n| **Mastermind-453** | Behaviour Characteristic | $(1.30 \\pm 0.150) \\times 10^6$ | $0.0036 \\pm 0.000021$ |\n| (30 features, $\\ge 3.5 \\times 10^9$ states) | Behaviour Shapley | $(7.91 \\pm 0.503) \\times 10^5$ | $0.0010 \\pm 0.000032$ |\n| **Mastermind-463** | Behaviour Characteristic | $(1.25 \\pm 0.174) \\times 10^6$ | $0.0037 \\pm 0.000013$ |\n| (36 features, $\\ge 2.8 \\times 10^{11}$ states) | Behaviour Shapley | $(6.56 \\pm 0.816) \\times 10^5$ | $0.0019 \\pm 0.000044$ |\n\nThe results in Table 1 confirm that the models successfully converge in these large-scale domains. The standard error across multiple runs demonstrates that this convergence is stable and highly reliable. Most importantly, the number of training updates required for convergence remains similar even as the domain complexity grows.\n\nWe also demonstrate that the significant computational advantages of the sampling-based approximations we propose (Section 5) hold at scale. This method is a key contribution of our work, and Table 2 confirms its efficiency benefits in these large domains.\n\n**Table 2: Total Computational Cost for Behaviour Explanations**\n| Domain | Total Updates (Model-Based) (Mean ± Std. Err.) | Total Updates (Sampling-Based) (Mean ± Std. Err.) |\n| :--- | :---: | :---: |\n| **Mastermind-443** | $(1.84 \\pm 0.136) \\times 10^6$ | $(5.69 \\pm 0.498) \\times 10^5$ |\n| **Mastermind-453** | $(2.09 \\pm 0.095) \\times 10^6$ | $(7.25 \\pm 0.638) \\times 10^5$ |\n| **Mastermind-463** | $(1.91 \\pm 0.104) \\times 10^6$ | $(6.06 \\pm 0.584) \\times 10^5$ |\n\nBy eliminating the need to pre-train a characteristic model, the sampling-based approach reduces the total computational cost by over $2\\times$ across all large-scale domains tested. This suggests that the substantial efficiency gains are consistent.\n\nWhile we agree that the practical deployment of FastSVERL is a key step towards validating its effectiveness, this is beyond the scope of the current manuscript, and we hope instead these new results will help to address your questions about scalability.\n\n**On Computational Overhead and Complexity (Q2)**\n\nThank you for this question. The main computational overhead of the framework is the training of the parametric models. Once trained, the cost to generate an explanation for any given state is a single, efficient forward pass through the networks.\n\nFor the standard case of explaining a fixed, stationary policy post-hoc, the training overhead consists of two main parts:\n1.  **Data Collection:** This is the cost of collecting experience to approximate the steady-state distribution. This can be running the fixed policy (on-policy) or mitigated by reusing a pre-existing data buffer (off-policy).\n2.  **Model Training:** This is the cost of training the parametric models. For the behaviour and value estimation characteristics and the Shapley models, this is analogous to a standard supervised learning training cost. For the performance characteristic, it is that of a standard reinforcement learning process.\n   \nIn the setting where the explanation models are updated continuously alongside the agent's policy, the only additional computational cost beyond the agent's standard policy updates is the cost of updating the explanation models, which for each model is roughly equivalent to the agent's own update cost at each training step.\n\nRegarding the complexity (`O` shape), providing a single theoretical `O` notation for the training of deep models is difficult, as it depends on many hyperparameters. However, the Hypercube experiments (Figure 2 in the paper and supplement) provide some empirical evidence of the framework's scaling properties. The results show that in this verifiable setting, the number of training updates required for convergence scales approximately polynomially, as opposed to the exponential complexity of computing Shapley values exactly.\n\n**On General Applicability and Requirements (Q3)**\n\nThis is an excellent question, and we're glad you prompted us to clarify the framework's scope. We will add a more detailed discussion of these points to the final paper.\n\nIn general, the framework applies to any problem where an agent observes, acts, and receives rewards for those actions. To be more specific:\n* **State/Action Spaces:** The work in this paper focuses on discrete action spaces because the behaviour characteristic is defined over them. However, the theoretical SVERL framework provides explanations for continuous action spaces, and the proposed approximation methods can be readily adapted. The parametric nature of the models also allows the framework to be applied to continuous state spaces; the primary challenge in such settings becomes approximating the steady-state distribution, a point we discuss below and in the supplement, which we will move to the main body.\n* **Episodic/Continuous Tasks:** The framework applies to both episodic and continuous tasks.\n* **Observability:** The framework can be applied in partially observable settings, where attributions would be over the features of the agent's observation. This setting is particularly interesting for explaining non-stationary policies, as such policies are often a direct consequence of partial observability.\n\n**On Generalisation and Data Requirements (Weakness 3 & Q4)**\n\nRegarding the limitations of parametric models, you raise a valid point about the general challenges of out-of-distribution generalisation. However, our primary goal in this work is amortisation, which is a form of in-distribution generalisation. Neural networks are well-suited for this task, as they learn to interpolate within the training distribution, allowing them to generalise across the vast combinatorial space of states and feature subsets. Furthermore, a parametric approach was a necessary choice to enable the continual learning aspect of the framework, as non-parametric sampling methods cannot be efficiently updated to handle non-stationary policies.\n\nYour question about data requirements is also very important. The practical applicability of the framework depends on collecting enough data to form a reasonable approximation of the relevant steady-state distributions. The amount of data required is analogous to that needed to learn a value or policy function in the same domain, as all three tasks fundamentally rely on the experience providing a sufficient sample of the underlying state visitation distribution. This is a key challenge for future work, especially in large or continuous state spaces where a frequentist approximation with a buffer is necessarily a sparse sample of the true distribution. We briefly discuss this important topic in the supplement, but will move it to the main body in our revision.\n\nThank you again for your supportive review and helpful feedback, and questions; they have helped to clarify many important aspects of the framework, and we will incorporate all of these clarifications and our new results into the final version of the paper. We hope this reinforces your positive assessment of our work."}, {"review_id": "4FlYXj3Wna", "reviewer": "Reviewer_ayeY", "summary": "The submission studies Shapley values for explaining reinforcement learning models. Similar to FastSHAP, the submission proposes FastSVERL to amortize estimation of Shapley values for reinforcement learning as introduced in SVERL. The submission comprehensively discusses amortization for both behavior, performance, and value estimation. Experiments on both synthetic and real-world datasets show the behavior of the proposed methods and discuss its limitations and potential future research directions.", "questions": "* Several times throughout the paper it is stated that \"Shapley value explanations do not generalize\" hence are \"inefficient for repeated explanations\". Could the authors clarify what they mean? What does it mean for an explanation to \"generalize\"?\n\n* Lines 202-204: I have a general question about explaining the performance of a model. Why is the explained state $s_e$ always the start state? How could one explain the performance of the model at any state $s$ of a trajectory which may not be the first one? Does the modified policy take into account that the trajectory may pass through state $s_e$ again later? What would happen at that point? \n\n* Could the authors clarify what they mean by \"earlier policies\"? Does this assume access to the model during training to collect observations? This is an important consideration for interpretability because usually we assume being given a fixed predictive model after training is completed. If FastSVERL requires access to trajectories during training, it should be made explicit and its implications discussed.\n\n* Could the authors expand on how ground-truth for MSE evaluation is obtained? Does \"exact values\" refer to exact Shapley values? Wouldn't these require expectation over an unknown distribution to mask the inputs?\n\n* Lines 325-326: I am missing how it can be that the models start with low error if the policy is random. If the policy is random, then there is \"nothing to explain\", so what are the models learning?\n\n* Does the sampling strategy apply to performance and value estimation as well?\n\n* It might be interesting to expand on the sampling strategy, since it seems to outperform the models. After reading the last paragraph in Sec. 5, it was unclear to me whether the message of the submission is that sampling is better than training models, which can be expensive and difficult to do in practice.\n\n---\n\n**Minor comments**\n\n* Abstract: \"real-time\" estimation feels outside the scope of the submission. Experiments do not discuss latency or role of real-time estimation for downstream tasks.\n* The difference between behavior, performance, and value could be spelled out more clearly for readers coming from the interpretability community who may not be familiar with reinforcement learning.\n* Lines 40-43 are too technical and hard to understand at this point in the paper.\n* Line 78: throughout the paper, the cardinality of the state space $\\lvert \\mathcal{X} \\rvert$ is mentioned several times. Is the state space assumed to be finite in this work? In several applications, this may not be finite. Same question about $\\mathcal{S}$\n* Line 82: it is unclear what \"state-of-the-art\" means in this context.\n* Line 86: can $p(C)$ be any distribution or should it be a uniform to retrieve the conditional expectation?\n* Eq. (8) typo in conditioning, should be $S_t = s^C$? Also $G_t$ is undefined\n* Line 213: reference for \"DQN-style loss\"?\n* Eq. (14): $V$ and $V'$ are also undefined\n* Line 337: it is unclear what \"Shapley models\" refer to", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. Explainability for reinforcement learning is a challenging setting that is worth exploring\n2. Amortization has proven to be a successful strategy in the past\n3. Experimental analysis is thorough\n\n**Weaknesses**\n\n1. Presentation could do a better job at separating behavior, performance, and evaluation estimation, which is hard to follow\n2. Certain claims are hand-wavy and could be clarified\n3. Sampling strategies could be given more space\n\nI have a few clarifying questions and I am looking forward to discussing with the authors!", "was_revised": true, "final_justification": "The submission builds on existing methods: SVERL + FastSHAP, to provide explanations for deep reinforcement learning models, both at the performance, behavior, and value estimation levels.\n\nThe method and findings are interesting to the community, and evidence supports the claim made in the paper, hence I recommend acceptance. \n\nIt will be important to include in the revised version of the paper the main points of discussion that were raised by reviewers. In particular, larger-scale experiments, and to clarify the message between models and sampling strategies.", "rebuttal": "Thank you for your thorough and constructive review, along with your positive assessment of our work. Your questions and comments have improved the paper's clarity significantly; we address each below.\n\n**On Presentation, Narrative, and the Sampling Strategy (Weaknesses 1 and 3, Q6-7)**\n\nWe apologise that the separation between the explanation types could be clearer, and intend to improve this in the final version.\n\nWe also agree that it would have been interesting to expand on the sampling strategies in Section 5 if we had more space. It is an interesting direction for future work. For this paper, however, our narrative choice was to focus more on establishing a comprehensive, model-based framework, which provides a solid foundation that the community can build upon. The sampling-based approach is therefore presented as one such promising extension. \n\nThe message of our submission is not that the sampling-based method is a definitive replacement for the model-based one. While it shows efficiency and accuracy gains in our experiments, such a claim would require the same rigorous evaluation that the foundational model-based framework received across all the different settings considered in the paper.\n\nTo answer your final question on this topic, a full formulation for applying the sampling-based approach to *value estimation* is detailed in Appendix F. For *performance*, while theoretically possible, generating a Monte Carlo return for each sample would likely be less sample-efficient than learning the characteristic function with the bootstrapping method in the paper—a standard motivation for algorithms such as Q-Learning.\n\n**On Your Main Questions (Q1-5)**\n\nBelow, we clarify several key aspects of the framework by addressing your first five questions in the order they appear.\n\n**On the \"Generalisation\" of Explanations (Q1).**\nTo clarify our use of the term \"generalise\": when we state in the paper that sampling-based estimates \"do not generalise across states\", we mean that a Monte Carlo approximation of a Shapley value is specific to a single input state. To approximate the Shapley value for a different state, the entire computationally expensive estimation must be repeated from scratch. \n\nIn contrast, FastSVERL's parametric models, implemented as neural networks, learn a single function that shares its parameters across all inputs. This means that when approximating the Shapley values for a new state, the models can reuse what they have already learned from other similar states, rather than starting the entire estimation process over again. It is this ability to leverage shared patterns across the state and feature space that we refer to as generalisation.\n\n**On Explaining Performance at Any State (Q2).**\nWe apologise for the confusing use of \"start state\" in the paper, by which we meant the state from which the performance characteristic's expected return is measured; that was a large oversight. We should have used the term \"state-to-be-explained\", $s_e$, which can be any state in a trajectory, not just the first. The proposed method is specifically designed to generate performance explanations for any state.\n\nTo clarify the mechanism, the framework uses a single *conditioned policy*, $\\hat{\\pi}(a|s; s_e, \\mathcal{C})$. This is a general policy that describes how the agent should behave in *any* current state $s$, given that the agent's policy has been modified at a specific state-to-be-explained $s_e$ by the feature subset $\\mathcal{C}$ being unknown.\n\nLet's consider one specific pair $(s_e, \\mathcal{C})$. For this pair, the general policy collapses into a standard policy with a simple rule: if the agent's current state $s$ matches the state being explained, $s_e$, it acts according to the behaviour characteristic model; otherwise, it follows its original policy $\\pi$. This directly answers your question about what happens if the trajectory passes through $s_e$ again: this is accounted for, and the agent will indeed act according to the behaviour characteristic model again.\n\nFrom this policy, we can learn a single value function, $V(s | s_e, \\mathcal{C})$, which estimates the expected return from *any* state $s$, given that the policy is modified at a particular state $s_e$ with unknown features $\\mathcal{C}$. While we also learn the values for states $s$ not equal to $s_e$, for bootstrapping during training, our ultimate goal is to approximate the performance characteristic for the pair $(s_e, \\mathcal{C})$. This is recovered by querying the learned function at the particular state $s = s_e$, that is, $V(s_e | s_e, \\mathcal{C})$.\n\nSince this entire process can be applied to any state-coalition pair, the single learned value function allows us to estimate the performance characteristic for any state in an environment that we wish to explain.\n\n**On 'Earlier Policies' and Training Access (Q3).**\nThank you, we are happy to clarify this critical point about the framework's flexibility. To be clear, the framework *does not* assume access to the model during training to collect observations and can be applied post-hoc to a fixed, trained model.\n\nThe comment you refer to in line 209 specifically concerns the choice between the two methods provided for approximating the performance characteristic:\n\n* The **on-policy method** (Eq. 14) is designed for precisely the standard scenario you describe. Assuming one is given a fixed predictive model after training is complete, explanations are generated post-hoc by using on-policy data from the trained model.\n\n* The **off-policy method** (Eq. 15) provides an alternative option. In this case, \"earlier policies\" refers to the policies that generated a pre-existing buffer of data. Accessing such a buffer is common practice for reinforcement learning agents, and this method requires only the historical *data*, not the models that generated it.\n\nWe deliberately provide both methods to make the framework general. It fully supports the standard post-hoc explanation setting you describe. The off-policy approach then extends this by opening the door to greater sample efficiency and, crucially, the ability to explain agents *during* the training process itself. This makes the proposed framework more flexible and less restrictive than the standard post-hoc-only paradigm.\n\n**On Obtaining Ground-Truth for MSE (Q4).**\nYes, \"exact values\" refers to exact Shapley values. You are correct that computing these requires taking an expectation over the steady-state distribution to \"mask\" the unknown features. Fortunately, for the tractable domains used in our validation experiments, this conditional distribution can be accurately estimated from experience. The required expectation for the characteristic values can then be computed directly. These exact characteristic values are then plugged into the Shapley value formula to yield the final ground-truth attributions. This is the same approach used to compute exact values in the original work on SVERL [1, 2].\n\n**On Low Error for a Random Policy (Q5).**\nFor an initial random (e.g. uniform) policy, the agent's actions and value estimates are independent of the state features. Therefore, the true Shapley value for any feature's contribution is zero. A randomly initialised neural network with small weights naturally outputs values close to zero, and can very quickly learn to predict zero for all inputs. The model thus starts with a low error because its initial predictions are already very close to the ground-truth attributions of zero.\n\n**On Your Minor Comments**\n\nThank you for these comments, which will help to improve the paper's clarity. We will address each in the order they appear.\n\n* **Abstract (\"real-time\"):** By \"real-time\", we mean that explanations can be generated continually as an agent learns and acts, at a cost equivalent to selecting actions: a single forward pass through a network. We will clarify this and reconsider our wording in the final version.\n* **Difference between explanations:** This is a great point. We agree and will expand this section in the final version to be clearer for our readers.\n* **Lines 40-43:** Thank you, that's a good point, and we are happy to revise.\n* **Line 78 (Finite state space):** To clarify, the framework does not assume a finite state space and is generally applicable to either setting. The primary challenge in infinite state spaces is likely to be approximating the steady-state distribution. We discuss this in the supplement, but will clarify it in the main body of the paper.\n* **Line 82 (\"State-of-the-art\"):** We agree this is vague and will change the phrasing to \"one approach\" in the final version.\n* **Line 86 (Distribution $p(\\mathcal{C})$):** Theoretically, any distribution $p(\\mathcal{C})$ with full support over the feature subsets can be used in Equation 4. This is clarified in the appendix proofs for the characteristic functions we proposed.\n* **Eq. (8) typo:** Thank you for the close reading. The conditioning on $s$ is correct, as $\\tilde{v}_{s}^{\\pi}(\\mathcal{C})$ is the expected return measured from that specific state $s$. We apologise that the return $G_t$ was undefined and will fix this oversight.\n* **Line 213 (DQN-style loss):** Thank you, we will add the reference.\n* **Eq. (14) undefined terms:** The value function $V$ is defined in line 206, and the target network $V'$ is introduced in line 216. However, the specific role of the target network could be more explicit; we will clarify this in our revision.\n* **Line 337 (\"Shapley models\"):** This is our shorthand for the parametric models trained to predict Shapley values. We will define this term explicitly to avoid confusion.\n\nThank you again for your constructive review, which helped improve the paper's clarity in several areas. We hope our response helps to address your initial concerns. We will incorporate these clarifications into the final manuscript, and we hope this will be reflected in your updated evaluation of our work."}, {"review_id": "GeoGusN1jK", "reviewer": "Reviewer_1dbC", "summary": "This paper introduces a scalable method for approximating Shapley values for RL policies. This method can be used to quantify the influence of individual state features on the agent's behavior, its performance, and its value estimates. The paper also addresses two key challenges: computing Shapley values from off-policy data, and doing so while the policy is being trained.", "questions": "1. *Section 3.2:* How is the single-conditioned policy \"defined\"? I understand it as a policy that takes the initial action based solely on the features in $\\mathcal{C}$ and then follows the actual policy afterward. But how are the probabilities computed for that first action? Are they approximated using a model trained with the loss defined in Equation 12?\n\n2. *Section 3.2:* How is $\\mathcal{B}$ obtained? The text states, \"we uniformly sample $s_e$ and $\\mathcal{C}$ at each decision stage.\" What does this mean? Does it imply the environment can be reset to any arbitrary state?\n\n3. *Section 3.2:* Could you elaborate on how the off-policy (Equation 14) and on-policy (Equation 15) versions work? This is confusing, as in both cases $s'$ and $a$ seem to be obtained while following the current policy. Additionally, Equation 15 is described as SARSA-style (an on-policy method), yet is presented as off-policy, which is contradictory.\n\n4. *Section 3.2* presents both an on-policy and off-policy method. Then, in Section 4, the challenge of learning the Shapley approximation from off-policy data is raised again, this time proposing importance sampling to reweight the loss and address distribution shift. Why is the first off-policy approach insufficient in this context? Is this instead referring to **offline** learning rather than off-policy learning?\n\n5. Could you provide intuition for why the method proposed in Section 5 converges faster than the one that uses the exact behavior characteristic function?", "limitations": "Although there is no specific section that explicitly discusses the limitations, some limitations are mentioned throughout the paper.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n1. The paper tackles the important problem of interpreting RL policies. It starts from a computationally expensive method and proposes an approximation that improves scalability.\n\n2. The experimental setting is sound, and the results are promising. The accuracy of the approach is compared against the exact method, showing low MSE. Moreover, the method is adapted for use with offline data collected from a different policy, and for explaining continually evolving policies, both of which are important use cases.\n\n**Weaknesses:**\n\n1. The introduction and background sections argue that RL poses additional challenges compared to supervised learning, particularly due to the temporal dependencies between state features and returns. However, this concern primarily applies when measuring the influence of state features on performance, not when measuring their influence on behavior or value estimation. The proposed approach for the latter two cases appears equivalent to existing supervised learning methods described in the background section. This raises the question of why these are presented as novel contributions if existing techniques can be directly applied.\n\n2. Section 3.2 is somewhat difficult to follow, particularly the discussion of the performance characteristic function. See Questions 1–3 below.\n\n3. Section 5 introduces a new method that, according to Figure 3c, appears more accurate and more scalable than the previously proposed method. It is unclear why this method is only introduced toward the end of the paper and not analyzed in greater depth earlier. This is especially puzzling given that, in line 180, the Monte Carlo estimation approach is dismissed for lacking scalability and generalization across different feature subsets.\n\n4. Aside from demonstrating accuracy on a toy problem where Shapley values can be computed exactly, the paper lacks experiments that showcase the method’s usefulness for explaining the behavior of RL policies in high-dimensional and complex environments.", "was_revised": true, "final_justification": "The authors addressed many of my concerns and provided an example that partly demonstrates the usefulness of the proposed approach.", "rebuttal": "Thank you for your detailed review and positive feedback. We address the points you raised below, starting with clarifying sections 3.2 and 4, which could have been clearer.\n\n**On the Conditioned Policy and Data Collection (Q1-2)**\n\nWe will answer questions 1 and 2 together, as they relate to the same mechanism. To clarify, the framework *does not* assume the ability to reset the environment. The buffer $\\mathcal{B}$ for the on-policy performance characteristic is collected via a continuous, online interaction.\n\nThe process for collecting a single transition at decision stage $t$ is as follows:\n1.  The agent is in its current environment state, $s_t$.\n2.  Separately, a state-to-be-explained, $s_e$, and a feature subset, $\\mathcal{C}$, are sampled (e.g. from a replay buffer and a uniform distribution, respectively). These are used as conditioning parameters for the value function we are learning, $V(s | s_e, \\mathcal{C})$.\n3.  Conditioning on the sampled $s_e$ and $\\mathcal{C}$ collapses the single conditioned policy $\\hat{\\pi}(a|s;s_e, \\mathcal{C})$ into a standard policy for this step. The agent takes an action, $a_t$, based on this policy: if its current state $s_t$ happens to be the state being explained, $s_e$, it acts according to the behaviour characteristic model (Equation 12); otherwise, it acts according to its original policy $\\pi$.\n4.  This action produces a single transition $(s_t, a_t, r_{t+1}, s_{t+1})$ in the environment, which is stored in buffer $\\mathcal{B}$.\n\nThis process is repeated at every decision stage, allowing us to learn a single value function $V(s | s_e, \\mathcal{C})$ that is capable of estimating the expected return from *any* state $s$, given that the policy is modified at state $s_e$, for *any* state-coalition pair $(s_e, \\mathcal{C})$. The values $V(s | s_e, \\mathcal{C})$ for states $s$ not equal to $s_e$, for each pair $(s_e, \\mathcal{C})$, are learnt to use for bootstrapping during training. However, our ultimate goal is to approximate the performance characteristic for $s_e$ and $\\mathcal{C}$. This is recovered by querying the learned function with $s = s_e$, that is, $V(s_e | s_e, \\mathcal{C})$. This mechanism allows us to learn the required values completely on-policy, without any environment resets.\n\n**On the On-Policy vs. Off-Policy Methods in Sec 3.2 (Q3)**\n\nWe apologise for the confusing terminology, particularly \"SARSA-style\", which we will remove. To clarify the two methods:\n\n* The **on-policy** method (Eq. 14) is so-named because it populates its buffer $\\mathcal{B}$ by collecting new experience. As just described, the entire tuple $(s, a, r, s')$ is generated *by the conditioned policy itself*.\n\n* The **off-policy** method (Eq. 15) is so-named because it uses a buffer $\\mathcal{B}$ where the experience tuples $(s, a, r, s')$ were generated *by a different, arbitrary policy*. The update rule is indeed off-policy because the data source is different from the policy being evaluated. It is more similar to the off-policy algorithm Q-learning: instead of bootstrapping from the value of the *maximising* action, it bootstraps from the value of the target policy's chosen action $a'$; the similarity of using the action $a'$ was our reason for the poor choice of the \"SARSA-style\" terminology.\n\n**On the Off-Policy Methods in Sec 3.2 vs. Sec 4 (Q4)**\n\nThis question highlights a distinction that we should have made clearer. The method in Section 3.2 is not insufficient; rather, the two sections present distinct tools for approximating different parts of the explanation framework in the off-policy setting.\n\n* The off-policy method in Section 3.2 is a specific tool designed to approximate the *performance characteristic*.\n* The importance sampling method in Section 4 is a more general technique used to approximate the *behaviour and value estimation characteristics* from a replay buffer.\n\nThe separation was narrative-driven, reflecting two different scenarios for data collection. Section 3.2 considers a setting where deliberate online interaction for explanations is possible, making off-policy learning a tool for more sample-efficient data reuse. In contrast, Section 4 addresses the more constrained setting where explanations must be learnt passively from the agent's existing replay buffer.\n\nYou are correct that the scenario in Section 4 is similar to \"offline learning\", but we prefer the more general term \"off-policy learning\" because \"offline\" typically implies a static, fixed dataset, which would be incompatible with our continual learning setting where the replay buffer is constantly updated.\n\n**On the Narrative Structure and Section 5 (Weakness 3, Q5)**\n\nOur narrative choice was to first establish a general, model-based framework to provide a solid foundation upon which extensions could be built, with the method in Section 5 being one such extension. We initially dismissed purely sampling-based methods because they are computationally expensive for repeat explanations, making them ill-suited for the core challenge of explaining non-stationary policies. In contrast, the hybrid approach in Section 5 preserves the crucial ability to continually and efficiently update explanations for evolving agents by integrating single-sample estimates directly into the parametric training.\n\nYour final question (Q5) raises an excellent point about why the sampling-based method converges faster than using the exact characteristic function. We believe this counter-intuitive result is due to the noise from the single-sample approximations acting as a *stochastic regulariser*. The noise prevents each model update from overfitting to a batch's exact targets and encourages the optimiser to learn the smoother, underlying expected function more efficiently. This phenomenon has also been explored in contemporary work on stochastic amortisation [6].\n\n**On the Originality of Behaviour and Value Estimation Approximations (Weakness 1)**\n\nWe agree that, when viewed in isolation, the base methods for approximating the behaviour and value estimation characteristics are structurally similar to existing supervised learning techniques and are not, by themselves, a notable contribution.\n\nOur primary contribution is not the individual equations but the complete, integrated framework designed specifically for the practical challenges of reinforcement learning. The behaviour and value estimation approximators are necessary components to complete this framework, which is collectively an important contribution because it also addresses: (1) approximating performance explanations in a tractable way, and (2) generating explanations for non-stationary policies and from off-policy data.\n\nFurthermore, the sampling-based update (Section 5) is a significant algorithmic contribution that is also directly applicable to improving Shapley value approximations in supervised learning, demonstrating originality beyond the specific context of reinforcement learning.\n\n**On Limited Evaluation in Complex Environments (Weakness 4)**\n\nOur deliberate use of smaller domains was a direct consequence of the need to compute exact Shapley values for validation. Approximation accuracy can be rigorously tested *only* in these verifiable settings. Therefore, our experiments were necessarily constrained to this regime to ensure scientific validity.\n\nThat said, while we cannot directly assess accuracy in larger domains, we agree that general scalability in more complex environments is important. Motivated by your feedback and that of others, we have run new experiments on substantially larger versions of Mastermind, designed to directly address the need for evaluation in more complex settings.\n\nWhile it is impossible to measure accuracy against an exact ground truth in these domains, our theoretical guarantees and the strong accuracy demonstrated in verifiable settings suggest that FastSVERL's models will converge towards the correct values. Therefore, our goal with these new experiments is to empirically test three key properties at scale: (1) if the models indeed converge, (2) the stability of that convergence across runs (measured by standard error), and (3) if the computational cost of convergence remains manageable.\n\nDue to space constraints, we present a representative subset of results below for the behaviour characteristic and Shapley models over 10 runs.\n\n**Table 1: Convergence and Stability of Behaviour Explanations**\n| Domain | Explanation Model | Updates to Converge (Mean ± Std. Err.) | Final Loss (Mean ± Std. Err.) |\n| :--- | :--- | :---: | :---: |\n| **Mastermind-443** | Behaviour Characteristic | $(1.13 \\pm 0.177) \\times 10^6$ | $0.0038 \\pm 0.000018$ |\n| (24 features, $\\ge 4.3 \\times 10^7$ states) | Behaviour Shapley | $(7.99 \\pm 0.902) \\times 10^5$ | $0.0013 \\pm 0.000037$ |\n| **Mastermind-453** | Behaviour Characteristic | $(1.30 \\pm 0.150) \\times 10^6$ | $0.0036 \\pm 0.000021$ |\n| (30 features, $\\ge 3.5 \\times 10^9$ states) | Behaviour Shapley | $(7.91 \\pm 0.503) \\times 10^5$ | $0.0010 \\pm 0.000032$ |\n| **Mastermind-463** | Behaviour Characteristic | $(1.25 \\pm 0.174) \\times 10^6$ | $0.0037 \\pm 0.000013$ |\n| (36 features, $\\ge 2.8 \\times 10^{11}$ states) | Behaviour Shapley | $(6.56 \\pm 0.816) \\times 10^5$ | $0.0019 \\pm 0.000044$ |\n\nThe results in Table 1 confirm that the models successfully converge in these large-scale domains. The standard error across runs demonstrates this convergence is stable and highly reliable. Most importantly, the number of training updates required for convergence remains similar even as the domain complexity grows.\n\nWe thank you again for your time and constructive feedback. We hope that our response and the new large-scale experiments will help to address your initial concerns. We will incorporate the full set of these new results, along with all the clarifications discussed, into the final version of the manuscript, and hope this new evidence will be reflected in your updated evaluation of our work."}, {"review_id": "YJjfhlB6ln", "reviewer": "Reviewer_qBRx", "summary": "The paper studies Shapley value explanations in RL and proposes a new efficient approximation method. Exactly computing SVs is known to be a computationally costly procedure in high-dimensional domains hindering practical scalability. The problem has been extensively addressed in the SL domain. However, there is no method tailored to the unique challenges the RL framework poses, such as temporal dependencies and coverage issues when only offline/off-policy data are available. The authors propose FastSVERL: a parametric approach for approximating state feature contributions (to policy action probs; expected and realized values) via SV in an efficient way.", "questions": "It was not clear to me how temporal dependencies, characteristic of RL environments, were explictly addressed by your framework.", "limitations": "Mentioned", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n- The paper provides a comprehensive summary of SVERL [1,2], allowing readers unfamiliar with this work to easily follow along. In general, the paper is very well-written and easy to read.\n- The problem addressed is nicely posed and well motivated.\n- I found the proposed approach to be reasonable, albeit most components are derived from existing work. I found the single conditioned policy idea (line 203) a nice touch.\n\n\n**Weaknesses**\n\n- The main weak point of this paper is the limited empirical evaluation (main reason for giving a low score). I would typically expect from such a paper to conduct extensive experiments in more complex (\"real-world\") environments. Furthermore, since the authors chose to use simpler simulated benchmarks, where ground truth attributions can be computed, the use of more baseline methods for comparison are needed. Such baseline methods would include SL techniques mentioned in the background section.\n- It was not clear from the write-up how some of the challenges recognized as unique to the RL method were explicitly addressed by the FastSVERL approach (see my questions below).\n- [Minor] since there are works in (MA)RL using SV for different purposes than the one considered in the paper, it would good to specify ealrier in the paper (even from appendix) the use-case of focus.", "was_revised": true, "final_justification": "The changes in the experiments although strengthening the paper constitute a change too big to be done at this point of the revewing process", "rebuttal": "Thank you for your positive and encouraging feedback, particularly on the paper's clarity. We are happy to address the weaknesses and questions raised in the review below.\n\n**On the Originality of Contributions (Strength 3)**\n\nWe appreciate that the proposed framework builds on established concepts from SVERL and the supervised learning literature. However, adapting these concepts to the practical reinforcement learning setting was a non-trivial process that required several contributions; a direct application of existing methods is insufficient for the unique challenges that reinforcement learning presents, which we discuss in more detail later.\n\nFor example, while the approximation of the performance characteristic involves the familiar process of learning a value function, its application to a conditioned policy (as described in Section 3.2) was a considered approach necessary to make the problem computationally tractable in the Shapley value setting. Similarly, to the best of our knowledge, this is the first work to address approximating Shapley values using off-policy data or for non-stationary models. These are core reinforcement learning challenges of practical interest to the community; extending existing Shapley value methods to these dynamic settings required careful consideration and was not a direct application.\n\nFurthermore, we introduce a more efficient approximation method altogether. The proposed sampling-based update (Section 5) is a significant algorithmic contribution that improves both computational efficiency and accuracy by replacing the need for a characteristic model entirely. This contribution is general and also directly applicable to improving Shapley value approximations in supervised learning, demonstrating originality beyond the specific context of reinforcement learning.\n\n**On Limited Evaluation in Complex Environments (Weakness 1)**\n\nThank you for this critical point about evaluating the proposed methods in more complex environments. We agree that the real-world applicability of a method is essential.\n\nOur deliberate methodological use of smaller domains was a direct consequence of the need to compute exact Shapley values for validation. Approximation accuracy can be rigorously tested *only* in these verifiable settings. Therefore, our experiments were necessarily constrained to this regime to ensure scientific validity. \n\nIt is only within this verifiable regime that we can demonstrate how the proposed framework’s accuracy scales. Our Hypercube experiments were designed specifically for this purpose, and they showed that the computational cost of achieving accurate explanations scales well with both the number of states and features. This graceful scaling of accuracy, in the one region where it can be principally measured, suggests the method should continue to perform well in larger domains. Furthermore, the results on Mastermind-333 in our supplement—with over 100,000 states and 15 features ($2^{15}$ subsets)—already represent a complex setting for Shapley value computation, providing further evidence of the framework's capabilities.\n\nThat said, while we cannot directly assess accuracy in larger domains, we agree that general scalability deserves greater focus. Motivated by your feedback and that of others, we have run new experiments on substantially larger versions of Mastermind. We continue to use Mastermind specifically to allow fine-grained control over complexity. These new domains, the largest with 36 features and over 280 billion states, are designed to directly address the need for evaluation in more complex settings, particularly from a Shapley approximation perspective where the summation is over $2^{36}$ feature subsets.\n\nWhile it is impossible to measure accuracy against an exact ground truth in these domains, our theoretical guarantees and the strong accuracy demonstrated in verifiable settings suggest that FastSVERL's models will converge towards the correct values. Therefore, our goal with these new experiments is to test three key properties at scale: (1) if the models indeed converge, (2) the stability of that convergence across runs (measured by standard error), and (3) if the computational cost of convergence remains manageable.\n\nDue to space constraints, we report the mean updates to convergence and final training loss over 10 runs for a representative subset of results below. It is important to clarify that this loss is not a measure of ground-truth accuracy. The characteristic model's loss is designed to converge to a non-zero value as it learns to predict an expectation over the unobserved features. In contrast, the Shapley model's loss is designed to converge to zero, but this indicates it has successfully learned to explain the values provided by the characteristic model, not necessarily the ground truth.\n\n**Table 1: Convergence and Stability of Behaviour Explanations**\n| Domain | Explanation Model | Updates to Converge (Mean ± Std. Err.) | Final Loss (Mean ± Std. Err.) |\n| :--- | :--- | :---: | :---: |\n| **Mastermind-443** | Behaviour Characteristic | $(1.13 \\pm 0.177) \\times 10^6$ | $0.0038 \\pm 0.000018$ |\n| (24 features, $\\ge 4.3 \\times 10^7$ states) | Behaviour Shapley | $(7.99 \\pm 0.902) \\times 10^5$ | $0.0013 \\pm 0.000037$ |\n| **Mastermind-453** | Behaviour Characteristic | $(1.30 \\pm 0.150) \\times 10^6$ | $0.0036 \\pm 0.000021$ |\n| (30 features, $\\ge 3.5 \\times 10^9$ states) | Behaviour Shapley | $(7.91 \\pm 0.503) \\times 10^5$ | $0.0010 \\pm 0.000032$ |\n| **Mastermind-463** | Behaviour Characteristic | $(1.25 \\pm 0.174) \\times 10^6$ | $0.0037 \\pm 0.000013$ |\n| (36 features, $\\ge 2.8 \\times 10^{11}$ states) | Behaviour Shapley | $(6.56 \\pm 0.816) \\times 10^5$ | $0.0019 \\pm 0.000044$ |\n\nThe results in Table 1 confirm that the models successfully converge in these large-scale domains. The standard error across multiple runs demonstrates that this convergence is stable and highly reliable. Most importantly, the number of training updates required for convergence remains similar even as the domain complexity grows.\n\nWe also demonstrate that the significant computational advantages of the sampling-based approximations we propose (Section 5) hold at scale. This method is a key contribution of our work, and Table 2 confirms its efficiency benefits in these large domains.\n\n**Table 2: Total Computational Cost for Behaviour Explanations**\n| Domain | Total Updates (Model-Based) (Mean ± Std. Err.) | Total Updates (Sampling-Based) (Mean ± Std. Err.) |\n| :--- | :---: | :---: |\n| **Mastermind-443** | $(1.84 \\pm 0.136) \\times 10^6$ | $(5.69 \\pm 0.498) \\times 10^5$ |\n| **Mastermind-453** | $(2.09 \\pm 0.095) \\times 10^6$ | $(7.25 \\pm 0.638) \\times 10^5$ |\n| **Mastermind-463** | $(1.91 \\pm 0.104) \\times 10^6$ | $(6.06 \\pm 0.584) \\times 10^5$ |\n\nBy eliminating the need to pre-train a characteristic model, the sampling-based approach reduces the total computational cost by over $2\\times$ across all large-scale domains tested. This suggests that the substantial efficiency gains are consistent.\n\nWe hope these clarifications and new results will help to address your concerns about empirical evaluation in more complex environments.\n\n**On the Lack of Baseline Methods (Weakness 1)**\n\nThank you for the point about comparing to the Monte Carlo sampling methods mentioned in our background section.\n\nIn a highly idealised setting (a fixed policy and resettable environment simulator) these methods could be adapted as a baseline. However, our work was motivated by the need to explain agents in more practical reinforcement learning scenarios. \n\nFor the core challenge of explaining a non-stationary policy, Monte Carlo methods are computationally prohibitive because they must recompute all explanations from scratch for every policy update. Our parametric approach was chosen specifically to handle this dynamic setting because its parameters can be continually updated as the policy evolves.\n\nFurthermore, approximating the performance characteristic via sampling, as proposed in the original SVERL framework [1], requires multi-step rollouts from arbitrary start states. This approach depends on the often unreasonable assumption of a resettable simulator, which we deliberately avoided to create a more general and practical framework.\n\n**On Addressing Temporal Dependencies (Weakness 2, Q1)**\n\nThank you, this question allows us to clarify a central aspect of FastSVERL. Temporal dependencies are addressed explicitly by the proposed approach to approximating performance explanations.\n\nThe performance characteristic function is defined as the expected discounted sum of future rewards, a value inherently dependent on the entire subsequent trajectory. The influence of observing a subset of features in a state is therefore not an instantaneous effect. When features are unknown in a given state $s$, the agent follows a modified policy at that state for the entire episode. This alters the distribution over all future trajectories, and if the agent returns to state $s$ at a later time, it continues to follow this modified behaviour, compounding the temporal impact.\n\nThe proposed approximation method—the conditioned policy and value function—was designed precisely to estimate this temporally-extended quantity in a tractable way.\n\n**On Specifying the Use-Case (Weakness 3)**\n\nThank you for this helpful suggestion. We agree that explicitly distinguishing our use from other applications of Shapley values in reinforcement learning earlier in the paper would improve clarity for the reader.\n\nWe thank you again for your time and the detailed and constructive feedback. We hope that our response and the new large-scale experiments have helped to address your initial concerns. We will incorporate the full set of these new results, along with all the clarifications discussed, into the final version of the manuscript. We hope this new evidence will be reflected in your updated evaluation of our work."}]

Author rebuttal:

Meta-review: The paper proposes a method for efficiently computing approximate Shapley values in RL. It builds on existing methods and extends these ideas to RL. The efficacy of the method is experimentally evaluated. The studied problem is interesting and important for explainable RL. While the reviewers initially expressed concerns about this submission---some suggesting that further experimentation is needed to demonstrate the utility of the approach in more complex settings---most concerns were addressed during the discussion, where the authors provided additional experimental results and clarified different aspects of their work. The authors should incorporate the new results and improve the quality and clarity of the presentation in the final version of the paper.
Decision: Accept (poster)