Feature Responsiveness Scores: Model-Agnostic Explanations for Recourse
Authors: Seung Hyun Cheon
Anneke Wernerfelt
Sorelle Friedler
Berk Ustun
Venue: ICLR 2025 Poster (2025)
Keywords: explainability
feature attribution
algorithmic recourse
regulation
Primary area: interpretability and explainable AI
TL;DR: We introduce *feature responsiveness scores*, the probability that an individual can change their model prediction by altering a feature.

Abstract: Machine learning models routinely automate decisions in applications like lending and hiring. In such settings, consumer protection rules require companies that deploy models to explain predictions to decision subjects. These rules are motivated, in part, by the belief that explanations can promote *recourse* by revealing information that individuals can use to contest or improve their outcomes. In practice, many companies comply with these rules by providing individuals with a list of the most important features for their prediction, which they identify based on feature importance scores from feature attribution methods such as SHAP or LIME. In this work, we show how these practices can undermine consumers by highlighting features that would not lead to an improved outcome and by explaining predictions that cannot be changed. We propose to address these issues by highlighting features based on their *responsiveness score*—i.e., the probability that an individual can attain a target prediction by changing a specific feature. We develop efficient methods to compute responsiveness scores for any model and any dataset. We conduct an extensive empirical study on the responsiveness of explanations in lending. Our results show that standard practices in consumer finance can backfire by presenting consumers with *reasons without recourse*, and demonstrate how our approach improves consumer protection by highlighting responsive features and identifying fixed predictions.

Published as a conference paper at ICLR 2025
FEATURE RESPONSIVENESS SCORES: MODEL-AGNOSTIC EXPLANATIONS FOR RECOURSE
Seung Hyun Cheon
Anneke Wernerfelt Haverford College
Sorelle A. Friedler Haverford College
Berk Ustun UC San Diego
UC San Diego
ABSTRACT
Machine learning models routinely automate decisions in applications like lending and hiring. such settings, consumer protection rules require companies that deploy models to explain predictions to decision subjects. These rules are moti- vated, in part, by the belief that explanations can promote recourse by revealing information that individuals can use to contest or improve their outcomes. In practice, many companies comply with these rules by providing individuals with a list of the most important features for their prediction, which they identify based on feature importance scores from feature attribution methods such as SHAP or LIME. In this work, show how these practices undermine consumers by highlighting features that would not lead to an improved outcome and by explaining predictions that cannot be changed. We propose address these issues by high- lighting features based on their responsiveness score - i.e., the probability that an individual can attain target prediction by changing specific feature. We develop efficient methods to compute responsiveness scores for any model and any dataset. We conduct an extensive empirical study on the responsiveness of explanations lending. Our results show that standard practices in consumer finance can backfire by presenting consumers with reasons without recourse, and demonstrate how our approach improves consumer protection by highlighting responsive features and identifying fixed predictions.
INTRODUCTION
Machine learning models are routinely used to automate and support decisions about people in consumer finance [30], employment [9, 47], and public services [64, 19, 27]. these domains, companies are increasingly expected and in some cases legally required - to provide explanations to individuals affected by their predictions [1. 61. 55, 20J. In the United States, for example, the adverse action provision in the Equal Credit Opportunity Act mandates that lenders provide list of "principal reasons" to consumers who are denied credit [1]. In the European Union, Article 86 of the Al Act [20] grants individuals right explanation in "high risk" domains [see Annex III of 20].
Explanations are a cornerstone of consumer protection such settings because they may reveal information that consumers adverse action notices are meant to support: anti-discrimination by revealing that a prediction was influenced by protected characteristics; rectification, by revealing that prediction was based on erroneous information; and recourse, by revealing how they could attain desired prediction in the future [54, 50]. In the European Union, the right an explanation in the GDPR meant reveal information that consumers could use to contest their decisions or request human review [33].
exercise broader rights [16]. In the United States, for example,
Many explainability mandates are developed in the absence of clear directives regarding the ap- propriate structure and construction methodology for explanations. This ambiguity exists partly because many mandates are new with policy makers debating how to enforce them or have not yet become law. In the United States, where the adverse action requirement has been in place for over half a century, companies provide consumers with teature-highlighhinn explanations that list the most important features for their prediction. They typically generate these explanations using feature attribution methods such as SHAP [40] and LIME [48], scoring features in terms of their relative contribution the prediction and including the top-scoring features in letter consumers. This approach has been recognized as a principled attempt to identify "principal reasons" and has gained wide-spread adoption due to its simplicity [see e.g., recommendations in 24].

---

Published as a conference paper at ICLR 2025
Feelure-Highighting Explanations with SHAP
Feeture-Highighinng Explanetions with Responsivenees
Principle Reasona
Priccple Reasons
1. Paaeoryn
MuliiiiCrnaiiiinn Merriyyccomm
Nurbe Depa stentu"
Crecit.tiiationn
Aunterr@@ppendeem
Age"
NumburODDppenn
DetiForro
VAB seCredtiies
0.4 04 O8
D7 0.1 08
Figure 1: Fealure-highlighting explanations for person denied credit by XGBoost model the givemecredit dataset Section 4), We construct each explanation by highlighting up to four features with the largest SHAP scores (Ieft) and responsiveness scores (right). As shown, an explanation built with SHAP highlights four features that are immutable** (e.g., Age) or unresponsive no intervention lcads to target prediction (e.g., CreditUt (in)) In contrast, explanation built with responsiveness scores returns the only 2 features that can be changed attain desired prediction: MonthlyTes conthlyInoomo and mulltipeerdddite nea. score for ipleredd nea is which means any intervention would lead a target prediction.
In this work, we analyze how explanations can effectively achieve one of their goals: helping consumers attain recourse. Our work is motivated by the fact that explainability mandates are designed to achieve multiple goals and yet overlook implementation details. To this end, we assess the efficacy of feature attribution methods in supporting recourse, and develop targeted approach for this objective. Our main contributions include:
1. We identify an inherent limitation of feature attribution methods in consumer-facing applications reasons without recourse where we present features in explanations, yet modifying them yields no path to the desired outcome.
2. We introduce new approach to explain individual predictions by measuring feature responsiveness the probability of attaining recourse through a randomly chosen intervention on its features.
3. We conduct an empirical study on feature-highlighting explanations in consumer finance, finding that standard methods can harm consumers by highlighting immutable and unresponsive features. Our framework improves upon feature attribution methods, promoting recourse and transparency by highlighting features that help attain target predictions and flagging predictions that are difficult or impossible to change. Our analysis also reveals the need for additional information in explanations to effectively facilitate recourse.
We include Python library to compute feature responsiveness scores available on GitHub.
Related Work Our work is broadly related to stream meth- ods to explain individual predictions [see, e.g., 48, 40, 41, 36]. We identify these methods can inflict harm in consumer- facing ap- plications by providing individuals with reasons without recourse. We view reasons without recourse as a structural limitation that affects how we operationalize explainability mandates, akin limitations of explainability that arise due to the multiplicity of predictions [42, 60, 8], the indeterminacy of explanations [11, 39], and the potential for fairwashing [4, 52, 28].
32
Our goal broadly inspired by work in algorithmic recourse, as we seek to highlight how individuals can change their predic- tions [57, 34, 58). Existing methods in this area are designed to return an action that an individual could perform to attain a target prediction. In contrast, our method estimates the proportion of actions each feature that lead to a target prediction (see Fig. 2). We construct these estimates by sampling or enumerating a set of reachable points [38] and can be adapted to address practi- cal challenges related to causality [35, 14, 26] and distributional robustness [45, 46, 56).
Figure 2: Standard methods for re- course provision return the closest action that would attain the target prediction "M Our method esti- mates the proportion of actions on each feature that attain the target pre- diction. Here, μl 3 and μr2 because x could attain the target pre- diction through 3/4 possible actions and 1/4 possible actions
2

---

Published as a conference paper at ICLR 2025
2 PROBLEM STATEMENT
Feature Values
Label Counts
Best Predictions
In what follows, we discuss how companies comply with rules and regulations that require them to explain the pre- dictions of a model to individuals. We consider a classi-
age 60
sevinge 2 60k
Ro n1
0
10
0
1
30
fication task where a company uses model f X + y to predict a label E y from a set of d features x [21, æ2,...., E X S Rd We restrict our attention to tasks where each instance represents a person, and where their features encode characteristics that are semantically meaningful for the task at hand. In such tasks, we can safely assume that features are bounded."
Stylized classification task where the most accurate model assigns fixed predic- tions due to the immutable feature age 2 60. We train model predict y = repayment from binary features (I1, #22 (age 2 60, savings 2 60R) on dataset with no negative labels and positive labels. Here, individ- uals with age ture values, but cannot change their predic- tion as the model outputs / (111, = for (21, x2) {(1, 0), (1,
Table 1:
We define the scope of explainability mandates in terms of a target prediction gjanget E V. We assume that the target prediction @ represents desirable outcome, e.g.. f(x) = junget = 1 a person with features æ will repay their loan, and that any other prediction represents an adverse outcome. Under these conventions, companies must provide explanations these practices would support recourse [57] if they include information that each person could use to attain a target prediction - i.e., to update their features a point x' such that f(x' garget
can change their fea-
all individuals with features such that f(x) gaaset Informally,
Feature-Highlighhting Explanations Companies comply with explainability mandates by con- structing a
explanation i.e., a list that contains the most important features that contribute to prediction [5]. In practice, companies select which features to highlight based on feature importance scores that they obtain from a feature attribution method [24, 22]:
Definition 1. Given a model f X -> y and point æ E X, feature astribution method is function & : X + Rd that returns a vector of feature importance scores Φ(x: f) == ( f)..... Φ2(2; f)] where و5 (æ; f) reflects the relative contribution feature j towards the prediction f(x). We write (xx) instead of (xx;; f) when is clear from context.
We can use the function Φ X + Rd to represent major classes of local explainability methods:
Local Surrogate Methods [48, 66, 65, 17, 53], which explain the prediction of model f at point x by fitting an surrogate model to approximate the decision boundary of f near æ. Given the surrogate model, we can use its parameters to determine the importance of each feature Φ, (x).
Shapley Value Methods [see c.g., 40, 31, 25], which cast the features of model f as players in cooperative game, and estimate Φ (ar) as the marginal contribution of feature j towards the prediction fxx under axioms of social choice [51].
Scores from these models indicate relative importance due to the following properties:
Relevance: A feature with an attribution score (z) = 0 can be changed arbitrarily without changing the prediction for æ [see e.g., the "missingness" axiom in 40].
Strength: Given two features j, k E [d] such that (x)()) > (x)),, feature j has a stronger
contribution to the prediction than feature [see e.g.. 44].
Given top-scoring features, companies can automatically convert them into natural language explana- tions for decision subjects [e.g., a reason code 21, 13].
Reasons without Recourse Feature attribution methods can highlight features that genuinely reflect the "principal reasons" or "main factors" " for each prediction. In practice, however, these methods may fail to highlight features that lead desired outcome:
Inaiiiiyy to Characterize Counterfactual Behavior: Feature attribution methods often assign high importance scores to features that are not indicative of the model's local behavior. Bilodeau et al.
In practice, many features will be bounded by definition e.g. a binary feature such as recent_payymee E {0, 1]. In other cases, we can set loose bounds that apply to all decision subjects c.g., age E [0, 120) nocome e [0, 10%).
3

---

Published as a conference paper at ICLR 2025
Requirement
Example
Features
Actionabiiity Construint
Immutability
ace cannot change
X, agu
1/ 4--0
Monooeiciity
recent paynent only incresse
recent paysent
by 2.20
Imeorality
Tate. Tate_payrert payrect a must posiiiee integor 12
55 eite-peemente
vy E 2 10- 10,12,11--11
12-13)
Foending Validity
preserve one-hot enooling of categorical
1 Ineoeineewwn
vy +., (o.1) (R.4.)) Exe(ht))) "/ M/ my
3 [oousingeeend |pouse
fcoture housing (onn, othex
Implication
Logical
if Than_pav-ngs, paw..npaaccooun.tttc.. pcccunt - 1808
else saviaga -0
then eastings balance 2
To cavingg, balence
bes_aevings_
Ue 2x S (0
t. (0.1)
E0, 10
Cunsal
if years_of__ waw.cceeut___a. histosy incresses
Implication
then açe will increase commenuuratly
X2 age
4. [0.
Table 2: Examples of actionability constraints semantically meaningful features for lending task. Each constraint can be expressed in natural language and embedded into optimization problem using standard techniques mathematical programming [see, c.g., 62]. See Appendix for more examples.
[7] demonstrate that methods like SHAP suffer from under-specifiicttoon - i.e., a feature with a given level of responsiveness can receive different attribution scores and are sensitive to baseline values, which may not be relevant in describing model behavior locally.
Ignorance of Actionability: Feature attribution methods do not account for how individuals can change their features. This can lead them to highlight features that are immutable or even highlight features when individuals assigned fixed predictions f(x²) * grarget y x' (see Table 1).
Highlighting important but unresponsive features undermines the value of explainability mandates. For example, feature attribution methods might highlight immutable features, even when recourse is available through other features. Alternatively, they may highlight features that seem actionable but do not lead to the desired prediction, causing individuals to focus on ineffective changes.
MEASURING FEATURE RESPONSIVENESS
Our goal is measure the responsiveness of the model with respect independent changes in each feature. First, we characterize how each feature can be changed:
Characterizing Actionability Let be the feature vector representing a person with the capability of changing their associated data to new point x':
x' + v + 2(v)
(1)
where v E Rd is an intervention with one non-zero element Bj. and z(v) is the downstream effect of v that captures how other features change as a result of the intervention v. Given a model that includes the features like srs___ccout__ttiteee
and age, intervention that increases yrs account history should lead to commensurate increase in age a downstream effect.
Each feature is inherently constrained by physical limits (i.c., bounds), type (c.g., one-hot encoding), or nature (c.g., monotonicity of ordinal encoding of edeeation_ievel). As shown in Table 2, we can elicit these constraints from human experts in natural language, and convert them into equations that we can embed into optimization problems [e.g., to search for recourse actions 57,38]. J. In fact, we can extend beyond inherent constraints and encode non-trivial constraints like the causal relationship between yrs__ccsoooo s___cccut_____c_e_eeee_e________ woroouuuthhhttttrooooeeeeee histtrry and age mentioned earlier. We refer to the set of such constraints, both inherent and non-trivial, as actionability constraints.
We describe the set of points we can reach by intervening on a feature j within these constraints:
Definition 2. Given a point æ, we define the set of interventions on feature j [d]:
V=()) (w ddd [U, + 0, Uj is feasible under the actionability constraints, 1x 0 for k k j) We also define the action set, which describes the set of points we can reach by intervening on j:
Ax(x) == (a = v + z(v) v E Vz())}
We are now ready introduce our main technical contribution the responsiveness score.
4

---

Published as a conference paper at ICLR 2025
Reachable Points
Prediction
Responsiveness Score
n.1oans
quarantor
ag2
repey
.1
22
25
f(z)
(1( [[(=))
Σ
11((')) =
21
21
w'e (())
21
24
Ma(m)) Ra(a))
=1(=1)=1==1
0
24
Ra(a) Ms(a)
2
21
Ra(a))
0
24
Figure 3: Responsiveness scores for a stylized task with three features (n__laan,, guarancor, age). We show all points that can be reached for person with features (3. 0, 24). Here, the reachable set (z())) (r) for each of the feature contains the set points that because age is immutable. Given these sets, we compute the responsiveness scores each feature by querying its predictions over points in their respective reachable set Hy (2:).
reached through single feature interventions, and Ra (a) Ø
Definition 3 (Responsiveness Score (1z())))). Given model f X' + that assigns an adverse prediction to a point æ E X, we define the responsiveness score of feature j as:
Az())) We write (z) instead of (2 A, when f and (x) are clear from context.
My (z f. (z)) Pr(f(x= a) = guurett
Responsiveness scores reflect the proportion of single-feature actions feature j that result in graret A score of P4 (x) = 0 indicates that changing feature j cannot achieve grarger while μı(()) 1 means any intervention on it will. These values depend on actionability constraints, which, in the simplest case, encode indisputable constraints related to feature encoding or physical limits, meaning that (z) represents an upper bound on the true responsiveness of feature j.
(x) can be interpreted as the probability of person achieving ga after random intervention on feature j. Hence, highhrihiigi explanations using responsiveness scores is optimal because these explanations do not specify the necessary degree of change; highlighting the most responsive feature maximizes the likelihood of recourse. We only include features that, when changed independently, lead recourse, enabling consumers to choose any highlighted feature.
This approach offers stronger protections by: (1) providing explanations only to individuals with recourse, and (2) detecting potentially misleading uninformative cature-highlighting explanations.
Remark 1. Given model f - V,,,, let (x)... ( (a(x) denote the feature responsiveness scores for an individual x E X. If (11(x)) 0 for all E [d], then either: (a) f assigns a fixed prediction to (b) J can only provide recourse to through a joint intervention on two more features.
According Remark 1, (z) for all j E [d] in two scenarios. We can triage these cases to mitigate harm. For fixed predictions (case (a)). we can withhold explanations and notify developers or regulators. For predictions requiring joint interventions (case (b)), we can include warning against assuming feature independence.
3.1 COMPUTING SCORES WITH REACHABLE SETS
The key challenge in computing responsiveness scores is include actionability constraints in the calculation. As such, we present an alternative representation of actionability, a reachable set: Definition 4. Given point æ, the reachable set for feature j contains all feature vectors that can be attained through actions on feature j: Rx (x: == (a + a E (x))
Given a reachable set (x), we can calculate the responsiveness score for any model by querying its predictions over reachable points (see Fig. 3) as
(= (=)) == Eg' R, (w) [1 [[(')) = (
where 1 is the indicator function. In such cases, we only need to compute the reachable sets once and use it to measure feature responsiveness for any model. This enables to take responsiveness into account in various stages of model development (i.e.. model selection). Here, we outline two methods construct reachable sets and include additional details in Appendix A.I.
5

---

Published as a conference paper at ICLR 2025
Sampling We present procedure to sample reachable points in Algorithm 1. Given a point and the action set Ax) this procedure turns a uniform sample of N reachable points by rejection sampling. In Line 2, it calls the SampleAction(#: Ay) routine to propose can- didate action vector that obeys separable ac- tionability constraints such as bounds and in- tegrality constraints. In Line 3, calls the CheckFeeaibilityyy â Aj) routine to check if à obeys joint actionability constraints by solving a mixed-integer program. The procedure terminates after it has found N feasible actions à E (x(x) each of which can define reachable point as x+
Algorithm 1 Sample Reachable Points
Require: X
point
Require: Aj (x)
action set for feature sample size (see Appendix A.2)
Require: N E
R, f-2
1: repeat
2:
á 4- --ampleAction(z; AA(x))
3:
if ChhckFeasibiltyylz: a, Az(x)) then
4:
R2 - Rg U fæ a
5:
end if
until |Ryl = N
Output R2
N reachoble points actions
Algorithm 2 Enumerate Reachable Points
Given (z) we can recover an unbiased esti- mate for the responsiveness score for feature for
Require: E X
point
Require: (x)
action for feature (discrete)
a model f as i. (z) :- & Emre [[f(²))=
Require: (x)
finite set interventions for feature
glarel). We can set the sample size N to control the reliability with which we flag predictions that satisfy the conditions in Remark which warrant more detailed explanations (see Appendix A.2).
Rg
A Aa()))
1:
for each v V5(x) do
2:
repeat
a'
<- Find1DAction(ar, v, Ay)
4:
R2 - U2 (2
Enumeration We present alternative proce-
5:
AG < A, {a"}
dure that can be used to enumerate a complete
6:
until Find1DAction(a, v, Aj) is infeasible
reachable set in Algorithm 2 to enumerate Ry (a:). This procedure can only be applied to construct reachable set for features that are discrete and whose downstream features are also discrete. The algorithm enumerates reachable points for a feature j by repeatedly solving the discrete optimization problem:
7:
end for
Output R
all reachable points via actions in
Find1DAction(z, v, A₂) argmin ||a]|a s.t. 0j Vij (EA;(x)
for each (x) (Line 1). (x)) is superset of (z( (i.e.. Vz(z) C Vz(),, where it con- tains possibly infeasible interventions. Generally, we can deduce Vz(x) from feature bounds and monotonicity constraints. We formulate Find1DAction(zz v, Ay) as a mixed- -ixteer program that we update each iteration with "no good constraint" remove previous solutions (Line 5). We store each of the enumerated actions into R2 (x), which can be used to calculate exact responsiveness scores. The procedure adapts a method used enumerate complete reachable set for all features in Kothari et al. [38], and is more tractable as we only enumerate single-feature interventions and their downstream effects.
Extensions One of the benefits of using reachable sets to compute responsiveness scores is that we easily customize scores to meet additional requirements (see e.g., Section 5). One such requirement is monotonicity - i.e., whether a person would be guaranteed target prediction after increasing (or decreasing) a feature beyond threshold value. In general, we can account for such properties through simple operations such as filtering weighing reachable points. We can also apply these operations to construct responsiveness scores that address practical challenges in settings where we are given additional inputs:
Individual Preferences: Given cost function that captures the difficulty of actions in each direction, we can highlight features that are easier to change (i.c., least costly features) using a cost-weighed score: (150m (x; cost)= EaeA, cost(a; x) 1if(x + a) ].arget].
Distributional Robustness: Given general reachable set R((r)) that contains all points that reach, we can highlight features can be changed to attain a target prediction regardless of how other features change by computing robust score: p.) y.oobus (x) minsea., Pr(f(x + + 8) glangel
a E A, (z)), Here, A == {8 E Rd 5, = 0. || < E} is the set of perturbations on features other than j given budget > 0 [see e.g., 45).

---

Published as a conference paper at ICLR 2025
4 EXPERIMENTS
We present empirical study the responsiveness of explanations. Our goals are to demonstrate the limitations of existing feature attribution methods, and show how our approach can support recourse and flag fixed predictions. We include additional details and results in Appendix B, and code to reproduce our results on GitHub.
Setup We work with three publicly available consumer finance classification datasets. Here, each instance represents a consumer and the label indicates if they will repay a loan. We consider discrete version of each dataset in which recourse. Given these features, we define inherent actionability constraints that reflect indisputable requirements that apply to all individuals (e.g., that disallow changes immutable attributes, preserve feature encodings, and adhere to deterministic causal effects).
compute exact responsiveness scores and certify existence of
We split each dataset into training sample (80%; to train models and tune parameters) and test sample (20%; to evaluate out-of-sample performance). We fit models using (I) logistic regression (LR), (2) XGBoost (XGB), and (3) random forests forests (RF). For each model, we construct feature- highlighting explanations for each person who is denied credit that highlight up to four features, which reflects the recommended number of reasons to include in adverse action notice by the U.S. Consumer Finance Protection Bureau [see 2]. We include the top-4 scoring features from the following methods:
Dataset
Metrics
LR
RF
XGB
zeloe
Denial
56.19
58.3%
" =, 5,842
5% No Recoume
22.2%
$1.39
53. 51.19
d-
Receurs:
41.0%
51.7%
25.35
FICO [23)
56 n-D Recoume
36.8%
37.0%
21.6%
S Decied
22.9%
17.5%
22.0%
1,000
d
16 1-D Recoume
74.2%
48.0%
68.2%
Dan Griff [15]
56 n-D Recsume
18.9%
23.4%
20.09
" 120, 268
S No Ressurse
15.698
0.24%
11.39
d Kaggle [32]
% I-D Recoume
72.4%%
93.2%
76.0%
n-D Resoume
12.0%
6.0%
12.55
Table 3: Fraction of individuals who would receive an explanation as a result of adverse prediction (% Denied) for each dataset and model. We characterize the poten- tial to highlight features that lead to recourse by reporting the following metrics:% Fixed, % of denied individuals who with fixed pre- diction (in red) - i.e., who have no recourse under any explanation; % I-D, % of denied individuals who can achieve recourse by in- tervening on single feature soce-ii.ce i.e., who could have recourse from highiggngg ex- planations; and n-D, the fraction of denied individuals who can only achieve recourse by changing 2 or more features;
Feature Responsiveness (RESP): We compute respon- sive scores by enumerating reachable sets using Algo- rithm 2 with respect inherent actionability constraints in Appendix B.
Standard Feature Attribution: We consider model- agnostic methods that widely used in the lending industry [24]: SHAP [40]; and LIME [48].
Actionable Feature Attribution: We consider action- aware variants of feature attribution methods, SHAP-AW and LIME-AW which aim to highlight more responsive features by setting dby < O for features that are not
actionable (i.e. can't 't be changed).
On the Limits of Feature Highlighting Explanations Our results in Table 3 highlight how widespread practices to comply with explainability mandates can help consumers achieve recourse. For example, given an LR model on the heloc dataset, a lender would provide highhig-hg explanations to 56.1 5% of individuals who are denied a loan. Among them: 41.0% can change a single feature to attain desired prediction: 36.8% can only achieve recourse by changing 2 or more features simultancously; and 22.2% are assigned fixed prediction. Although the magnitude of these segments can vary considerably across datasets and model classes, there is no case where every person who is denied credit can change their prediction with single-feature intervention. Some will require joint interventions. Others will have no path to recourse.
These results reflect best-case estimates of providing recourse with neature-highlightnn. explanations. Specifically, the 41.0% of individuals who could achieve recourse by single feature intervention could only do so if we used an ideal method that assigns the highest scores to responsive features, and did not face additional actionability constraints. They also characterize where feature attribution methods may fail - i.e., 22.2% of denied individuals would be misled by any explanation as their prediction is fixed. These results underscore the need to impose stricter guidelines on compliance and devise an alternative notification framework for individuals fixed predictions.
On Explanations with Feature Attribution Scores Our results show how standard methods for feature attribution can highlight features that are uninformative or misleading. Given the LR model
7

---

Published as a conference paper at ICLR 2025
LR
XGB
All Feutures
Actionalle
All Festures
Actionable Feutrres
Datasct
Meuics
LIME
LIME-AW
8HAP-AW
BHAP
LINE.AW
SHAP-AW
selce
8. Presented with Explantions
100,0%
100.0%
100.09
100,75
43.0'5
100.0%
100.09
100.0%
100.05
25.35
5,842
All Feotures Uxsesponsive
92.7%
76.85
70.37
0.0%
50.2%
52.31
80.016
79,05
0.0%
43 features
Lesst Fasure Roquiaive
7,3%
22.7%
23.2%
29.75
100.0%
6.8%
17.7%
20.05
20.4%
100.05
da 31 mutaable
All Features Respussive
0.0%%
0.0%
0.0%
0.25
100.01
0.0%
0.0%5
0.0%
0.0%
10000%
(23]
Restuns Hightighed
4,0
4.0
4.0
quarann
% Presented with Explanations
100.0%
100.0%
100.0%
100,0%%
74,2%
100.0%
100.0%
100.0%
100.0%
68.2%
1. (10
All Festures Utsesponiive
91.7%
100,0%
59.4%
65.155
0.015
100.0%
99.155
70.51
67.3%
0.0R
35 features
Least Peauuee Responsive
9,3%
0.0%
34.9%
100,0%5
0.9%
29.51
52.7%
100.0%
d4 - 9 mutuble
All Fantures Resputaive
0.0%5
0.0%
0.05
0.315
100.0%
0.0%
0.0%5
0.0%
0.0P
100.0
Dua Graff
n Features Highlighhed
4,0
4,0
4,0
4.0
18
civenaceadis
Presented Explanations
100,0%
100.0%
100.0%
100,0%
72.4%
100.0%
100.0%
100.0%
100.0%
76.0%
120,28 208
6 All Feeatrres Unsesponeive
65.5%
46.8%
56.0%
13.11
0.0%
41.8%
43.3%
31.616
30,0%
0.0%
d- features
Least Feuture Respossive
66.9%
100,015
58.2%
56.7%
68.45
100.0%
da- 13 mutable
% All l'eotrres Responsive
0.0%
0.0%%
0.0%
22.8%
100.01%
0.0%8
0.0%
4.2%
13.2%
100.0%
Kagale [32]
A Features Highlighedd
4.3
4.9
4.0
4.0
2.4
4.0
4.0
4.0
4.0
2.6
Table 4: Responsiveness of Peature-highlighee datasets. We defer results for RF to Appendix C.1 for clarity. For each model, we generate explanation explanations that highlight up to 4 top-scoring features from a given method. We report the proportion of individuals receiving an explanation (% Presented with Explanations) and the mean numher of features in cach explanation (# Features Highlighted). We also show the proportion of instances where all features are unresponsive (% All Features Unresponsive) highlighting positive values; at least one feature is responsive (% At Least Feature Responsive), or all features are responsive (% All Features Responsive) highlighting the best value.
explanations for LR and XGB models for all methods and
on the hel oc dataset, we find that 92.7% and 77.3% of explanations from LIME and SHAP fail to include single responsive feature. This stems from two issues:
Low Scores for Responsive Features: Under the LR model on the heloc dataset, for example, 41.0% of denied individuals can achieve recourse by altering a single feature. However, LIME and SHAP do not assign higher scores to these features (see plots in Appendix C.2).
Fixed Predictions: Under the LR model on the heloc dataset, 22.2% of denied individu- als are assigned a fixed prediction. These are instances where providing any explanation can be harmful. However, LIME, SHAP and their variants still highlight important features. For example, SHAP explanation for fixed prediction includes Avgyears nF i and NetFracti enRevol wenvooviigite niguurdeen, which gives the impression that the individual could change them to attain recourse.
Provided that feature attribution methods like LIME and SHAP can output unresponsive features by overlooking actionability constraints, we study the potential to improve responsiveness using their action-aware variants SHAP-AW and LIME-AW. We construct explanations using only actionable features, following common belief that we can account for actionability by post-processing [e.g.. 43). Our results in Table 4 show that action-aware variants can highlight more responsive features. For the LR model in hel oc, 29.7% of SHAP-AW explanations contained least one responsive feature fc.f., 22.7% for SHAP), meaning that consumers are more likely to be informed of a feature that can lead to recourse. One shortcoming of this approach that we are required to filter features based on their actionability at a global level (c.g., whether individuals can intervene on this feature in principle). This is the most conservative approach in accounting for actionability, which does not require additional assumptions about constraints at an individual-level.
On Explanations with Responsiveness Scores Our results in Table show how our approach can be used to comply with regulatory requirements and address the two limitations that affect feature attribution methods. When we construct fegture-highlighhing explanations using responsiveness scores, present individuals with explanations that only contain responsive features, achicving 100% on the % All Features Responsive metric across datasets and models. In contrast, only 0.2% of SHAP-AW explanations of the LR model in heloc were fully responsive. For the remaining 99.8%, we are guaranteed that the explanation contains at least one unresponsive feature, potentially leading individuals to intervene on them and still fail to attain recourse.
Furthermore, explanations based on responsiveness scores contain the most responsive features that can be changed independently obtain recourse. In effect, we only provide explanations to
8

---

Published as a conference paper at ICLR 2025
individuals who achieve recourse with single-feature action. This may result in explanations that highlight fewer features on average for example, individuals receiving explanations from the LR model heloc receive 2.3 out of 4 reasons on average. This behavior can mitigate harm avoid presenting explanations to individuals with fixed predictions or those who require joint actions to change their outcomes.
DISCUSSION
Our results Section show that feature attribution methods fail to highlight responsive features. In principle, explanations should highlight features like income, that are responsive, that will lead to desired prediction when altered past (or below) a certain threshold, and that most people know they should increase (or decrease). In practice, this may not be the case; features may have to be changed in ways that are not monotonic (e.g., increase savings by at least $1,00 $1,000 but not more than 500 or not intuitive (i.e., decrease income be approved for loan). what follows, we evaluate how often explanations highlight features that meet these additional conditions. Our goal is to show how such analyses can support the kind of information we must include an explanation.
Setup We work with version of the givenecredit t dataset where we keep continuous features as-is rather than binarize them (Appendix D). We use the same setup as in Section 4 to train an XGB model and build feature-highlighting explanations using SHAP and LIME. We evaluate the responsiveness of features using reachable sets built using Algorithm I: for each individual with an adverse prediction under the XGB model (n = 23, 459), we constructed sampled reachable set for each actionable feature. We chose the sample size N 500 to ensure that the 100(1 a3% confidence interval in Appendix A.2 had an upper bound < 0.01 when (x) = 0 with a = 0.01.
On the Pitfalls of Missing Information Our results reveal how individuals who are shown a list of responsive features can fail to attain desired prediction due to missing information.
Missing Information on Degree of Change: Explanations can often highlight features without including information on how much to change them. As an example, 32.3% of SHAP explanations highlight Creditut 1zt ion, the proportion of available credit in use. In this case, the feature is responsive in 10.7% explanations. However, 9% of individuals would only be approved they change this feature by specific amounts. As example, with Credittti wwttttilizaatii (0, 0.504) U (0.651,
can point 0.99 who would only be approved by reducing this value to 2y e -i..,, they would be approved if they were to reduce to 0.66. but not to 0.55. This is an instance where responsiveness isn't monotonic Credi tut i 1 i on is responsive in disjoint intervals.
individual
Missing Information on Direction of Change: Explanations can also highlight features without stating whether they must increased or decreased. This can backfire when features have to be changed in counterintuitive ways. As an example, we consider highlighted in 48.1% % of SHAP explanations. In this case, the feature is responsive in 20.6% of them. However, 10.7% of cases would only lead to approval if individuals were to decrease conthlyInceme, For example, we had an applicant with Monthly monnthlynncomm Income = 4100, that could get approved for loan but only if they were to decrease it by 581 or more.
Incomme,, which is
We show how these pitfalls affect explanations from SHAP and LIME in Table 5. Our results show that less than 7% of explanations highlight at least one feature that is responsive, monotonic and intuitive. However, 0% of people are given feature- highulihhhghiggtting explanations where all features meet these conditions. Hence, at least one feature in each explanation violate these conditions. These results broadly highlight that there is regime where an individual can reliably achieve recourse using standard feature- hhuuurr-iighlihhting explanation, and underscore the need to include additional information when explanations are meant to support recourse.
Discussion One of the consequences of building explanations using methods like SHAP and LIME is that we cannot reliably tell when these conditions are met. We could impose some of these conditions by enforcing conditions linear classifier like LR rather than XGB. Alternatively, we could use custom responsiveness scores to highlight features that meet all these conditions (i.e., responsive, monotonic, and intuitive). RESP
how train the model - i.e., we could ensure monotonicity by using

---

Published as a conference paper at ICLR 2025
column in Table 5 show how effective custom responsiveness scores can be as standalone solution. can highlight instances where there are no features that meet the these conditions (i.e., the "O" bar) which point could highlight features that achieve weaker forms of responsiveness alongside additional information to help consumers. For example, we find features that are responsive but non-monotonie, then we may add additional information the thresholds. This would provide an alternative approach to ensure these conditions in way that would not interfere with model development.
LIME
SHAP
RESP
CONCLUDING REMARKS
Explanations are essential for con- sumer safeguards domains like lending and hiring, providing critical information that enables consumers to exercise their rights [16]. We demon- strate how explanations can backfire by highlighting unresponsive features and by explaining fixed predictions. Our work addresses these issues by constructing explanations based on feature responsiveness- - the probabil- ity that feature can be changed to attain a desired prediction. Our model- agnostic framework can readily be used in place of methods like SHAP and LIME, which are eurrently being used to comply with regulations. In doing so, we can strengthen consumer protection by highlighting responsive features and flagging instances where is no recourse.
Responsive
40% 0%
1/b.
41%
.1%
2%
21%
20%
4
Responsive Monotonic
2% 48
45%
41%
Responsive
67%
Monotonic
Intuitive
K3
45%
0%
X15 41%
Table 5: Distribution of the number of features in feature- highlighting explanations from LIME. SHAP and RESP for XGB model on givemecredit that are: responsive (can change indepen- dently to obtain recourse), monotonic (consistently responsive past threshold) and intuitive (responsive in direction that aligns with common expectations). We quantify to what extent explanations highlight features that can reliably provide recourse without ad- ditional information. We evaluate these characteristics using the feature's 's reachable set (x(x)) (see Appendix D).
Our results also underscore key lessons for how design, enforce, and comply with explainability mandates. Specifically, we show that these mandates should establish clearer guidelines pertaining to the content of explanations and how they are constructed so that explanations serve their outlined objectives i.e., require companies use tailored solutions that support recourse. This is especially important as many the other desiderata for explanations have standalone solutions e.g., anti- discrimination via auditing [49, 6] and less discriminatory models [29, 63].
Limitations The main limitations of our work stem from assumptions about actionability. this work, have used conservative of assumptions that reflect indisputable constraints. In effect, responsiveness scores under these constraints can flag individuals with fixed predictions. However, may not guarantee recourse individuals may face additional constraints that are not aware of. In practice, we can mitigate these issues by highlighting additional features, reporting features that exceed minimal level of responsiveness, or eliciting constraints from decision subjects [see e.g.. 18, 12, 37]. We have also restricted our attention to settings where interventions lead to deterministic causal effects on downstream features. In principle, our machinery can handle settings where interventions induce probabilistic effects on downstream features [35, 14, 59]: given an individual probabilistic graphical model, we can compute responsiveness score reflecting the expected recourse rate. Nevertheless, these causal assumptions are difficult to validate an individual level and thus require an approach to measure responsiveness in a way that is robust to misspecification.
Broader Applications Our framework for measuring feature responsiveness extends to various contexts beyond consumer finance. More broadly, responsiveness scores can evaluate model hehavior inputs interactions. For instance, in content moderation, these scores can
in relation assess a model's robustness against strategic manipulations. In criminal justice applications, they can determine whether a model responds inappropriately changes in protected attributes (e.g., sex) or appropriately to relevant factors (e.g., type of criminal offense).
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work is supported by the National Science Foundation under award IIS-2313105, and the NIH
Bridge2AI Center Grant U54HG012510.
REFERENCES
[1] 12 cfr part 1002 equal credit opportunity act (regulation b). https: hwtp://ww...t///o...o.oo http///wcconssmeefaann....ee gww/rules-policeeee po/.cc//ssgpllaaioon//0////////2
Accessed: 2024-07-16.
[2] Comment for 1002.9 notifications. httpt://////tt http://www..cooouuueef.i.aaaeeee ott.ccaiis...teess/.......eeeeee wtt.////////////////p9//
gww/rules-poll..
Accessed: 2024-07-16.
[3] Alan Agresti and Brent A Coull. Approximate better than "exact" for interval estimation of binomial proportions. The American Statisticiam, 1992):19--26,, 1998.
[4] UIrich Alvodji, Hiromi Arai, Olivier Fortineau, Sébastien Gambs, Satoshi Hara, and Alain Tapp, Fairwash- ing: the risk of rationalization. In International Conference on Machine Learning, pp. 161-170. PMLR, 2019.
[5] Solon Barocas, Andrew D Selbst, and Manish Raghavan. The hidden assumptions behind counterfactual explanations and principal reasons. In Proceedingy of the 2020 Conference Fairness, Accountability, and Transparency, pp. 80-89, 2020.
[6] Rachel Bellamy, Kuntal Dey, Michael Hind, Samuel C Hoffman, Stephanie Houde, Kalapriya Kannan, Pranay Lohia, Jacquelyn Martino, Sameep Mehta, Aleksandra Mojsilovié, et al. Ai fairness 360: An extensible toolkit for detecting and mitigating algorithmic bias. IBM Journal of Research and Development, 63(4/5):4-1, 2019.
[7] Blair Bilodcau, Natasha Jaques, Pang Wei Koh, and Been Kim. Impossibility theorems for feature attribution. Proceedingy of the National Academy of Sciences, 121())ee334406122) 20, 2024.
[8] Emily Black, Manish Raghavan, and Solon Barocas. Model multiplicity: Opportunities, concers,, and solutions. In Proceedings of 2022 ACM Conference on Fairness, Accountability, and Transpurency, pp- 850-863, 2022.
[9] Miranda Bogen and Aaron Ricke. Help wanted: An examination of hiring algorithms, equity, and bias. Uptunn, December, 7, 2018.
[10] Lawrence D Brown, T Tony Cai, and Anirhan DasGupta. Interval cstimation for a binomial proportion. Statistical science, 16(2):001-133, 2001.
[11] Marc-Etienne Brunet, Ashton Anderson, and Richard Zemel. Implications of model indeterminacy for explanations of automated decisions. Advances in Neural Information Processing Systems, 35:78 15:7810-7823, 2022.
[12] Giovanni Toni. Paolo Viappiani. Stefano Teso, Bruno Lepri. and Andrea Passerini. Personalized algorithmic recourse with preference elicitation. arXiv preprint arXiv:2205. 2022.
[13] Louis DeNicola. What are credit score reason codes? myFico, February 2022. URL https://wwx my fico. wttt///eedt-eeeeeeeeeeee i www.///ll///e cewooo/rrassoccooeeooo
[14] Ricardo Domingucz-Olmedd Amir H Karimi, and Berhaard Schölkopf. On the adversarial robustness of causal algorithmic recourse. In International Conference Machine Learning, pp. 5324-5342. PMLR, 2022.
[15] Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL htl.htttp//archiveee uei. . edu/ml.
[16] Lilian Edwards Michael Veale. Slave to algorithm: Why a right to an explanation is probably not the remedy you are looking for. Duke L. & Tech. Rev., 16:18, 2017.
[17] Radwa ElShawi, Youssef Sherif, Mouaz AI-Mallah, and Sherif Sakr. Ilime: local and global interpretable model-agnostic explainer of black-bo box decision. In Advances in Databases and Information Systems: 23rd European Conference, ADBIS 2019, Bled, Slovenia, September 8-11, 2019, Proceedings 23, pp. 53-68. Springer, 2019.
11

---

Published as a conference paper at ICLR 2025
[18] Seyedehdelaram Esfahami, Giovanni De Toni, Bruno Lepri, Andrea Passerini, Katya Tentori, and Massimo Zancanaro. Exploiting preference elicitation interactive and user-centered algorithmic recourse: initial exploration. arXiv preprint arXiv:2404.05277 2024.
[19] Virginia Bubanks. Automating inequality: How high-tech tools profile, police, and punish the poor. St. Martin's Press, 2018
[20] Council of the European Union European Parliament. Regulation (eu) 2024/1689. http://99..ttts///eeeeeeee europa.
/oj.. Accessed: 2024-08-30.
[21] Experian. Experian/fair, isaac model v2 score factor codes. https: http:::/wwwwtruudllgeeeee..
[22] Patrice Alexander Ficklin, Tom Pahl, and Paul Watkins. Innovation spotlight: Providing adverse action
notices when using ai/ml models. https: / wwn.e...nnn.eereeaaneeeeeeeeeeee htt.://eee....aoooo.oo///aaaaaaa innovation spot Accessed: 2024-07-16.
[23] FICO. Explainable machine learning challenge, 2018. URL htt.ttt///commmitty..ffet waaiinaaleemmccii -eane--eeeeeeeeeeeennnnn cha lengee.
[24] FinRegLab. Empirical white paper: Explainability and faireess: Insights from consumer lending.
Technical report, FinRegLab, July 2023. URL
http://fiiinegla.ooo////////// orgp 2alcodd//2033/12/inneee Laiiaaab_22aaa22 1ttp:/00-111_mmmriaa1---------1- Bxpplalnnaalltyyy-----aaaamm
Lending pdf.
[25] Fabian Fumagalli, Maximilian Muschalik, Patrick Kolpaczki, Eyke Hüllermeier. and Barbara Hammer. Shap-iq: Unified approximation of any-order shapley interactions. Advances in Neural Information Processing Systems, 36, 2024.
[26] Sainyam Galhotra. Romila Pradhan, and Babak Salimi. Explaining black-box algorithms using probabilistic contrastive counterfactuals. In Proceedings of the 202! International Conference Management of Data, pp. 577-590, 2021.
[27] Michele Gilman. Poverty lawgorithms: A poverty lawyer's 's guide fighting automated decision decision-making harms on low-income communities. Data & Society, 2020.
[28] Sofie Goethals, David Martens, and Theodoros Evgeniou. Manipulation risks in explainable ai: The implications of the disagreement problem. arXiv preprint arXiv:2306. 13885, 2023.
[29] Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. Advances in Neural Information Processing Systems, 29, 2016.
[30] Mikella Hurley and Julius Adehayo. Credit scoring in the era of big data. Yale JI. & Tech., 18:148, 2016.
[31] Neil Jethani, Mukund Sudarshan, Ian Connick Covert, Su-In Lee, and Rajesh Ranganath. Fastshap: Real-time shapley value estimation. International conference on learning representations, 2021.
[32] Kaggle. Give Me Some Credit. http: //tp://wwwww /www. kaggle e. //to/////ii cctp://cc/iveeeeeeeeerreetttt... 2011.
[33] Margot E Kaminski. The right to explanation, explained. In Research Handbook on Information Law and Governance, pp. 278-299 Edward Elgar Publishing, 2021.
[34] Amir- moir-Hossein Karimi, Julius Von Kügelgen, Bernhard Schölkopf, and Isabel Valera. Algorithmic recourse under imperfect causal knowledge: a probabilistic approach. Advances neural information processing systems, 33:265-277, 2020.
[35] Amir-Hossein Karimi, Bernhard Schölkopf. and Isabel Valera. Algorithmic recourse: from counterfactual explanations to interventions. In Proceedings of the 2021 ACM Conference Fairness, Accountability and Transparency, FAceT '21, pp. 353-362 New York, NY, USA, 2021. Association for Computing Machinery, ISBN 978145085 450383033 doi: 10.1145/34421888.4455555 URL https:///oo....... //tp::////o......./////oo/
1145/3442188 .141118884448854 3445899.
[36] Harmanpreet Kaur, Harsha Nori, Samuel Jenkins, Rich Caruana, Hanna Wallach, and Jennifer Wort- man Vaughan. Interpreting interpretability: understanding data scientists' use of interpretability tools for machine learning. In Proceedings of the 2020 CHI conference on human factors in computing systems, pp. I-14, 2020.
12

---

Published as a conference paper at ICLR 2025
[37] Seunghun Koh, Byung Hyung Kim, and Sungho Jo. Understanding the user perception and experience of interactive algorithmic recourse customization. ACM Transactions on Computer- Computer-Human Human Interaction, 2024.
[38] Avni Kothari, Bogdan Kulynych, Tsui- Wei Weng, and Berk Ustun. Prediction without preclusion: Recourse verification with reachable sets. In The Twelfth International Conference Learning Represeniations. net wt-.ummioosCC22ppooo
2024. URL pesttps://ppezeeeeee
[39] Satyapriya Krishna, Tessa Han, Alex Gu, Javin Pombra, Shahin Jabbari, Steven Wu, and Himabindu Lakkaraju. The disagreement problem in explainable machine learning: A practitioner's perspective. arXiv preprint arX:::2202.0 01602 2022.
[40] Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. NeurIPS, 2017.
[41] Charles Marx, Richard Phillips, Sorelle Friedler, Carlos Scheidegger, and Suresh Venkatasubrammnnian. Disentangling influence: Using disentangled representations to audit model predictions. Advances in Neural Information Processing Systems, 32, 2019.
[42] Charles Marx, Flavio Calmon, and Berk Ustun. Predictive multiplicity in classification. In Proceedings of Machine Learning and Systems 2020, pp. 9215-9224. 2020.
[43] Ramaravind K Mothilal, Amit Sharma, and Chenhao Tan. Explaining machine learning classifiers through diverse counterfactual explanations. In Proceedings of the 2020 conference fairness, accountabiliry, and transparency, pp. 607-611 617, 2020.
[44] Bitya Neuhof and Yuval Benjamini. Confident feature ranking. In Sanjoy Dasgupta, Stephan Mandt, and Yingzhen Li (eds.), Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, volume 238 of Proceedings of Machine Learning Research, pp. 1468-1476. PMLR, 02-04 May 2024. URL https: : /tee:://prrreee ppooeedingge . ml presssppessss.... seress/v233 v2338 2 neuho html.
[45] Duy Nguyen, Ngoc Bui, and Viet Anh Nguyen. Distributionally robust recourse action. arXiv preprint arXiv:2302. 11211 2023.
[46] Martin Pawelczyk, Teresa Datta, Johan Van den Heuvel, Gjergji Kasneci, and Himabindu Lakkaraju. Probabilistically robust recourse: Navigating the trade-off's between costs and robustness in algorithmic recourse. The Eleventh International Conference on Learning Representations. 2023.
[47] Manish Raghavan, Solon Barocas, Jon Kleinberg, and Karen Levy. Mitigating bias in algorithmic hiring: Evaluating claims and practices. In Proceedings of the 2020 conference on faireess accountability, and transparency, pp. 469-481, 2020.
[48] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Why should I trust you?: Explaining the predictions of any classifier. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1135-1144, ACM, 2016.
[49] Pedro Saleiro, Benedict Kuester, Loren Hinkson, Jesse London, Abby Stevens, Ari Anisfeld, Kit T Rodolfa, and Rayid Ghani. Aequitas: bias and faimeess audit toolkit. arXiv preprint arXiv:1811.05577 2018.
[50] Andrew D Selbst and Solon Barocas. The intuitive appeal of explainable machines. 2018.
[51] Lloyd S Shapley. A value for n-person games. Contribution to the Theory of Games, 2, 1953.
[52] Dylan Slack, Sophie Hilgard, Emily Jia, Sameer Singh, and Himabindu Lakkaraju. Fooling lime and shap: Adversarial attacks on post hoc explanation methods. In Proceedings of the AAAI/ACM Conference on Al, Ethics, and Society, pp. 180-186, 2020.
[53] Kacper Sokol. Alexander Hepbum. Raul Santos Rodriiuzz,, and Peter Flach. blimey: surrogate prediction explanations beyond lime. arXiv preprint arXiv:1910.13011 2019.
[54] Winnie F Taylor. Meeting the equal credit opportunity act's specificity requirement: Judgmental and statistical scoring systems. Buff. I.. Rev., 29:73, 1980.
[55] The Lawyers' Committee for Civil Rights Under Law. Online civil rights act, December, 2023, URL tpp::///att //wp::///wwww.. wawwersscomm toc. org/ wwwlllnnncivvlrriiieee
[56] Sohini Upadhyay, Shalmali Joshi, and Himabindu Lakkaraju. Towards robust and reliable algorithmic recourse. arXiv preprint arXiv:2102.1 13620, 2021.
[57] Berk Ustun, Alexander Spangher, and Yang Liu. Actionable recourse in linear classification. In Proceedings of the Conference on Fairness, Accountability and Transparency, FAT* '19 pp. 10-19. ACM, 2019. ISBN 978-1 978-1-4503-6125-5. doi: 10.1 14453287550.32875566
13

---

Published as a conference paper at ICLR 2025
[58] Suresh Venkatasuuramaaiia and Mark Alfano. The philosophical basis of algorithmic recourse. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pp. 284 244-293, 2020.
[59] Julius Kügelgen, Schölkopf. On the fairness of causal algorithmic recourse. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 9584-9594, 2022.
Karimi, Umang Bhatt, Isabel Valera, Adrian Weller, and Berhaard
[60] Jamelle Watson Daason, David Parkes, and Berk Ustun. Predictive multiplicity in probabilistic classification. In AAAI Conference on Artificial Intelligence, 06 2023.
[61] White House. Blueprint for an Al of rights: Making automated systems work the American people. The White House Office of Science and Technology Policy, October, 2022. URL ht http / //www. whi tehcuse. gwv/ostp. ottttsstp/aassao
[62] Laurence Wolsey. Integer programming. John Wiley & Sons, 2020.
[63] Blake Woodworth, Suriya Gunasekar, Mesrob Ohannessian, and Nathan Srebro. Learning non- discriminatory predictors. In Conference on learning theory, pp. 1920-198099 1953, PMLR, 2017.
[64] Wykstra. Government's use algorithm serves up false fraud charges. undark, 6 january. 2020.
[65] Muhammad Rehman Zafar and Naimul Mefraz Khan. Dlime: A deterministic local interpretable model- agnostic explanations approach computer-aided diagnosis systems, arXiv preprint arXiv:1906. /906. 10263,
2019.
[66] Zhengze Zhou, Giles Hooker, and Wang. S-lime: Stabilized- model explanation. Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, pp. 2429-2438, 2021.
14

---

Supplementary Materials
Supplementary Material for Section
A.I Implementation Details for Reachable Set Enumeration
A.2 Implementation Details for Reachable Set Sampling
B Supplementary Experiment Details
B.1 Details for the he Dataset..
B.2 Details for the german Dataset
10
B.3 Details for the givemecredit Dataset
11
B.4 Overview Model Performance
Supplementary Experiment Results
12
C.I Responsiveness of Explanations for Random Forests
12
C.2 Feature Responsiveness Rankings
13
Supplementary Case Study Details
17
D.1 Actionability Constraints
17
D.2 Model Performance
18

---

Published as a conference paper at ICLR 2025
A SUPPLEMENTARY MATERIAL FOR SECTION 3
Define partition {TI, :2y.... ja of [d] such that given two parts Ti, Wn. there joint constraints between all pairs (p, q) E Tm x of features. Another way to think about feature partitions would be connected components in a graph, where features are nodes and edges represent joint constraints (i.c., edge (p. q) < there are joint actionability constraints between and q).
Let x be part such that j E I²'
A.1 IMPLRMENTATION DETAILS FOR REACHABLE SET ENUMERATION
Description of Find1 DAction Routine The Find1 DAction routine in Algorithm 2 enumerates set of possible actions given an intervention on a feature by repeatedly solving a mixed-integer program. At each iteration, the routine takes input:
E X, point
(a)). valid intervention on feature j
(z), the action set for feature j (which is specified by separable and joint actionability constraints
on j)
Ag set of [[] actions enumerated over previous iterations.
Given these inputs, the procedure solves the following mixed-integer program in Eq. (3) to return the nearest single- feature actions from the a E Az(x) \ AAP when they exist, to confirm that there are no more actions to enumerate.
min Σ kEW!
0g
0j 1/
(3a)
k E n'
reconstruction av positive component of
(3b)
0.
k E n'
(3c)
4 2 -a+ de
k E
negative component of az
(3d)
a < sup V=((=) ITk
E m' & n'
a 0 σx
(3e)
a. inf Vi V=(=))( (1 oTk)
ax 0g
(3f)
au. 5/. 1F
& n' a Ag"
maintoin from prior
(3g)
Σ (0 + 07,1)
É ATT
solution away from
(3h)
REA!
1 M, uk.1
h ee kEn
(3i)
15 M2 uk.1)
6,,1 UE
(3j)
Az(=))
joint actionahiliry constraints absolute value of ex signed distances from az.t
(3k)
02, af E RR
NEn &E
(3l)
t 87, E 1t
(3m)
Me, {0, 1)
sign indicator of b sign indicator of 6xk.
(3n)
de f0, 1}
kEn
(3o)
As mentioned in Section 3.1, DAction solves the following optimization problem:
Find1DAction(z, v, A,) = argmin ||a]|a s.t. =
(x)
Since ||a]]| Exetg |axl. we model the absolute value using positive and negative components + ax (constraints (3b), (31)). In similar fashion, we decompose 6u,,, which is the distance from existing solutions a; A, into tt. o, (constraints (3g), (3m)). ds and uay E (0, 1) indicator
2

---

Published as a conference paper at ICLR 2025
variables for the sign of ag and Ök,1, where indicates its positive and 0 otherwise (constraint (3n), (3o)).
The first constraint, (3a) enforces that we intervene feature j with v. The remaining constraints describe three key requirements for a:
1. Sufficient distance from prior solutions (constraint (3h))
Adherence to separable actionability constraints (constraint (3e), (3f), (31), (3j)
Adherence joint actionability constraints (constraint (3k))
Constraint (3h) ensures that given Emin our experiments with discrete datasets.
0, ||a aill2 2 Emin Va; A. We set Emin 0.5 for
Constraints (3e), (3f) ensure that aş is feasible under separable constraints on k E x and that only one of dg or Ox is strictly positive. Similarly, constraints (3i), (3j) ensure that the distances between and each a, are within some bound. We achieve this by setting "Big-M" parameters M Mg. which represent the upper bound for 6,,, and o5,-- For each feature & x', we let
Mt. suu Vxx agiil, My == inf V (x)) ax.11l
Along with the indicator variable ua,,, Mi,,, My ensure that only one of oz, or 0% is strictly positive and is feasible under separable actionability constraints.
Constraint (3k) ensures that also adheres to joint actionability constraints. These constraints will exist if and only | > 1. See [38] for examples of how we can explicitly encode joint actionability constraints into Eq. (3).
This formulation an adaptation of the MIP presented in [38]. Our formulation above differs that solve for actions with respect fixed intervention e.
A.2 IMPLEMENTATION DETAILS FOR REACHABLE SET SAMPLING
Sampling a reachable set requires additional considerations.
The sample size N controls the precision of the estimated responsiveness score (2). We formalize precision using confidence intervals by treating úú (x) as a binomial distribution parameter:
Choosing Sample Size
Remark 2. Given point æ E X, let (x) denote a sample of N points drawn uniformly from the reachable set Ry(x). Given any model f X + y. we can estimate the responsiveness score for feature j as (x)(x)= == 1o EwvER())) 1[f(x') = glarse)) Given a significance level a E (0, 1), we have that:
Pr(uç(a) E (z) (,);(a) + 8)) 1
Here: E K. ( (1) (x)) and (z) -IL ( 2 is corrected estimator to
improve coverage when (ı,(x) (0, 1} [10], S l== E R2 (ar) [(x') = jurrse))) the subset of responsive points, and k 4 - )) is a constant based on the Normal CDF 4(-).
The Agresti-Coull interval above is an approximate confidence interval for a binomial proportion [3]. offering improvement the standard normal approximation known as the Wald Interval. It is particularly effective for small proportion values, providing more reliable coverage - the probability that the interval contains the true parameter value [10].
Fact 3. For a fixed o, N, & is maximized when = and atlains its minimum at S = 0 and S S=N N.
8+
2 No*2. Then, we have:
Proof. Let
2(1 - 2)
E = kg (N + x?)
3

---

Published as a conference paper at ICLR 2025
Since N and a are fixed (x is defined by a),, & is function of of the form h(2) c/2(11 2]
where E IR+.
Take the derivative with respect to wt
(1(z) 2 21|=(1 [x]] (1 2z)
1
dl - 2z)
2/2(1 -2)
See that h²())= when
Take the second derivative to check that is concave.
(2) 2 -2[x(1 1 5 -1 22) -2)1-
2)]
=(1 - 2) + -1=
[=(1 - =]}
[a(1 -
[1]=- =)] -
4
Since 0 and c 0, (2) < 0, hence it is concave and attains its maximum when
S=
We see that h1(2) > where 2 < 4. meaning it increasing for E (0, } Hence the local minimum achieved at the smallest possible x when S = 0.
Similarly, for = E 1 1), (2) < and the local minimum achieved at the largest possible when S = N.
Note that the value of h (or E) are the same those two points.
Using the fact above, we list two ways of setting N (given o:):
1. Control the precision when (i.e., points in R2 are responsive) width of the shortest interval
control the
Control the precision when S= the width of the widest interval 2 (i.e., half the points in R, (2:) are responsive)
control
Either way, fix and solve for N given the width of the interval & at specified S. Below provide a table the smallest N required for different interval widths at common values of for the two methods:
3
E
0.01
0.02
0.05
0.10
0.01
0.02
0.05
0.10
0.01
461
227
86
39
0.01
16581
4141
657
160
0.05
267
132
50
23
0.05
9600
2398
381
93
0.10
188
93
35
0.10
6762
1689
268
65
Table 6: Minimum N required to ensure the shortest confidence interval is less than 2 (Method 1)
Table 7: Minimum N required to ensure the widest confidence interval is less than 2E (Method 2)
Let be the feature that we are intervening on. Case 1: |r|| 1 (i.e., x' {j). j is not jointly constrained with other features).
1Action Description of the Sample1 DAction Routine
4

---

Published as a conference paper at ICLR 2025
Here, there are no downstream effects from intervening on j. We take uniformly random intervention from (x):
(z)
which abides by j's separable actionability constraints like feature bounds and monotonicity.
Case 2: |m|] > 1 (i.e., j is jointly constrained with other features)
Similarly Case 1, we first proceed by taking a uniformly random intervention on j:
(x)
However, there downstream effects the intervention. We breakdown x' into three disjoint subsets:
= U U a
where diise and Thz are the sets of discrete and and continuous features x' respectively. We consider the following three sub-cases:
Case 2a: |uu = 0 - all other features in are discrete.
We repeatedly solve the MIP in Find1 DAction to build S(v) {a e Ay(x) la v, the set of all possible actions resulting from intervention v. The resulting action is a uniformly random sample from (()) a" ((v)
Case 2b: | = all other features in are continuous.
We sample interventions for each feature in Tissc
z(v)
Σ
2/
RE
where #5 (x),, sample from the intervention set for feature k.
Then we check feasibility for a* = v + =(v) by running CheckFeasibilityaa AAyla"",""(()) (z)).
We repeat until we find feasible a".
Case 2c: Is | > 0 part contains discrete and continuous features.
We run the sampling steps in Case 2a, 2b for T'isc and i''s to get Adise and Ccets
We then check feasibility on a" adise + aets v (we subtract v since both adise aas contain v) by running CheckFeasibiliyy "ataasiiiiittaa,,,,,, Ax(r)).
repeat until we find a feasible a".
Description of CheckFeasibility Routine We describe the implementation for the Check baackkeaiibil;y((;;; a" Aj) in Algorithm 1. Contrary to the MIP formulation in Appendix A. given the original point X' and the sampled action a", solve the MIP once.
min
aj = 1/j
intervene with vo
(4a)
0A 0 4x
k
reconstruction of ax
(4b)
44 2
k n'
positive component ak
(4c)
4
k T'
negative component of ax
(4d)
< sup (=
h E m'
0 au =1
(4e)
0 < inf (1))([1-0)
l BEW
o
(4f)
(z)
joint actionability construints on
(4g)
at 0g R4
k E
of as
(4h)
5

---

Published as a conference paper at ICLR 2025
on E {0,
k EET
sign indicator os
(4))
The formulation is variant of the problem in Appendix A.I, where:
a A3 2
and set the objective to mina
Hence CheckFeasibility(z, a", AA) = 1 a" is feasible under actionability constraints and other- wise.

---

leads Published as a conference paper at ICLR 2025
B SUPPLEMENTARY EXPERIMENT DETAILS
B.1 DETAILS FOR THE oe DATASET
The heloc dataset created to predict repayment Home Equity Line of Credit HELOC applications; these are loans that use people's 's homes as collateral. The anonymized version of the dataset was developed by FICO for use in an Explainable Machine Learning Challenge in 2018 [23]. Each instance in the dataset is an application for a home equity loan from single applicant. There are 10, 459 instances and d 23 features. Here, the label y; if applicant has been more than 90 days overdue on their payments in the last years and y: = otherwise. We thermometer encode continuous integer features after dropping rows and features with missing data (see Table 8). See GitHub for dataset processing code.
Nama
Articnuhiitt
Sign
Joine Coutralnte
Partisin III
(6:1) 0
Yec
30
21,
24,
(6.1)
0
Yec
31
21,
24,
24,
12.37
[6.13 6
31
21
12.3
14
Resinatale www.eetaaaaaeeooo......o..a. aaacee2)
[6.13 6
Yee
21
21
24
12.33
14
Reninatati Coadeage
66.13 6
Yea
20
21
32.33
14
Hestratali caadeetiieaaceeee
6.11 6
Yee
20
21
32.33
14
Honinatalt Coadede
6.11 6
Yea
20.
21
24
32.33
HentratallCow www.rraaeeeettteeaeeeeeeeeo
6.41 6
20. 21
24
32.93
Henferolvtte togeeeee
6.11
34.35
66.1
(5.13
(5.15
(5.15
34.13
Henferoirttg wet.:r/gggoddeeettteeeeoooo
65.13 6
34.35
05.13
00.13
34.75
05.15
20.15
20.13
Het wws.Pactkkoennne anw.aaaiiiiiiiiiiii
00.13
(5.11
Hetfi www.aaattotttttaaeeeaeeeea
05.11
65.11
05.13
05:13
05.11
05.13
05.13
(6.13
(6.1)
(6.1)
No
Table 8: Separable Actionability Constraints for the processed heloc dataset. Type indicates the feature type (Z integer. [0, 1} for binary). LB, UB are the lower and upper bounds for the feature. Actionability indicates whether the feature is globally actionable. Sign indicates if the feature can only increase (+) or decrease (-). Joint Constraints are list non-separable constraint indices it is tied to (if any). Partition ID indicates which partition the feature belongs to.
Actionability Constraints
The joint actionability constraints listed in Table include:
1. DirectionalLinkage: Actions on Muuneroa..iigiIrrdeeeeeeeeeee will induce to actions ('NumRevol SwwnneevolvingTradee22222 2'ac unit change NumRevolvvingTradessli 10:1.00 change in NumtevolviingTrades222
annce leads
2. DirectionalLinkage: Actions
will induce to actions
on ['Num I'NumInstal Insaal "Trades22').aan Rach unit change in Numinstal TradesWia TumlnsttallraddsWWs.. lance22 leads
to: 1, 10.00.0 change in NumInatall 11rddes222
3. DirectionalLinkage: Actions on W "NuuievoLvingTradee Each unit change in SainReevoovinggTaddetttalala to:1. 10000nn change in www.eeeveeinnnnlrrdessss
will induce to actions
7

---

Published as a conference paper at ICLR 2025
4. DirectionalLinkage: Actions on Nuntnssal will induce to actions on SaunInstallTrades222 to: lo:0..tummt change in NumInstallTrades22
unit change in IaminstaallTTaadsaaaaaaaaaaa leads
DirectionalLinkage. Actions on NumRevoiv MunneeewwriiggrradeeeEEaaeeee will induce to actions on
NNumRevolving wrggg unit change in NumievolvingTTradessiii ance leads 1o.1000-uit change NumtevolvingTrraes:25
DirectionalL.inage:: Actions on W on ['NunInstallrrade555]]]] to: 10:1.00-unit change in NumInstallTrades:
will induce to actions unit change in funtoesaallrraddelaaaaaaaa2 leads
7. DirectionalLinkage: Actions on NumRevol wiw.grrrraadsssslacccce222 will induce to actions on "NunderoovvingIIeeee222// to: to:1.00-umit change in NunnevoLriiggIrade=>>
unit change Nunneevoovviggrrateeeaaaa...... leads
DirectionalLinkage: Actions on Fundatttaardds/saaaaaeeeeee will induce actions on ['NumInstall 1 unit change in KumlnstaallTrades/haal 2ncce leads to: 1..1..0-umit change NamInstallTrades]
DirectionalLinkage: Actions on baarsShcelastDeegrraaeees will induce to actions on "YearssFAccouttissttrr/
unit change in YearsSäncellstteelgrraasss leads to:- 1. 1.00-unit change in YearsofAccountilistoo
10. Directionall innkage: Actions on YearsSi hhellasttDeeqgrrddss will induce to actions on
('Year wwoooucccossttooooooooo 3.00-umit change in Yearsoffcccuntllssorry
unit change in I rTrades33 leads to:-
11. DirectionalLinkage: Actions on YearsSince LeeeaaaDeelgrrdess will induce to actions on unit change in Yerrshncciatttolgaeeee leads to:- 5,00-unit change in Years0fAccoutHHsttryy
12. ReachabilitCConstraint: htt.eeeeettraaeeiiiannaaaassttse reachability conditions.
The values of [
must belong to one of 4 values with custom
13. ThermometerFFncoding::
[YearsSi inelaiineelaaateeeereeeaa inclaasteee [f must preserve ther- mometer encoding of CearsiincelastDDelllaaaee which can only Actionss.Actions can only turn off higher-level dummies that are on, where SearsssnccIassteellgraaaa lowest-level dummy and VearsinceLasDelgraaaae
Actions
Yearssi ncee wtt.eaaatteellaareeeeeeeeee
is the
is the highest-level-dummy
14. ThermommeterEncodingg:
AvgYearsInFile25 SonXXearIIFFlee>1 must preserve thermometer encoding of AvgYearsinFilegeq.. which can only Actronse.Actions can only turn on higher-level dummies that are off, where AvgYears inFi 1e23 is the lowest-level dummy and Avayears IF 1e27 is the highest-level- dummy.
Actions on [avgYearsInFile>)
15. ThermometerFFncoding::
Actions on NetFract iueFraattooneoooovnnggeeeeo which can only Actions can is is the highest-leeel-dumyy
[Net www.raccttneeeeeeeeoooooo oo
O, must preserve
NetFract DottFractt.ooeevovvinne www.ooooooiggnrdeeeeeeoeooeeeeee thermometer encoding of DndFractioRRvoolvinnnueeeeng only off higher-level dummies that are on, where oottrrottoooreevv..nnggrr.r.r.. the lowest-level dummy NetFractions bet
16. ThermometerEnccding:
Actions
ReiFractiionnnntaaliibrrrer DettrracticnIInsttLLLeerrrr must preserve
N wtt.eettiinnnstaaaluuuaaaa thermometer encoding of NatFrrationnnnsallBurreeenn which can only decrease only turn off higher-level dummies that are on, where raat on Instal Burden211 10 is the lowest-level dummy and WetrraattoonnntallBbie
is the highest-level-uummy
17. ThermometcrEEccoding::
huuneewwllvinggrrraeeesaaaaeeee Nunntevolviingrradeesaaaaaaaa wunteero..viingzzccdeesleeeeee must preserve thermometer encoding of NumRe-
Actions
on
TaangTTTrresWaaaaeceee which only decrease.Actions only turn off higher-level dummies that are on, where MinafevwouinnggiaadssiisiLLeee is the lowest-level dummy and Numievol rades is the highest-leeel-dummy.
8

---

Published as a conference paper at ICLR 2025
18. ThermometerEncooding: Actions on [BunRevolviingrraass22, Sumnevooriingrraee:23. NumRevolving Vumneevo-vvinnIaadee222 NuntevolvingTradde27]] must preserve thermometer encoding of MumnRevolvinTTradesgoo level dummies that are on, where Numkevclvingg ( is the lowest-level dummy and NumRevo vi ngTrades27 is the highest-level-dummyy
which can only decrease.Actions only turn off higher-
19. ThermometerEEccoding::
Fuunriatta.LLPdddeReaaaaaeeee NumInstaliTran auunrntttallaadddaaaaameeeee Nun.hnsttallITaadeiaaaaaaaeeee must preserve thermometer encoding of Numinstall- TradesWBalancegeee which can only decrease.Actions can only turn off higher-level dummies that are on, where bun.nnstalllrradeeeeeeaacnc
Actions
is the lowest-level dummy and wwn.instta.rrrddeeeaaaaeeee. is the highest-level-dummy,
20. ThermometerEnccoding: Actions on [NumInstallTrades22 must preserve thermometer encoding of Nu- minstallTradesgeq. which can only decrease can only turn off higher-level dummies that on, where Num Insta aettallrrddde22 is the lowest-level dummy and Num nstal l'rades?7 the highest--evel-dummy.
NumInstallTrades2: Www.nstallTradesss

---

Published as a conference paper at ICLR 2025
B.2 DETAILS FOR THE german DATASET
The german dataset was created in 1994 and contains information about loan history. demographics, occupation, payment history, and whether or not somebody is good customer [15]. Each instance is credit applicant. There are n = 1, 000 instances and = 20 features. The features are all either categorical discrete. The label indicates is loan applicant is "good" (y 1) or "bad" (v/ 0). There are no missing values in the dataset. We renamed some of the features to be indicative of the values they represent. The dataset is self-contained and anonymous, and it includes features describing gender, age, and marital status.
Name
UB
Actionalility
Constraiuts
Partition
Age
19
75
0,4,1 12
moke Restidence
Yzs
4, 12
Yearralip Layedgg
[0.1]
Yzs
Chhecle ligieetr
[0,1]
Yis
92, 33
30
Checktnggten
Yzs
SavAnguhestt_ easaaa
[0.1]
Yos
36,
31
Gavtnngskent2210
(0.1)
Y:s
34, 35
[0.1]
singlu
(0,1)
No
zi sob
2
No
Engetenttz
(0.1)
Reue ingröwner
[0.1]
aomingetine
(0,1)
No
Jobmneellldd
[0,1)
No
[0,1]
10
Creditiktt 21
(0.1)
Creditfat 22000X
(0.1)
No
12
CxediiiAt.250000
[0.1]
Crecitiet wetsstttt220000 210000K
[0.1)
No
14
[0.1)
LoanDuratiionn
(0.1)
16
Scanburation2:4
(0.1)
No
17
Aoantuu atinn1336
[0.1)
Leaniaat
2
No
19
(0.1)
Yes
ucantte podr eddrod auallees Locao
(0.1)
Regu i inderootteeeca
(0,1)
23
Leanfegus Care
[0.1]
Reanieee in indeoootiinn
[0,1]
No
24
2i slaseavddiaas a
(0.1)
No
26
anerydnne negneney
(0,1)
27
tlaberyd: mment
[0.1]
Yzs
Slatesyd/Storeeiae Lirenta
(0,1)
Yes
29
Table 9: Separable Actionability Constraints for the processed german dataset. Type indicates the feature type (Z for integer, {0, }} for binary). LB, UB are the lower and upper bounds the feature. Actionability indicates whether the feature is globally actionable. Sign indicates if the feature can only increase (+) or decrease (-). Joint Constraints are a list non-separable constraint indices it is tied to (if any). Partition ID indicates which partition the feature belongs to.
Actionability Constraints
The joint actionability constraints listed in Table 9 include:
1. DirectionalLinkage: Actions on TcarsAcResidence will induce to actions on ['Age"] E.ch unit change in "earsAtResidence leads to:1.00 change in Age
2. DirectionalLinkage: Actions on YearsEmployed21 will induce to actions on ['Age'].Each J..Ecch unit change Years leads to:1.00-unit change Age
ThermometeerEncodingg Actions on StheckingAcetexist. CheckingAcet20] must preserve thermometer encoding of CheckingAcct, which can only increase, Actions can only turn on higher-level dummies that are off, where Checkinngcctexist.. the lowest-level dummy and CheckingAcct20 is the highest-level-dummy.
Actroase.Actions
4. TherrmometrrEncodding:: Actions on serve thermometer encoding of SavingsAcct, which can only increase.Actions only turn higher-level dummies that are off, where SavingsAcctexist. is the lowest-level dummy and Savingsåcct21 is the highest-level-dummy.
SavingsAcct2100 must pre-
10

---

Published as a conference paper at ICLR 2025
B.3 DETAILS FOR THE givemecredit DATASET
The givenecredit dataset is used to determine whether a loan should be given or denied [32]. The label indicates whether someone was 90 days past due in the two years following data collection. Delinquency refers to debt with an overdue payment; this dataset is used to predict someone will experience financial distress in the next two years. contains information about recipients, and each instance represents borrower. There are d = 10 features before preprocessing. Here, the label is y; if applicant has had serious delinquency in two years and y/s otherwise. The data is gelf-contained and anonymous, and it contains features describing age, income, and the number of dependents.
120, 268 loan
Name
Type
1.B
UB
Actionability
Siga
Joint
Construints
Partition ID
credit 1352acion210
[6.1]
Yes
12, 13,
15
10
Coudd tiissttii11iea.nm233000
[0.1]
12,
credi www.eeeeiieeaaaioooooo
[0.1]
1
Yes
12, 13,
14,
16
10
15,
Coeds credittiiiatttsssssseennttttttt
[0.1]
12.
Honthhy
[0.1]
1
Yes
9010, 11
[0.1]
1
Yes
10, 11
[0,]]
17,
11
19,
Anydredt weneetttiiineeeeeeeiooooo
[6.1]
Yos
10,
12
19,20
Age
[5.1]
Apuiat _5_0a__8 30
[0.1]
1
No
Age to.55
Aqu2600
[0.1]
1
No
[0.1]
1
No
[0.1]
[0.1]
1 No
BustaroEEepaaddeee e
[0.1]
[6.1]
1
No
13
oratorge Eiggeency
[0.1]
Table 10: Separable Actionability Constraints for the processed rivemecredis dataset. Type indicates the feature type (Z for integer, {0, 1} for binary). LB, UB are the lower and upper bounds for the feature. Actionability indicates whether the feature is globally actionable. Sign indicates if the feature only increase (+) or decrease (-). Joint Constraints are list non-separable constraint indices it is tied to (if any). Partition ID indicates which partition the feature belongs to.
Actionability Constraints
The joint actionability constraints listed in Table 10 include:
1. ThermometerEnncoding:: Actions on [MonthlyInnome2238 MenthlyInccoe225k MorthlyIncome2100 must preserve thermometer encoding of MonthlyIncomegeq.. which can only Ancronse.Actions can only turn on higher-level dummies that are off, where MonthlyIncome2 3k the lowest-level dummy and MonthlyIncome2> is the highest-level- dummy.
2. ThermometerEnccdding::
Actions on
[Creddtiinn@tthlliessiiii/2
Credit wtw.:neittiiiiaiii22000110 Credi neut iiiiztio2770 0, Preditiineutiii iitiiiiiiiiiiii natiio22000000 must preserve thermometer encoding of Credit inttiiitinntllizziiinggg.. which can only decrease can only turn off higher-level dummies that are on, where lowest-level dummy and Credittineettliiatiiooo
0 is the is the highest-level-dummy
Thermometer Actions on [AnyRea must preserve thermometer encoding of continuusaatribuut. which can only Iecrease.Actions can only turn off higher-level dummies that are on, where anyRcalEsttteeoans the lowest-level dummy and Multi MultipleReall waw.eeeellattttooann is the highest-level-dummy
4. ThermometerEncooinng:
on (AnyCred:ttinesAnddI must preserve thermometer encoding of contin- uousattribute, which can only decrease.Actions can only turn off higher-level dum- mies that are on, where wnycredittiieesndd... is the lowest-level dummy and multipleCredi Li is the highest-level-dummy
Actions
Multip
11

---

Published as a conference paper at ICLR 2025
B.4 OVERVIEW OF MODEL PERFORMANCE
We include the performance of the classifiers used in Section 4.
LR
XGB
RF
Dataset
Train
Test
Train
Test
Train
Test
heloc
= 5, 842
0.772
0.788
0.859
0.785
0.780
0.790
FICO [23]
gezman
000
0.819
0.760
0.971
0.794
0.828
0.766
= 36
Dua & Graff [15]
g gävemecredüt
7 120, 268
0.841
0.844
0.875
0.793
0.864
0.835
d 23
Kaggle [32]
Table 11: Train and Test AUC for models across all datasets. We optimized the model's hyperparameters through randomized search and divided the data into training and testing sets at an 80% and 20% ratio.
C SUPPLEMENTARY EXPERIMENT RESULTS
C. RESPONSIVENESS OF EXPLANATIONS FOR RANDOM FORESTS
All Features
Actionable Features
Dataset
Metrics
LIME
SHAP
LIME-AW
SHAP-AW
RESP
helo2
% Presented with Explanations
100.0%
100.0%
100.0%
100.0%
31.7%
842
L % All Features Unresponsive
86.5%
78.2%
77.1%
76.7%
0.0%
= 43 features
4 % At Least Feature Responsive
13.5%
21.8%
22.9%
23.3%
100.0%
A mutable
% All Features Responsive
0.0%
0.0%
0.0%
0.5%
100.0%
FICO [23]
Features Highlighted
4.0
4.0
4.0
4.0
2.4
german
% Presented with Explanations
100.0%
100.0%
100.0%
100.0%
48.0%
1, 1,000
% All Features Unresponsive
100.0%
89.1%
76.6%
64.6%
0.0%
= 36 features
L % At Least Feature Responsive
0.0%
10.9%
23.4%
35.4%
100.0%
d 9 mutable
Features Responsive
0.0%
0.0%
0.0%
100.0% %
Dua & Graff [15]
Features Highlighted
4.0
4.0
4.0
4.0
2.2
gi givemecredi:
% Presented with Explanations
100.0%
100.0%
100.0%
100.0%
93.2%
120, 268
All Features Unresponsive
56.5%
26.8%
28.4%
21.0%
0.0%
d = 23 features
4 % At Least Feature Responsive
43.5%
73.2%
71.6%
79.0%
100.0%
A 13 mutable
% Features Responsive
0.0%
0.5%
1.4%
11.4%
100.0%
Kaggle [32)
Features Highlighted
4.0
4.0
4.0
4.0
2.9
Table 12: Responsiveness of feature- feature-highlige highlighting explanations for RF for all methods and datasets. We generate explanations that highlight up top-scoring features from a given method. We report the proportion of individuals receiving an explanation (% Presented with Explanations)
mean number of features in each explanation (# Features Highlighted). We also show the proportion of instances where all features are unresponsive (% All Features Unresponsive) highlighting positive values; least one feature is responsive (% At Least I Feature Responsive), all features responsive (% All Features Responsive) highlighting the best value.
12

---

Published as a conference paper at ICLR 2025
C.2 FEATURE RESPONSIVENESS RANKINGS
We include a plot to show how responsive features are at different rankings by LIME. SHAP, LIME-AW, SHAP-AW and RESP for each dataset. For every denied individual, rank features by their absolute feature importance score returned by these methods. We exclude features with 0 attribution from the rankings.
The plots below show the % of times where the feature at each rank are responsive (i.e., feature has RESP 0). It allows visualize and compare how often these methods assign high attribution responsive features.
C.2.1 heloc
Dataset: lico, Model: LR, Method: SHAP
Dataset: fice, Model: LR, Method: LIME
Times Responsive
Responsivve
SHAP
SHAP-AW
RESP
LIVE
LIMEAW
RESP
Figure 4: Responsiveness of features for individuals who are denied credit by the LR model on according to absolute feature attribution rank using the original feature attribution method, its action-aware variant and RESP. For cach method, we report the proportion of individuals with at least one responsive intervention feature with the k-th largest score (k-th ranked feature). Features must have non-zero score be included "rank." "
dataset
13

---

Published as a conference paper at ICLR 2025
Dataset: XGB, SHAP
XGB, Method
Pank
40
20
100
of Times Responsive
Times Resporsive
SHAP M SHAP-AW
RESP
LINE
K
LIME-W
RESP
Figure 5: Responsiveness of features for individuals who denied credit by the XGB model the fico dataset according to absolute feature attribution rank using original feature attribution method, action- aware variant and RESP. For each method, we report the proportion of individuals with at least one responsive intervention on feature with the k-th largest score (k-th ranked feature). Features must have non-zero score to be included in "rank."
Dataset: fico, Model RF, Method: SHAP
Dataset: fioo, Modet RF, Mothod: LIME
2222
Times Responsive
Resporsivve
SHAP
SHAP-AW
RESP
LIVE
LIMEAW
RESP
Figure Responsiveness of features for individuals who are denied credit by the RF model on according to absolute feature attribution rank using the original feature attribution method, its action-aware variant and RESP. For cach method, we report the proportion of individuals with at least one responsive intervention feature with the k-th largest score (k-th ranked feature). Features must have non-zero score be included in "rank." "
dataset
14

---

Published as a conference paper at ICLR 2025
C.2.2
german
Datasat: Modal: LR, Method: SHAP
Dataset: german, Modal: LR, Method: LIME
20
40
80
80
100
20
40
60
30
100
% of Times Responsive
% Times Responsive
SHAP
Th SHAP-AW
RESP
LIME
LIME-W
RESP
Figure 7: Responsiveness of features for individuals who are denied credit by the LR model the german. dataset according to absolute feature attribution rank using the original feature attribution method, its action- aware variant and RESP. For each method, we report the proportion individuals intervention on a feature with the k-th largest score (k-tth ranked feature), Features must have non zero score to be included in "rank."
least one responsive
Dataset: german, Model: XGB, Method: SHAP
Dataset: german, Model: XGB, Method: LIME
Times Responsive
Responsive
SHAP
SHAP-AW
RESP
LIVE
/
LIMEAW
RESP
Figure Responsiveness features for individuals dataset according to absolute feature attribution rank using the original feature attribution method, its action- aware variant and RESP. For each method, we report the proportion of individuals with at least one responsive intervention feature with the k-th largest score (k-thh ranked feature). Features must have non-zene score be included "rank."
denied credit by the XGB model
german:
15

---

Published as a conference paper at ICLR 2025
german, Model
Datsset: german, Method:
40
20
100
of Times Responsive
Times Resporsive
SHAP K
SHAP-AW
RESP
LINE
K
LIMEAW
RESP
Figure 9: Responsiveness of features for individuals who denied credit by the RF model the german. dataset according to absolute feature attribution rank using original feature attribution method, action- aware variant and RESP. For each method, we report the proportion of individuals with at least one responsive intervention on feature with the k-th largest score (k-th ranked feature). Features must have non-zero score to be included in "rank."
C.2.3 givemecredit
Dataset: givemezredt. Model: LR, Methad SHAP
Dataset: givembaredi, Mode: LR, Method: LIME
Times Responsive
Responsivee
SHAP
SHAP-AW
RESP
LIVE
LIMEAW
RESP
Figure 10: Responsiveness features for individuals who are denied eredit by the LR model on givemecredit dataset according to absolute feature attribution rank using the original feature attribution method, its action-aware variant and RESP, For each method, we report the proportion of individuals with at least responsive intervention feature with the k-th largest score (k th ranked feature). Features must have non zero score included "rank."
16

---

Published as a conference paper at ICLR 2025
giverneredit,
Dataset: givemecredi.
40 of Times Responsive
20
100
of Times Resporsive
SHAP M SHAP-AW
RESP
LINE
K
LIME-AW
RESP
Figure 11: Responsiveness of features for individuals who denied credit the XGB model the givemecredit dataset according to absolute feature attribution rank using original feature attribution method, its action-aware variant and RESP. For each method, we report the proportion of individuals with at least one responsive intervention on a feature with the k-th largest score (k-tth ranked feature). Features must have non-zero score be included in "rank."
Datasel: givemecredit, Model: RF, Method: SHAP
Dataset: givemecredit. Model: RF,
Responsive
Resporsive
SHAP
SHAP-AW
RESP
LINE
/
LIME AW
RESP
Responsiveness of fcatures for individuals who denied credit by the RF model on the givemecredit dataset according to absolute feature attribution rank using the original feature attribution method, its action-aware variant and RESP. For each method, we report the proportion of individuals with at least one responsive intervention on a feature with the k-th largest score (k-th ranked feature). Features must have non-zero be included in "rank."
Figure 12:
D SUPPLEMENTARY CASE STUDY DETAILS
D.1 ACTIONABILITY CONSTRAINTS
The joint actionability constraints listed in Table 13 include:
17

---

Published as a conference paper at ICLR 2025
Type
Actionbillty
fige
storgeflatteeyy tnsssst thears
[6:
(6,
Yes
0.8.
100
Yes
5,0
Yie
Nowthh twooee
Ored mest L' Licationn
0.0
90706 0
Yee
EastoryOttataaaggee
[0.1]
No
Table 13: Separable Actionability Constraints for the processed continuous g vemecredic dataset. Type indicates the feature type (& for integer, {0, 1 for binary). LB, UB are the lower and upper bounds for the feature. Actionability indicates whether the feature is globally actionable. Sign indicates if the feature can only increase (+) decrease (-). Joint Constraints are list non-separable constraint indices it is tied to (if any). Partition ID indicates which partition the feature belongs
1. CenetioonaLinnkagee duce
Actions on NumberReal satte will in- actions on Thammeocoopaaatttttttaaeeeaasssr E.cch unit change in Sumberrre.usstateooooorrii hatbberrffopecrreeeiiieeeeeer..e
leads to:1.00-unit
change in
2. Directionall.inkage Actions Hi aistorooffaateerppmmetntt.asssss will induce actions on ['Age'].Each JaEacch unit change in #istoryOtaateeyyeenttnnsszezz leads to:2.00-unit change Age
3. Directionallinkage Actions #istorryffaallmneeen/nnnnsasssss will induce actions on [' J.Each unit change Historrgffhaalnnngeeey////maaas leads to:2.00-unit change in Age
D.2 MODEL PERFORMANCE
XGB
Dataset
Train
Test
gi vemeereeiit
120, 268
0.937
0.830
Kaggle [32]
Table Model Performance XGB model on the givemecredit dataset Section
18


Reviews:
[{"review_id": "fiJ24ksXj3", "reviewer": "Reviewer_hbQN", "summary": "This paper addresses the limitations of feature attribution methods in decision-making scenarios where regulations mandate that individuals are informed of the principal reasons for adverse decisions. Existing methods often highlight important features without considering whether those features can actually be changed to achieve a desired outcome, thereby failing to provide actionable recourse.\nThe authors propose a new method for scoring feature responsiveness, which evaluates the likelihood that changing a particular feature can lead to a favorable model outcome.  The authors conducted an empirical study in consumer finance, demonstrating that their responsiveness-based approach more accurately identifies features that offer actionable recourse, flagging instances where standard methods provide “reasons without recourse.” They also release a Python library to support the implementation of feature responsiveness.", "questions": "Address questions in the weakness section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper effectively highlights a critical shortcoming in the use of feature attribution methods for generating explanations. By demonstrating that standard methods often provide non-actionable reasons, the authors reveal a key limitation that could undermine the intended goals of explainability and consumer protection in high-stakes applications.\n\n2. The introduction of responsiveness scores offers a novel/nuanced approach to measuring \"actionability.\" These scores help flag instances where individuals cannot achieve recourse, thereby preventing the issuance of misleading explanations and enhancing the practical value of model explanations.", "weaknesses": "(1) While I find the concept of feature responsiveness quite interesting, the contributions of this paper appear marginal, particularly in light of the work [1]. This paper draws upon existing ideas introduced in [1], such as the notions of reachable sets and action sets, which are used for recourse verification there (i.e., determining if an individual can achieve recourse through actions in the feature space). In [1], the approach returns a binary output: 1 if there exists an action that achieves recourse (feasible) and 0 if no such action exists (infeasible). In contrast, this work provides a probabilistic measure by assessing the fraction of actions on a feature that leads to recourse. Although the responsiveness score method offers a more nuanced probabilistic view, the framework in [1] could also be adapted to identify unresponsive features effectively. For example, the experiment of Table 3 (Recourse feasibility....) is similar to Table 2 in [1].\n\nTo improve the contributions, I would like to see this work go beyond identifying unresponsive features or \"actionability\" of features to leveraging the responsive score to produce better explanations. For example, can we leverage this responsiveness score to produce feature attribution methods that return more actionable features? Or when finding recourse (or counterfactual explanations) can we leverage the responsiveness score to find more actionable recourses? \n\n(2) Generally, I also believe your method can go beyond checking the responsiveness of top-k features from a feature attribution method. You can also extend this to various recourse generation methods, counterfactual explanations, or any feature explainability methods.\n\n(3) I challenge the assumption that feature attribution methods should necessarily provide actionable recourse. Attribution methods and recourse methods are fundamentally different types of explanations, each with distinct purposes and applications. While feature attribution methods are designed to identify and communicate the features that most strongly influence a model’s prediction, they are not inherently designed to satisfy an \"actionability axiom\". This distinction is important because blending the goals of feature attribution and recourse may lead to confusion and misaligned expectations. The paper might benefit from a brief discussion on this but I don't think the premise of \"feature attribution method should be able to provide recourse\" needs to be true. \n\n\n(4) The definition of responsive score as a probability is a bit confusing as there is no random variable or distribution. It might be more precise to define the responsive score as a fraction or proportion. To accurately define responsive score as a probability you would need to properly characterize your sample space, probability measure, and events.\n\nWhat if two features are individually irresponsive, but when considered together can be responsive (and can provide a recourse)?  \n\n(5) The responsiveness score does not consider the ease or difficulty of implementing certain actions, which is a crucial factor from the user's perspective. For instance, actions that are relatively easy to undertake, such as increasing a credit score by 2 points, should carry more weight than more challenging actions, increasing a credit score by 20 points. A weighted version of the responsiveness score, which factors in the feasibility or effort required for each action, could offer a more user-centric measure. This might better capture \"user responsiveness\" rather than a uniformly distributed action space.  For instance, what do you do when a feature is continuous and unbounded (e.g., income feature)? \n\n  \nminor --\nIncrease the text font size of Figure 1 for clarity.\n\n\n[1] Kothari, Avni, et al. \"Prediction without Preclusion: Recourse Verification with Reachable Sets.\" arXiv preprint arXiv:2308.12820 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vh0O3cDKUd", "reviewer": "Reviewer_PaRQ", "summary": "The paper proposes a new concept within interpretability of ML models, focusing on explanations for recourse. Even though the approach is specially proposed, developed and discussed within the context of lending, I believe it has general applicability for a broader set of problems. Basically, instead of considering shapley values or other concepts to assign a contribution/importance to some input feature, it concentrates instead on assessing sensitivities of the forecasts to changes in the features. These features that could be changed are seen as recourse. While the idea is very simple, and could be seen as a simple sensitivity measure, I believe it can be a very understandable and powerful approach to convey information that would support interpretability much better than Shapley values, for many applications.", "questions": "I do not have any specific point or question to the authors.\nPossibly, if they could try to discuss their approach in a more general context (i.e., envisaging what could be done for other application areas) at various stages of the paper, it could make it stronger.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "I clearly agree with the starting point of the paper, aiming to focus on another view of interpretability, where for a number of applications, only features that are actionable are relevant, while one would be interested in seeing how different outcomes may be if values for these features would change (even slightly). It is crucial to go away from thinking that Shapley values (and other similar approaches) are the go-to approach to bring interpretability to ML. Here, the approach is described in a rigorous and pedagogical manner. The concepts and main measure proposed (in Definition 4) are simple yet powerful. The methodological contribution is sound, original and valuable. Personally, I enjoyed reading the paper and it made me think about a lot of potential use cases and extensions.", "weaknesses": "In my view the only weakness of the paper is that it only concentrates on a given application area, while it could have been interesting to consider the ideas and concepts in the paper in a more general framework.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kvjy63aafr", "reviewer": "Reviewer_6js1", "summary": "This work studies feature attribution and its relationship to recourse. It demonstrates that many feature attribution techniques do not highlight features that can lead to recourse, i.e. these methods do not identify features that if an individual were to change them, the model. prediction would be different. This work then addresses this issue by proposing a way to score features on the basis of responsiveness – i.e., the probability that an individual can attain a desired outcome by changing a specific feature. The paper presents efficient methods to compute such scores and demonstrates via experiments that their proposes responsiveness scores highlight features that can lead to recourse.", "questions": "What is line 96 saying? \n- \"By highlighting features that can be changed to consumers who could not attain a different outcome\n- I thought current methods fail to identify features that are actionable, i.e. those that could be changed to achieve a different, hopefully desirable outcome? I am not sure what this line is saying.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1) I believe this paper aims to address a very important problem with current feature attribution methods, that is, the features these methods identify as important are rarely those that can be modified/changed so that a different prediction may occur. \n\n2) I believe the idea of action and reachable sets is an interesting one and I appreciate the authors trying to make these notions precise via theory.\n\n3. The paper performs numerous experiments to demonstrate their result", "weaknesses": "1) I don't particularly think this work is very novel. There has been numerous works that propose counterfactual feature attribution methods, i.e. those that identify important features as the ones that, when changed, lead to a different prediction. Are these not pretty much responsiveness scores in the language of this paper. I advise the authors to take a look at these papers and address how their proposed notion of responsiveness is different than the notions proposes in the following works\n\nhttps://proceedings.mlr.press/v162/dutta22a/dutta22a.pdf\nhttps://arxiv.org/abs/2409.20427\nhttps://arxiv.org/pdf/1912.03277\n\nThe beginning of the 3rd page of the first paper [Dutta et al 2022] in fact list many works that focus on develop feature importance scores that identify important features as those when changed, lead to a different prediction\n\n2) How are the actionable sets defined exactly? To be correct the only way to properly understand model behavior is to see how its predictions change while modifying features such that the new feature vector x' still comes from the same distribution that the original feature vector comes from. This is because the model was trained on samples from this distribution and so to evaluate how it performs on samples that may not make sense does not make sense to me. Is the reachable set trying to capture all \"feasible\" ways that we can change the feature vector? How do identify this reachable set?\n\n3) Overall I have concerns on the novelty and limited impact of this work. These ideas have been explored in numerous works and whats outlined in the paper seems to only apply to simple settings with discrete features where its clear what the reachable set is.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cNx6u8LF0z", "reviewer": "Reviewer_pPyj", "summary": "The authors propose feature responsiveness scores to highlight the features that can be changed to receive a better model outcome and assess how feature attribution methods can inflict harm by providing users with reasons without recourse.", "questions": "The questions are integrated with the context in the weakness section. Authors should please respond to the questions added in the sub-section \"weaknesses and questions\"", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The authors propose a novel way of looking at actionability. \nThe authors perform comprehensive experiments using three datasets and explainability methods and avail their code.", "weaknesses": "**Weaknesses and questions**\n\n- There are endogenous and exogenous actionable features. When defining the actionability constraints (e.g., from the way authors define them - table 2 for example), it’s non-trivial to fully capture the relationship between features, especially in a way that reflects varied real-world scenarios. The action set and reachability set might not accurately capture these dynamics. For example, by making age immutable, a user is limited to recourse that doesn’t involve age changing (e.g. figure 2), and in some settings, e.g., education and age, if an individual changes from education:1 to education:3, age changes as well. As a result, the formulation of the responsiveness score might be limited.\n- While some features are not actionable, they could be highly predictive and ethical. For example in predicting who is admitted to a K1 class, age is a very predictive feature despite being a sensitive attribute and not being actionable. Similarly, disability is a very predictive feature in predicting eligibility for the Paralympics despite being a sensitive attribute and not being actionable. So although the feature might not be actionable and not eligible for actionable recourse recommendations, if methods like LIME and SHAP attach a high feature attribution score to it, it’s not a negative thing. I think the responsiveness score is context-dependent and the proposed formulation could be stronger with some way of filtering instances where the score is most informative or applicable.\n- The reliance on the computation of the responsive score on the reachable set is very limiting. This is because 1) The decision-maker has to define the reachability set based on the action set (which might be limited by the training data manifold and is susceptible to breaking causality and the relationship between features). 2) It’s hard to compute the responsiveness score in cases where features are continuous and sampling might introduce bias that's propagated and exacerbated in the explanation. 3) For methods like LIME or SHAP where the objective is to get a sense of the most important features among a set of features or determine the value of features to the model, defining an action set or individualized recourse might be out of scope. \n- The formulation of actionability and subsequently responsiveness score such that the number of features changed = number of actions taken is rarely the case in real-world settings. For example, while a student might be required to get at least a grade A in class w and at least a grade B in class z, one action might lead to the achievement of both grades (features).\n- In addition to the quantitative results (tables and numbers ), it might be helpful to add more qualitative results as well. For example, on page 9, lines 467 to 468, the authors briefly do this.\n\n**Minor  or other observations**\n\n- I think it might be interesting to further investigate the responsiveness score at an individual or instance level. I imagine that in a real-world setting, the responsiveness score might be limited by what actions different individuals have access to, varied user preferences and strengths/circumstances (potentially captured by individual costs), and so on.\n- Additionally, in several recourse methods, several alternative sets of actions, e.g., with different costs, or diverse actionable features, are returned, where e.g., in some instances, the least costly is chosen. \\\nHow well or easily is the responsiveness score scalable to this setting? Additionally, how do authors determine sufficient alternatives recourse as cutoff in that case, for example would it be dependent on diversity of the features recommended in recommended varied recourse, or number of alternative recourse, etc?\n- Given the dependence on a manual action set definition, the responsiveness score is susceptible to biases and replication or reproducibility challenges. \n- The figures and tables communicate what the authors want to convey. However, the text font is very small. Authors could find ways to increase the font to improve the readability of the figures and tables. \n- Given that only users that have received a negative outcome get recourse, would it be computationally cheaper to more generally bound the reachability set, since the action set is more general -specific to the training set-?\n- How does a change in training data distribution affect the feature responsiveness score? \n- The reachability set and responsiveness scores are more likely to leak significant information about the model than the local explanation tools like SHAP and LIME.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)