Feature Responsiveness Scores: Model-Agnostic Explanations for Recourse

## Abstract

Abstract Machine learning models are often used to automate or support decisions in applications such as lending and hiring.
In such settings, consumer protection rules mandate that we provide a list of “principal reasons” to consumers who receive adverse decisions.
In practice, lenders and employers identify principal reasons by returning the top-scoring features from a feature attribution method.
In this work, we study how such practices align with one of the underlying goals of consumer protection – recourse – i.e., educating individuals on how they can attain a desired outcome.
We show that standard attribution methods can mislead individuals by highlighting reasons without recourse – i.e., by presenting consumers with features that cannot be changed to achieve recourse.
We propose to address these issues by scoring features on the basis of responsiveness – i.e., the probability that an individual can attain a desired outcome by changing a specific feature.
We develop efficient methods to compute responsiveness scores for any model and any dataset under complex actionability constraints.
We present an extensive empirical study on the responsiveness of explanations in lending, and demonstrate how responsiveness scores can be used to construct feature-highlighting explanations that lead to recourse and to mitigate harm by flagging instances with fixed predictions.

## 1 Introduction

Machine learning models are now routinely used to automate or support decisions about people in domains such as employment [, ], consumer finance [], and public services [, , ]. In such applications, explanations are often seen as an essential tool to protect consumers who are adversely affected by the predictions of a machine learning model [, , , ]. Existing and proposed laws and regulations include provisions that require lenders or employers to provide explanations to individuals in such situations [, , , ]. In the United States, for example, the adverse action notice requirement in the Equal Credit Opportunity Act mandates that lenders provide “principal reasons” explaining why individuals are denied credit []. In the European Union, Article 86 of the AI Act [] grants individuals a right to obtain explanations to describe the “main elements” of their decision in high-risk applications regarding employment, education, financial systems, government benefits, law enforcement, and border control [see Annex III , for a definition of “high risk”].

The use of explanations in such settings reflects widespread beliefs about the effectiveness of transparency for consumer protection [] – i.e., that revealing information can protect and empower consumers []. For example, the adverse action notice is motivated by the fact that presenting consumers with “principal reasons” can: (1) promote *anti-discrimination* by highlighting potential bias; (2) facilitate *rectification*, by allowing individuals to identify and correct data errors, and; (3) support *recourse* by educating individuals on how to improve future outcomes []. Regulators provide model deployers with substantial flexibility in complying with these requirements. In practice, model owners comply with these regulations using feature attribution methods such as LIME and SHAP []. These are general-purpose methods that can explain the predictions of a model after it has been trained and generate explanations that can be communicated to consumers. These methods output scores that reflect the importance of each feature for a given prediction. Given these scores, model deployers can retrieve the top scoring features and present them to consumers as a list of “principal reasons” or “main elements” (see Fig. 1).

In this work, we study how to explain model predictions in a way that can achieve one of the main goals of consumer protection: *recourse*. We focus on achieving recourse through the use of feature attribution – techniques that are widely used in practice. Our work is motivated by the fact that regulations seek to achieve multiple goals; we claim that it is useful to align the design of an explanatory method with the goals it seeks to achieve. To this end, we study how well existing approaches for feature attribution methods support recourse, and develop an approach tailored to communicating with respect to this goal.
Our main contributions include:

- 1.
We present a feature attribution method to measure the responsiveness of predictions from a model. The *responsiveness score* measures the probability of changing the prediction of a model by intervening on a given feature. Our approach highlights features that can be changed to receive a better model outcome, and identifies instances that may lead to harm.
- 2.
We develop model-agnostic methods to compute feature responsiveness scores using *reachable sets*. Our methods can evaluate scores for any model, paired with theoretical guarantees that support our ability to flag harm, and can be readily adapted to achieve other goals.
- 3.
We conduct a comprehensive empirical study on the responsiveness of feature attribution in consumer finance. Our results demonstrate that common feature attribution methods output *reasons without recourse* by highlighting features that do not provide recourse, and underscore the benefits of our approach.
- 4.
We include Python source code to measure feature responsiveness available at the [project repository](https://github.com/harrycheon/responsiveness-public).

Figure: Figure 1: Feature-highlighting explanations for an individual denied credit by a logistic regression model for a lending task (see heloc, Section 4). We show explanations from top-scoring features using SHAP [] (left) and responsiveness scores (right). As shown, SHAP highlights features immutable (Age, HistoryOfLatePayment, and HistoryOfDeliquency) or unresponsive (CreditLineUtilization). In contrast, explanations build using responsiveness score (right) only two features that provide recourse for the individual.
Refer to caption: x1.png

##### Related Work

Our work is related to a stream of research on post-hoc explanations  [, , , , , , ]. We focus on methods for feature attribution, which are designed to evaluate the importance of feature for a given prediction. Many methods are built for use cases in model development [e.g., , , ], but are now used to construct “feature-highlighting explanations” to comply with regulations on explanations in consumer applications [see e.g., , ].

Our work shows how feature attribution methods can inflict harm in such cases by highlighting *reasons without recourse* – i.e., by highlighting features, when acted upon, do not change the prediction – sometimes to consumers who are assigned fixed predictions. This is a failure mode that affects a broad class of local explainability techniques – adding to a growing literature on how local explanations are prone to manipulation [e.g., , , , , ], and indeterminate [e.g., due to multiplicity , , , ]. Our work complements impossibility results on recourse and feature attribution showing that attribution methods that satisfy completeness and linearity (e.g., SHAP) perform no better than random guessing when inferring model behavior (e.g., recourse)  [see e.g., , ]. Here, we establish the prevalence of this effect empirically and develop a principled approach to mitigate it.

Our approach is related to a stream of work on algorithmic recourse [, ]. The vast majority of work on this topic develops algorithms for recourse provision – i.e., to present consumers with *actions* that can change the prediction a specific model [see e.g., ]. Our goal is to highlight features that can be reliably changed to achieve recourse. To this end, responsiveness scores measure the number of actions on a *single* feature. Our approach builds on a line of work that elicits and enforces complex actionability constraints [, ].
Here, we use this machinery to represent actionability constraints at an instance level, and to generate a set of all points that a person can reach under a set of actionability constraints [].
Our approach outputs feature responsiveness scores that can be used with any model, and can be adapted to address practical challenges in providing recourse – e.g., robustness [, , ] and causality [, , ].

## 2 Problem Statement

We formalize the problem of explaining the predictions of a machine learning model through feature attribution.
We consider a standard classification task where we wish to predict a label $y\in\mathcal{Y}=\{0,1\}$ from a set of $d$ *features* $\bm{x}=[x_{1},x_{2},\ldots,x_{d}]\in\mathcal{X}\subseteq\mathbb{R}^{d}$.
We assume that we given a model $h({}):\mathcal{X}\to\mathcal{Y}$ where each instance represents a person, and their features $\bm{x}_{i}\in\mathcal{X}$ encode semantically meaningful characteristics for the task at hand (e.g., age or income).
We assume that the feature values are bounded so that $x_{j}\in[l_{j},u_{j}]$ and $\left\|\bm{x}\right\|\leq B$ for all $\bm{x}\in\mathcal{X}$ and $B$ sufficiently large.
(^1^11This assumption holds for most semantically meaningful features [see ]. Some features have bounds by construction (i.e. binary features). In other cases, we can set loose bounds (e.g., for income).)

We consider a task where we provide explanations to individuals who are adversely affected by the prediction of a given model (see e.g., [, ]). We assume that $h({\bm{x}_{i}})=1$ represents a *target prediction* that is desirable – e.g., $h({\bm{x}_{i}})=1$ if applicant $i$ is predicted to repay their loan within 2 years – and thus will explain the predictions for individuals where $h({\bm{x}_{i}})=0.$

##### Feature-Highlighting Explanations

Our goal is to construct explanations where each feature is *responsive* – i.e., can be changed independently to achieve recourse.

The standard practice of explaining predictions is to use *feature-highlighting explanations* [see e.g. ]. These explanations consist of a list of “most important” features using a given method that we convert into a natural language description [e.g., a reason code ].

##### Feature Attribution Methods

The standard approach to construct feature-highlighting explanation is to use feature attribution method [].

###### Definition 1 .

Given a model $h({}):\mathcal{X}\to\mathcal{Y}$ and its training dataset $\mathcal{D}{}=\{(\bm{x}_{i},y_{i})\}_{i=1}^{n}$, a *feature attribution method* for point $\bm{x}_{i}$ is a function
$\bm{\phi}({\bm{x}_{i}\mid h({}),\mathcal{D}{}}):\mathcal{X}\to\mathbb{R}^{d}$, where the $j$th element of the output, $\phi_{j}({\bm{x}_{i}\mid h({}),\mathcal{D}{}})$ is the attribution for feature $j\in[d]$.

In what follows, we write $\bm{\phi}({\bm{x}_{i}})$ instead of $\bm{\phi}({\bm{x}_{i}\mid h({}),\mathcal{D}{}})$ when $h({})$ and $\mathcal{D}{}$ are clear from context. This function captures the behavior of several methods that are used to explain the prediction of a model in terms of its features:

- •
*Local Linear Explainers* [see e.g., , , , ]:
Given a model $h({})$ and a point $\bm{x}_{i}$, these methods fit a linear model $g:\mathbb{R}^{d}\to\mathbb{R}$ to approximate the decision boundary surrounding $\bm{x}_{i}$ such that $g(\bm{x}^{\prime})=\langle{\bm{\phi}(\bm{x}_{i})},{\bm{x}^{\prime}}\rangle$. The resulting attribution for each feature is its weight in $g$.
- •
*Shapley Value Methods* [see e.g., , , ]: Given a model $h({})$ and a point $\bm{x}_{i}$, these methods cast features as players in a cooperative game, and estimate $\phi_{j}({\bm{x}_{i}})$ as the marginal contribution of feature $j$ to the prediction $h({\bm{x}_{i}})$ under basic axioms of social choice [].

Given a model $h({})$ and its training dataset $\mathcal{D}$, the scores $\bm{\phi}(\bm{x}_{i})$ capture how each feature captures the prediction of a model at the $\bm{x}_{i}$ in different ways. In all cases, the scores satisfy the following properties:

- •
*Relevance*: A feature with an attribution score $\phi_{j}({\bm{x}_{i}})=0$ is not relevant to the prediction for $\bm{x}_{i}$ – i.e., it can be changed arbitrarily without changing the prediction [see e.g., the “missingness” axiom in ].
- •
*Strength*: Features with larger attribution scores have larger impact on the prediction – i.e., if $|\phi_{j}({\bm{x}_{i}})|>|\phi_{j^{\prime}}({\bm{x}_{i}})|$, then feature $j$ has a stronger contribution to the prediction than feature $j^{\prime}$.

These properties allow model developers to comply with consumer protection rules, but can promote misinterpretation among consumers [].

###### Definition 1 .

##### Reasons without Recourse

**Table 1: Stylized lending task where the best model assigns fixed predictions. We predict $y\in\{0,1\}={\scriptsize{\small\texttt{repayment}}}$ from two binary features $(x_{1},x_{2})$ = ( age $\geq$ 60, has_IRA). We fit a classifier data with $n_{0}$ negative labels and $n_{1}$ positive labels for each $(x_{1},x_{2})\in\{0,1\}^{2}.$ Individuals with $x_{1}=1$ can only change their features to $(x_{1},x_{2})\in\{(1,0),(1,1)\}$ since age $\geq$ 60 is immutable and has_IRA is binary.**
| Features | Label Counts | Best Model |  |  |
| --- | --- | --- | --- | --- |
| age $\geq$ 60 | has_IRA | $n_{0}$ | $n_{1}$ | $h({\bm{x}})$ |
| 0 | 0 | 51 | 10 | 0 |
| 0 | 1 | 7 | 30 | 1 |
| 1 | 0 | 21 | 8 | 0 0 |
| 1 | 1 | 31 | 17 | 0 0 |

One failure mode of machine learning in consumer-facing applications is that models can assign *fixed predictions* – i.e., predictions that cannot be changed by their decision subjects (see e.g., Table 1). In lending, for example, fixed predictions inflict harm through *preclusion* – i.e., permanently barring consumers from access to credit. Fixed predictions are a recently-discovered failure mode [], which must be mitigated through changes in feature construction, model development, or implementation (e.g.,using a separate model for re-applicants who are assigned fixed predictions). In practice, these instances are often left undetected. We can mitigate harm if instead of providing misleading explanations of fixed predictions they are flagged to model owners or auditors.

These issues stem from an oversight of *actionability* – how we can change the features of a model. On the one hand, models assign fixed predictions because they use features that can only be changed in specific ways. On the other, we are unable to detect these instances through feature attribution methods because they are designed to explain a prediction, rather than how it can be changed.

##### Accounting for Actionability

Given these challenges, we introduce machinery to capture how features can be changed at the instance level. For example, a change in one feature might necessitate a change in another; this makes strictly independent changes to certain features infeasible.

###### Definition 2 .

An *action* is a vector $\bm{a}=[a_{1},\ldots,a_{d}]\in\mathbb{R}^{d}$ that a person can perform to change their features from $\bm{x}_{i}$ to $\bm{x}_{i}+\bm{a}=\bm{x}^{\prime}\in\mathcal{X}$. Given a point $\bm{x}_{i}\in\mathcal{X}$, the *action set* $A({\bm{x}_{i}})$ contains all possible actions for $\bm{x}_{i}$. We assume that every action set contains the *null action* $\bm{\emptyset}\in A({\bm{x}_{i}})$.

Action sets captures how we can change features from a given point as a set of *actionability constraints*. As shown in Table 2, we can elicit complex constraints from human experts in natural language, and convert them into equations that we can embed into an optimization problem. In this way, we can enforce actionability in – for example – algorithms to find recourse actions [see e.g., , ].

**Table 2: Examples of actionability constraints on semantically meaningful features for a lending task (see Section A for additional examples). Each constraint can be expressed in natural language and embedded into an optimization problem using standard techniques in mathematical programming [see, e.g., ].**
| Class |  | Example | Features | Actionability Constraint |
| --- | --- | --- | --- | --- |
| Immutability |  | age cannot change | $x_{j}={\small\texttt{age}}$ | $a_{j}=0$ |
| Monotonicity |  | recent_payment can only increase | $x_{j}={\small\texttt{recent_payment}}$ | $a_{j}\geq 0$ |
| Integrality |  | late_payments must be positive integer $\leq 12$ | $x_{j}={\small\texttt{late_payments}}$ | $a_{j}\in\mathbb{Z}^{+}\cap[0-x_{j},12-x_{j}]$ |
| Encoding Validity |  | preserve one-hot encoding of categorical feature<br>housing_status $\in\{{\small\texttt{own}},{\small\texttt{rent}},{\small\texttt{other}}\}$ | $x_{k}={\small\texttt{housing_status=own}}$<br>$x_{l}={\small\texttt{housing_status=rent}}$<br>$x_{m}={\small\texttt{housing_status=other}}$ | $a_{j}+x_{j}\in\{0,1\}~{}\textrm{for}~{}j\in\{k,l,m\}$<br>$\sum_{j\in\{k,l,m\}}a_{j}+x_{j}=1$ |
| Logical Implication |  | if ${\small\texttt{has_savings_account}}={\small\texttt{TRUE}}$<br>then ${\small\texttt{savings_account_balance}}\geq 0$<br>else ${\small\texttt{savings_account_balance}}=0$ | $x_{j}={\small\texttt{has_savings_account}}$<br>$x_{k}={\small\texttt{savings_account_balance}}$ | $a_{j}+x_{j}\in\{0,1\}$<br>$a_{k}+x_{k}\in[0,10^{12}]$<br>$a_{j}+x_{j}\leq 10^{12}(x_{k}+a_{k})$ |
| Causal Implication |  | if years_of_account_history increases<br>then age will increase commensurately | $x_{j}={\small\texttt{years_of_account_history}}$<br>$x_{k}={\small\texttt{age}}$ | $x_{j}+a_{j}\leq x_{k}+\delta_{k}$<br>$\delta_{k}\in[0,100]$ |

To highlight features that are responsive, we must assign a score to features that accounts for actionability constraints. In practice, the actionability constraints for a given feature will include constraints that pertain to the feature as well as other features. We refer to the features that may change as a result of interventions on feature $j$ as *downstream features*, $C_{j}$.

###### Definition 3 .

Given an action set $A({\bm{x}_{i}})$ for a point $\bm{x}_{i}\in\mathcal{X}$, the *action set* for feature $j\in[d]$ is:

$$ $\displaystyle A_{j}({\bm{x}_{i}}):=\{\bm{a}\in A({\bm{x}_{i}})~{}|~{}\bm{a}_{j }\neq 0\wedge\bm{a}_{k}=0,k\in[d]\setminus C_{j}\}$ $$

Here, the downstream set $C_{j}:=\{k\in[d]\setminus\{j\}~{}|~{}\bm{a}_{j}\neq 0\implies\bm{a}_{k}\neq 0~
{}~{}\forall\bm{a}\in A({\bm{x}})\}$ is the subset of all features that must change as a result of interventions on feature $j$.
Definition 3 captures cases where actions on a feature can induce changes in other features. Such cases can stem from deterministic causal relationships – e.g., increasing years_of_account_history should lead to a commensurate change in age. In general, they can capture dependencies that would not be included in a traditional causal graph – e.g., changing a categorical attribute will require switching a binary feature “off” while turning another binary feature “on” (so that $x_{j}=1\to 0\implies x_{j}^{\prime}=0\to 1$).

###### Definition 2 .

###### Definition 3 .

## 3 Measuring Feature Responsiveness

In this section, we introduce our main technical contribution – the *responsiveness score*. We first define the responsiveness score, then discuss its interpretation and computation.

### 3.1 Responsiveness Scores

Our goal is to measure the *responsiveness* of the prediction of a model at a point $\bm{x}_{i}$ with respect to the set of feasible actions on specific features. We propose to measure the sensitivity for each feature through the *feature responsiveness score*.

###### Definition 4 .

Given a model $h({}):\mathcal{X}\to\mathcal{Y}$, a point $\bm{x}_{i}$ with action set $A({\bm{x}_{i}})$ and feature $j\in[d]$, the *responsiveness score* for feature $j$ is defined as:

$$ $\displaystyle\mu_{j}({\bm{x}_{i}\mid h({}),A({\bm{x}_{i}})}):=\textnormal{Pr}( h({\bm{x}^{\prime}})=1\mid\bm{x}^{\prime}=\bm{x}_{i}+\bm{a},\bm{a}\in A_{j}({ \bm{x}_{i}})$ $$

The responsive score for a feature $j$ captures the proportion of single-feature actions on feature $j$ that change the prediction of a model $h({})$ at $\bm{x}_{i}$. In what follows, we write $\mu_{j}({\bm{x}})$ instead of $\mu_{j}({\bm{x}\mid h({}),A({\bm{x}_{i}})})$ when $h({})$ and $A({\bm{x}_{i}})$ are clear from context. Given a feature where $\mu_{j}({\bm{x}_{i}})=p$, we know that $100(p)\%$ of the single-feature actions on $j$, $\bm{a}\in A_{j}({\bm{x}_{i}})$ will change the prediction of the model. Thus, all actions to a feature where $\mu_{j}({\bm{x}_{i}})=0$ would not change the prediction while all actions on a feature where $\mu_{j}({\bm{x}_{i}})=1$ would result in a different prediction.

These interpretations are contingent on the actionability constraints used to compute the responsiveness score. In the simplest case, actionability constraints encode indisputable constraints on how a feature can be changed (e.g., feature encoding or physical limits) and so the responsiveness score for a given feature represent an upper bound on responsiveness: “at most $100\mu_{j}({\bm{x}_{i}})$% of single-feature actions on feature $j$ attain a desired prediction.” Such constraints let us flag undeniable instances of harm.
More generally, actionability constraints encode information about how other features are expected to vary when a single feature is changed. For example, if a model has a feature indicating the job rank of an individual, we can create actionability constraints that encode the expectation that if job rank increases, so does income.

###### Definition 4 .

##### Safeguards for Consumer Protection

One benefit of responsiveness scores is that we can reliably use them to detect when consumers are assigned fixed predictions, and when feature-based explanations can provide recourse.

###### Remark 1 .

Given a model $h({}):\mathcal{X}\to\mathcal{Y}$, let $\mu_{1}({\bm{x}_{i}}),\ldots,\mu_{d}({\bm{x}_{i}})$ denote the responsiveness scores of $\bm{x}_{i}\in\mathcal{X}$ with respect to the action set $A({\bm{x}_{i}}).$

- a)
If $\mu_{j}({\bm{x}_{i}})>0$ for some feature $j\in[d]$, then $h({})$ can provide recourse to $\bm{x}_{i}$ through a single-feature action on $j$.
- b)
If $\mu_{j}({\bm{x}_{i}})=0$ for all features $j\in[d]$, then either:
(i) $h({})$assigns a fixed prediction to $\bm{x}_{i}$, or
(ii) $h({})$can only provide recourse to $\bm{x}_{i}$ through actions that alter two or more features.

Remark 1a) states that every person ($\bm{x}_{i}$) who receives a positive responsiveness score for at least one feature has recourse. This implies that when we construct feature-highlighting explanations using the top-$k$ responsiveness scores, we will *only* provide explanations to individuals who have recourse.
Remark 1b) also illustrates how the responsiveness scores can flag for potential harm when $\mu_{j}({\bm{x}_{i}})=0$ and allows us to mitigate harm on a case by case basis. In case b)(i) – where a person is assigned fixed predictions – we would refrain from providing explanations to avoid misleading consumers, and flag the issue so that model development can be potentially revisited. In case b)(ii) – where a person is assigned predictions that can change through multiple actions – we could provide explanations that highlight subsets of responsive features, include explicit warning against presumptions of feature independence, or proceed in a similar manner to case b)(i).

###### Remark 1 .

### 3.2 Computing Scores with Reachable Sets

Figure: Figure 2: Simple example of how to compute responsiveness scores involving three independent features. age is an immutable feature, n_loans is a discrete feature taking values from 0 to 3 and has_guarantor is a binary feature. The original prediction of 0 is shown in the row highlighted in green. Single-feature actions for n_loans, has_guarantor are highlighted yellow and red respectively. We can see that we can construct a complete reachable set and directly calculate the responsiveness score for discrete datasets. The responsiveness score for age is 0 since it is immutable.
Refer to caption: x2.png

We compute responsiveness scores using a *reachable set*:

###### Definition 5 .

Given a point $\bm{x}_{i}$ and its action set $A({\bm{x}_{i}})$, we refer to the set of all points that are attainable through actions in $A({\bm{x}})$ as the *reachable set*:
$R({\bm{x}_{i}}):=\{\bm{x}_{i}+\bm{a}\mid\bm{a}\in A({\bm{x}_{i}})\}$.
We refer to the subset of points that are reachable through actions on feature $j\in[d]$ as the *reachable set* for feature $j$ and denote it as:
$R_{j}({\bm{x}_{i}}):=\{\bm{x}_{i}+\bm{a}^{\prime}\mid\bm{a}^{\prime}\in A_{j}(
{\bm{x}_{i}})\}$.

Reachable sets represents an alternative way to store and process information about actionability at the instance level. In particular, a reachable set $R({\bm{x}_{i}})$ encodes this information as a set of feature vectors that can be reached through feasible actions. Given reachable sets for each feature $R_{j}({\bm{x}_{i}})$ for $j\in[d]$, we can calculate responsiveness scores (see Definition 4) for a model by querying its predictions (see Fig. 2). This has the benefits that: (1) it can work with any model; (2) we only need to compute the reachable set once; and (3) it can allow us to evaluate other notions of responsiveness.

###### Definition 5 .

##### Enumeration for Discrete Features

When dealing with discrete feature spaces, we can obtain a complete reachable set through enumeration following an approach of Kothari et al. []. Given a point $\bm{x}_{i}$ and its action set $A({\bm{x}_{i}})$, this approach returns a set of all reachable points. Given a complete reachable set, we can calculate the responsiveness score of feature $j$ by evaluating the model $h({})$ on points reachable via single-feature actions on $j$ (see Fig. 2).
This approach can certify when an individual cannot change the prediction of a model. Unfortunately, the reachable set grows exponentially with the number of actionable features, which can lead to practical challenges in storage and compute.(^2^22Kothari et al. [] propose to enumerate reachable sets for subsets of features that can be altered independently. In practice, this can allow construction of reachable set for real-world classifcation datasets with discrete features. However, it may require considerable compute and storage. For example, on the heloc dataset which contains 31 actionable features and 5,616 distinct points, enumerating all reachable sets requires roughly 1 hour and storing them requires 18.8GB)

##### Sampling for Continuous Features

The enumeration technique described above is infeasible when we wish to evaluate the responsiveness of a continuous feature, or when a feature has downstream effects on continuous features. In such cases, we estimate the responsiveness score for feature $j$ by drawing a uniform sample of reachable points from $R_{j}({\bm{x}_{i}}).$
Given a feature without downstream effects – i.e., without downstream features so that $|C_{j}|=0$ – we can sample values from $[l_{j},u_{j}]$. When features have downstream effects, we can apply a rejection sampling procedure where we first sample values for all features from $[l_{j},u_{j}]$ and reject values that violate actionability constraints.

###### Definition 6 .

Given a point $\bm{x}_{i}\in\mathcal{X}$, let $\hat{R}_{j}({\bm{x}_{i}})$ denote a sample of $N{}$ points drawn uniformly from the reachable set for feature $j$, $R_{j}({\bm{x}_{i}})$.
Given any model $h({}):\mathcal{X}\to\mathcal{Y}$, we can estimate the responsiveness score for feature $j$ as

$$ $\displaystyle\hat{\mu}_{j}({\bm{x}_{i}}):=\frac{1}{N}\sum_{\bm{x}^{\prime}\in \hat{R}_{j}({\bm{x}_{i}})}\mathbbm{1}[h({\bm{x}^{\prime}})=1$ $$

When the samples are drawn uniformly from $R_{j}({\bm{x}_{i}})$, the number of responsive predictions $S\sim\textrm{Bin}(N,\mu_{j}({\bm{x}_{i}}))$. We can then quantify the uncertainty associated with our estimate $\hat{\mu}_{j}({\bm{x}_{i}})$:

###### Proposition 2 .

Given a level of significance $\alpha\in(0,1)$, the $100(1-\alpha)\%$ confidence interval for $\mu_{j}({\bm{x}_{i}})$ is:

$$ $\displaystyle\tilde{\mu}_{j}({\bm{x}_{i}})\;\pm\;\kappa\sqrt{\frac{1}{\tilde{N }}\tilde{\mu}_{j}({\bm{x}_{i}})(1-\tilde{\mu}_{j}({\bm{x}_{i}}))}$ $$

Here: $\kappa:=\Phi^{-1}(1-\tfrac{\alpha}{2})$, $\Phi(\cdot)$ is the Normal CDF,
and $\tilde{\mu}_{j}({\bm{x}_{i}}):=\frac{1}{N+\kappa^{2}}\left(S+\frac{\kappa^{2}}
{2}\right)$ is the corrected estimator.

We can alter two parameters to adjust the uncertainty: the level of significance $\alpha$ and the sample size $N$. For a fixed $N$, a larger $\alpha$ will yield a narrower confidence interval at a lower confidence level. Similarly, with a fixed $\alpha$, we can adjust $N$ to attain a desired level of certainty for the estimate $\hat{\mu}_{j}({\bm{x}_{i}})$. For example, given $\alpha=0.05$, $N>42$ would ensure that the width of the confidence interval surrounding $\hat{\mu}_{j}({\bm{x}_{i}})$ is at most 0.1 when we don’t observe any points with the target prediction – i.e. $S=0$. Note that we use a correction upon the standard binomial confidence interval formulation to improve coverage when $\mu_{j}({\bm{x}_{i}})=0$ or $1$ [see ].

This is a general-purpose approach that can compute responsiveness scores for features that are continuous or discrete. In settings where we are computing the responsiveness score of a discrete feature, we can reap the benefits of responsiveness using a small sample of reachable points. This approach avoids the computation and storage costs of enumeration but sacrifices the ability to identify individuals with fixed predictions with complete certainty.

###### Definition 6 .

###### Proposition 2 .

##### Discussion

One of the benefits of working with reachable sets is that they can readily be extended to handle other desiderata by weighing and filtering reachable points. As an example, we can highlight features that are robust to changes through a *robust* variant defined below.

###### Definition 7 .

Given a model $h({}):\mathcal{X}\to\mathcal{Y}$, a point $\bm{x}_{i}$ with action set $A({\bm{x}_{i}})$ and feature $j\in[d]$, the *robust responsiveness score* for feature $j$ measures the probability that an action on feature $j$ will flip the prediction regardless of the actions on other features:

$$ $\displaystyle\nu_{j}({\bm{x}_{i}\mid h({}),A({\bm{x}_{i}})}):=\textnormal{Pr}( h({\bm{x}^{\prime}})=1\mid\bm{x}^{\prime}=\bm{x}_{i}+\bm{a},\bm{a}_{j}\neq 0, \bm{a}\in A({x})$ $$

This variant can measure the prevalence of actions that would change a prediction even as other features change [see e.g., ]. We can calculate the robust responsiveness score by enumerating a subset of the entire reachable set $R({\bm{x}_{i}})$ containing points reachable by all actions that alter feature $j$.

###### Definition 7 .

## 4 Experiments

We present an empirical study on the responsiveness of explanations. Our goals are:
(1) to evaluate how our approach can support recourse and flag fixed predictions; and
(2) to demonstrate the limitations of existing feature attribution methods in practice.
We include additional results and details in Sections A and B, and code to reproduce these results at the [project repository](https://github.com/harrycheon/responsiveness-public).

##### Setup

We work with three classification datasets from consumer finance that are publicly available and used in prior work (see Section A for details).
Here, each instance represents a consumer and each label indicates whether they will repay a loan. For each dataset, we define *inherent actionability constraints* that capture indisputable requirements and that apply to all individuals – e.g., no changes for immutable and protected attributes, changes must preserve feature encoding and adhere to deterministic causal effects (see Section A).

We split each dataset into a training sample (80%; to
train models and tune hyperparameters) and a test sample (20%; to evaluate out-of-sample performance). We train classifiers using (1) *logistic regression* ($\mathsf{LR}$), (2) *XGBoost* ($\mathsf{XGB}$), and (3) *random forests* ($\mathsf{RF}$). For each model, we construct a feature-based explanation for each individual who is denied credit by listing the top-$k$ highest-scoring features from the following methods:

- •
*Feature Responsiveness Score* (RESP): We compute the score in Definition 4 using the procedure in Section 3.2, and the actionability constraints in Section A.
- •
*Standard Feature Attribution*: We consider local feature attribution methods that are model-agnostic
and widely used in the lending industry []: *Shapley additive explanation* (SHAP)  []; and *local interpretable model-agnostic explanations* (LIME)  [].
- •
*Actionable Feature Attribution*: We also consider *action-aware* variants of feature attribution methods SHAP-AW and LIME-AW, which seek to promote responsiveness by setting the scores for immutable features to 0 such that $\phi_{j}({\bm{x}_{i}})\leftarrow 0$ when feature $j$ is immutable.

**Table 3: Recourse feasibility across datasets and model classes. *% Denied* – the fraction of individuals denied credit by a model; *% 1-D* – the fraction of denied individuals who can achieve recourse with actions that alter a single feature; *% n-D* – the fraction of denied individuals who can achieve recourse with actions that alter 2 or more features; and *% Fixed* – the fraction of denied individuals who are assigned a fixed prediction (in red if $>0$).**
| Dataset | Metrics | $\mathsf{LR}$ | $\mathsf{RF}$ | $\mathsf{XGB}$ |
| --- | --- | --- | --- | --- |
| heloc<br>$n={5,842}$<br>$d={43}$ ($d_{A}={31}$)<br>FICO [] | % Denied<br>$\Rsh$<br>% Fixed<br>$\Rsh$<br>% 1-D Rec<br>$\Rsh$<br>% n-D Rec | 56.1%<br>19.1%<br>44.4%<br>36.6% | 58.3%<br>28.1%<br>34.6%<br>37.4% | 57.0%<br>49.1%<br>29.8%<br>21.2% |
| german<br>$n={1,000}$<br>$d={36}$ ($d_{A}={9}$)<br>Dua and Graff [] | % Denied<br>$\Rsh$<br>% Fixed<br>$\Rsh$<br>% 1-D Rec<br>$\Rsh$<br>% n-D Rec | 22.9%<br>7.4%<br>73.4%<br>19.2% | 17.5%<br>29.1%<br>51.4%<br>19.4% | 22.0%<br>15.5%<br>65.5%<br>19.1% |
| givemecredit<br>$n={120,268}$<br>$d={23}$ ($d_{A}={13}$)<br>Kaggle [] | % Denied<br>$\Rsh$<br>% Fixed<br>$\Rsh$<br>% 1-D Rec<br>$\Rsh$<br>% n-D Rec | 24.6%<br>15.6%<br>72.4%<br>12.0% | 24.7%<br>0.2%<br>93.2%<br>6.6% | 24.8%<br>11.5%<br>76.0%<br>12.5% |

##### Results

We summarize the viability of promoting recourse using feature-highlighting explanations in Table 3, and the responsiveness of explanations from each method in Table 4. We evaluate explanations built using the top-4 scoring features from each method, which reflects the recommended number of reasons to include in an adverse action notice required by the U.S. Equal Credit Opportunity Act [see , ].

Our results in Table 3 show that models admit features that allow *some* individuals to change them to attain a desired prediction (29.8% to 93.2% across models and datasets). At the same time, they reveal their potential to mislead individuals who are assigned fixed predictions (i.e., 0.2% to 49.1% across all models and datasets).
For example, given the $\mathsf{LR}$ model for the heloc dataset, we would present an explanation to 56.1% of individuals who are a denied loan. Among them, 44.4% can achieve recourse through single-feature actions; 35.6% can only achieve recourse through joint actions; and 19.1% have no path to recourse because they receive a fixed prediction.

##### On Responsiveness Scores

Our results in Table 4 show how our approach can support consumers by presenting responsive features and by flagging instances where explanations may be misleading. Explanations are only provided to individuals who can achieve recourse through a single-feature action, and are given to *all* such individuals (the values for *% Presented with Reasons* in Table 4 match the values for *% 1-D Rec* in Table 3). When we construct feature-based explanations using responsiveness scores, we present individuals with explanations that only contain responsive features, achieving 100% on the *% All Reasons Responsive* metric across datasets and models. This may result in explanations that highlight fewer reasons on average—for example, individuals receiving explanations from the $\mathsf{LR}$ model on german receive 1.9 out of 4 reasons on average. This behavior can mitigate harm as we avoid presenting explanations to individuals with fixed predictions or those who require joint actions to change their outcomes.

##### On Feature Attribution Scores

Our results show how standard methods for feature attribution can output explanations that are ineffective and potentially misleading. For example, under the $\mathsf{LR}$ model for the heloc dataset, we find that 82% and 75.6% of explanations from LIME and SHAP include 4/4 unresponsive features respectively. This behavior arises as a result of algorithm design, as the scores do not account for responsiveness nor actionability. This results in two key problems:

- *Low Scores for Responsive Features:* Methods can assign low scores to responsive features. On the heloc dataset, for example, 44.4% of denied individuals by the $\mathsf{LR}$ model can achieve recourse by altering a single feature. However, explanations built using LIME and SHAP fail to include them since their scoring mechanisms do not account for feature responsiveness. For instance, an individual could achieve recourse by acting on NumRevolvingTrades, but a feature-based explanation produced by LIME does not include it, as it assigns higher scores to four other features that are unresponsive. We also observe this phenomenon beyond the top-4 features in Fig. 3.
- *Reasons without Recourse:* Methods provide explanations to individuals with fixed predictions. On the heloc dataset, the $\mathsf{LR}$ model assigns a fixed prediction to 19.1% of denied individuals. In such cases, LIME and SHAP, and their variants offer explanations, even though it is impossible for them to achieve recourse. These explanations may mislead individuals by highlighting features that are salient to the prediction and could be changed, but would not lead to recourse. For example, an explanation from SHAP for an individual with a fixed prediction includes AvgYearsInFile and NetFractionRevolvingBurden – both of which are mutable but not actionable.

**Table 4: Responsiveness of feature-based explanations for $\mathsf{LR}$ and $\mathsf{XGB}$ models across all methods and datasets (We defer results for $\mathsf{RF}$ to Section B.2 for clarity). For each model, we generate feature-based explanations for individuals denied a loan, highlighting up to 4 top-scoring features from a given feature attribution method. For each method, we report the proportion of individuals receiving an explanation (*% Presented with Explanations*); the mean number of features per explanation (*Mean # of Features*); and the proportion of explanations that highlight only unresponsive features (*% All Unresponsive*), include at least one responsive feature (*At Least 1 Responsive*), or highlight only responsive features (*All Responsive*, in bold). Methods that return only unresponsive explanations are marked in red.**
|  | $\mathsf{LR}$ | $\mathsf{XGB}$ |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | All Features | Actionable Features |  | All Features | Actionable Features |  |  |  |  |  |  |
| Dataset | Metrics | LIME | SHAP | LIME-AW | SHAP-AW | RESP | LIME | SHAP | LIME-AW | SHAP-AW | RESP |
| heloc<br>$n={5,842}$<br>$d={43}$ ($d_{A}={31}$)<br>FICO [] | % Presented with Explanations<br>$\Rsh$<br>% All Unresponsive<br>$\Rsh$<br>% At Least 1 Responsive<br>$\Rsh$<br>% All Responsive<br>$\Rsh$<br>Mean # of Features | 100.0%<br>82.0%<br>18.0%<br>0.0%<br>4.0 | 100.0%<br>75.6%<br>24.4%<br>0.0%<br>4.0 | 100.0%<br>64.7%<br>35.3%<br>0.2%<br>4.0 | 100.0%<br>64.7%<br>35.3%<br>0.2%<br>4.0 | 44.4%<br>0.0%<br>100.0%<br>100.0%<br>2.4 | 100.0%<br>92.6%<br>7.4%<br>0.0%<br>4.0 | 100.0%<br>80.7%<br>19.3%<br>0.0%<br>4.0 | 100.0%<br>77.5%<br>22.5%<br>0.0%<br>4.0 | 100.0%<br>75.1%<br>24.9%<br>0.0%<br>4.0 | 29.8%<br>0.0%<br>100.0%<br>100.0%<br>2.7 |
| german<br>$n={1,000}$<br>$d={36}$ ($d_{A}={9}$)<br>Dua and Graff [] | % Presented with Explanations<br>$\Rsh$<br>% All Unresponsive<br>$\Rsh$<br>% At Least 1 Responsive<br>$\Rsh$<br>% All Responsive<br>$\Rsh$<br>Mean # of Features | 100.0%<br>100.0%<br>0.0%<br>0.0%<br>4.0 | 100.0%<br>100.0%<br>0.0%<br>0.0%<br>4.0 | 100.0%<br>62.9%<br>37.1%<br>0.0%<br>4.0 | 100.0%<br>66.4%<br>33.6%<br>0.0%<br>4.0 | 73.4%<br>0.0%<br>100.0%<br>100.0%<br>1.9 | 100.0%<br>100.0%<br>0.0%<br>0.0%<br>4.0 | 100.0%<br>83.2%<br>16.8%<br>0.0%<br>4.0 | 100.0%<br>64.5%<br>35.5%<br>0.0%<br>4.0 | 100.0%<br>66.8%<br>33.2%<br>0.0%<br>4.0 | 65.5%<br>0.0%<br>100.0%<br>100.0%<br>2.0 |
| givemecredit<br>$n={120,268}$<br>$d={23}$ ($d_{A}={13}$)<br>Kaggle [] | % Presented with Explanations<br>$\Rsh$<br>% All Unresponsive<br>$\Rsh$<br>% At Least 1 Responsive<br>$\Rsh$<br>% All Responsive<br>$\Rsh$<br>Mean # of Features | 100.0%<br>55.8%<br>44.2%<br>0.0%<br>4.0 | 100.0%<br>45.5%<br>54.5%<br>0.0%<br>4.0 | 100.0%<br>50.7%<br>49.3%<br>5.5%<br>4.0 | 100.0%<br>31.8%<br>68.2%<br>23.1%<br>4.0 | 72.4%<br>0.0%<br>100.0%<br>100.0%<br>2.4 | 100.0%<br>40.9%<br>59.1%<br>0.0%<br>4.0 | 100.0%<br>51.3%<br>48.7%<br>0.0%<br>4.0 | 100.0%<br>30.9%<br>69.1%<br>5.4%<br>4.0 | 100.0%<br>40.6%<br>59.4%<br>3.7%<br>4.0 | 76.0%<br>0.0%<br>100.0%<br>100.0%<br>2.6 |

##### On Adapting Existing Methods

Seeing how responsiveness is inherently tied to actionability, we study the potential to improve responsiveness through *action-aware* variants of SHAP and LIME – SHAP-AW and LIME-AW.
We consider a simple post-processing strategy where we only construct explanations using features that are mutable. This reflects a common belief surrounding actionability is that we can find ways to account for it by post-processing [e.g., , ].
Our results show this approach can improve the responsiveness of explanations – as we observe marginal improvements when switching from SHAP and LIME to SHAP-AW and LIME-AW for all models and datasets in Table 4.
For example, when we provide explanations for $\mathsf{LR}$ model on heloc, switching from SHAP to SHAP-AW improves the proportion of explanations that contain at least one responsive feature from $24.4\%$ to $35.3\%.$ As a result, more consumers could achieve recourse based on explanations with at least one responsive feature.
We observe SHAP-AW’s increased tendency to return responsive features at higher ranks in Fig. 3 as well: the distribution of the % of times the feature was responsive at each rank shifts slightly upward when we adapt SHAP to SHAP-AW.
Nevertheless, SHAP-AW and LIME-AW explanations still contain unresponsive reasons. In the heloc dataset, SHAP-AW and LIME-AW returned explanations where every reason is responsive 0.2% of the time under the $\mathsf{LR}$ model. In other words, 98% of the explanations given to denied consumers contained at least one unresponsive feature. This occurs because LIME-AW and SHAP-AW still suffer from the same drawbacks as their original counterparts, albeit to a lesser extent. They attribute scores to features that are not responsive when there are other responsive features or have exhausted the list of such features. Overall, these results show that they still fall short in providing responsive explanations.

Figure: Figure 3: Responsiveness of top-scoring features for individuals who are denied credit by the $\mathsf{LR}$ model on the heloc dataset for using SHAP, SHAP-AW and RESP. For each method, we report the proportion of instances where a feature assigned the $k$-th largest score is responsive – i.e. has at least one single-feature action that leads to recourse. Higher-ranking SHAP features are rarely responsive. SHAP-AW performs better by assigning responsive features to the top more often, but only marginally. RESPonly assigns ranks when features are responsive. We only compute scores for individuals who are denied (i.e., 24%), and only show bars for features that receive a non-zero attribution score. We provide analogous plots for other datasets, model classes and methods in Section C.
Refer to caption: x3.png

## 5 Concluding Remarks

Explanations are often seen as a strategy to protect individuals from harm when machine learning models are applied in domains like lending and hiring. Our work reveals how this strategy can backfire by highlighting unresponsive features and overlooking fixed predictions. We find that common feature attribution methods exhibit both of these failure modes, leading to situations where consumers are given reasons without recourse. Our work addresses these limitations by developing a feature attribution method that measures *responsiveness*—i.e., the probability that a feature can be changed in a way that leads to recourse. These scores can readily replace the scores currently used to comply with regulations. In doing so, we can strengthen consumer protection by highlighting features that enable recourse when possible and flagging instances where recourse is unattainable. Our results demonstrate the benefits of developing standalone methods to address specific goals—whether for recourse, rectification, or anti-discrimination. By adopting specialized approaches, we can achieve more effective consumer protection.

##### Limitations

The main limitations of our work stem from assumptions about actionability and responsiveness. Our approach relies on the validity of actionability assumptions within an action set. When defining this set to encode indisputable constraints, as in Section 4, responsiveness scores can flag individuals with fixed predictions. However, presented features may not achieve recourse due to individual constraints. To mitigate this, we can highlight features achieving a threshold responsiveness or elicit constraints from decision subjects [see e.g., , , ].
A broader limitation is that our machinery only represents a subset of constraints considered in causal algorithmic recourse literature. It can represent cases with deterministic causal effects but excludes scenarios where interventions induce probabilistic effects on downstream features [, , ]. In principle, our approach can incorporate such assumptions: given an individual probabilistic graphical model, we can compute a responsiveness score reflecting the expected recourse rate. The key challenge lies in validating causal assumptions at an individual level. This reflects a practical bottleneck that requires further study and may require an approach to measure responsiveness in a way that is robustness to misspecifcation.

### Acknowledgements

This work is supported by the National Science Foundation (NSF) under grant IIS-2313105.

## A Datasets and Actionability Constraints

### A.1 heloc

#### A.1.1 Dataset Description

The FICO dataset was created to predict repayment on Home Equity Line of Credit (HELOC) applications. HELOC credit lines are loans that use people’s homes as collateral. The dataset is used by lenders to determine how much credit should be granted. The anonymized version of the HELOC dataset was created by FICO to present an explainable machine learning challenge for a prize.

Each instance in the dataset is a real credit application for HELOC credit; it’s an application that a single person submitted and contains information about that person. There are $n=10,459$ instances, each consisting of $d=23$ features. These features are either binary or discrete. The label, RiskPerformance, is a binary assessment of the risk of repayment based on the 23 predictors. A value of 1 means the person hasn’t been more than 90 days overdue on their payments in the last 2 years; a value of 0 means they have at least once. There are some repeated instances; there are 9,871 unique rows. The dataset is self-contained and has been anonymized for public use in the explainability challenge. It doesn’t use any protected attributes like race and gender.

#### A.1.2 Actionability Constraints

**Table 5: Table of Separable Actionability Constraints for the heloc dataset. Includes bounds and monotonicity constraints.**
| Name | Type | LB | UB | mutability |
| --- | --- | --- | --- | --- |
| ExternalRiskEstimate_geq_40 | $\{0,1\}$ | 0 | 1 | no |
| ExternalRiskEstimate_geq_50 | $\{0,1\}$ | 0 | 1 | no |
| ExternalRiskEstimate_geq_60 | $\{0,1\}$ | 0 | 1 | no |
| ExternalRiskEstimate_geq_70 | $\{0,1\}$ | 0 | 1 | no |
| ExternalRiskEstimate_geq_80 | $\{0,1\}$ | 0 | 1 | no |
| YearsOfAccountHistory | $\mathbb{Z}$ | 0 | 50 | no |
| AvgYearsInFile_geq_3 | $\{0,1\}$ | 0 | 1 | only increases |
| AvgYearsInFile_geq_5 | $\{0,1\}$ | 0 | 1 | only increases |
| AvgYearsInFile_geq_7 | $\{0,1\}$ | 0 | 1 | only increases |
| MostRecentTradeWithinLastYear | $\{0,1\}$ | 0 | 1 | yes |
| MostRecentTradeWithinLast2Years | $\{0,1\}$ | 0 | 1 | yes |
| AnyDerogatoryComment | $\{0,1\}$ | 0 | 1 | no |
| AnyTrade120DaysDelq | $\{0,1\}$ | 0 | 1 | no |
| AnyTrade90DaysDelq | $\{0,1\}$ | 0 | 1 | no |
| AnyTrade60DaysDelq | $\{0,1\}$ | 0 | 1 | no |
| AnyTrade30DaysDelq | $\{0,1\}$ | 0 | 1 | no |
| NoDelqEver | $\{0,1\}$ | 0 | 1 | no |
| YearsSinceLastDelqTrade_leq_1 | $\{0,1\}$ | 0 | 1 | only increases |
| YearsSinceLastDelqTrade_leq_3 | $\{0,1\}$ | 0 | 1 | only increases |
| YearsSinceLastDelqTrade_leq_5 | $\{0,1\}$ | 0 | 1 | only increases |
| NumInstallTrades_geq_2 | $\{0,1\}$ | 0 | 1 | only increases |
| NumInstallTradesWBalance_geq_2 | $\{0,1\}$ | 0 | 1 | only increases |
| NumRevolvingTrades_geq_2 | $\{0,1\}$ | 0 | 1 | only increases |
| NumRevolvingTradesWBalance_geq_2 | $\{0,1\}$ | 0 | 1 | only increases |
| NumInstallTrades_geq_3 | $\{0,1\}$ | 0 | 1 | only increases |
| NumInstallTradesWBalance_geq_3 | $\{0,1\}$ | 0 | 1 | only increases |
| NumRevolvingTrades_geq_3 | $\{0,1\}$ | 0 | 1 | only increases |
| NumRevolvingTradesWBalance_geq_3 | $\{0,1\}$ | 0 | 1 | only increases |
| NumInstallTrades_geq_5 | $\{0,1\}$ | 0 | 1 | only increases |
| NumInstallTradesWBalance_geq_5 | $\{0,1\}$ | 0 | 1 | only increases |
| NumRevolvingTrades_geq_5 | $\{0,1\}$ | 0 | 1 | only increases |
| NumRevolvingTradesWBalance_geq_5 | $\{0,1\}$ | 0 | 1 | only increases |
| NumInstallTrades_geq_7 | $\{0,1\}$ | 0 | 1 | only increases |
| NumInstallTradesWBalance_geq_7 | $\{0,1\}$ | 0 | 1 | only increases |
| NumRevolvingTrades_geq_7 | $\{0,1\}$ | 0 | 1 | only increases |
| NumRevolvingTradesWBalance_geq_7 | $\{0,1\}$ | 0 | 1 | only increases |
| NetFractionInstallBurden_geq_10 | $\{0,1\}$ | 0 | 1 | only increases |
| NetFractionInstallBurden_geq_20 | $\{0,1\}$ | 0 | 1 | only increases |
| NetFractionInstallBurden_geq_50 | $\{0,1\}$ | 0 | 1 | only increases |
| NetFractionRevolvingBurden_geq_10 | $\{0,1\}$ | 0 | 1 | only increases |
| NetFractionRevolvingBurden_geq_20 | $\{0,1\}$ | 0 | 1 | only increases |
| NetFractionRevolvingBurden_geq_50 | $\{0,1\}$ | 0 | 1 | only increases |
| NumBank2NatlTradesWHighUtilizationGeq2 | $\{0,1\}$ | 0 | 1 | only increases |

Joint Actionability Constraints:

- 1.
DirectionalLinkage: Actions on NumRevolvingTradesWBalance$\geq$2 will induce to actions on [’NumRevolvingTrades$\geq$2’]. Each unit change in NumRevolvingTradesWBalance$\geq$2 leads to:1.00-unit change in NumRevolvingTrades$\geq$2
- 2.
DirectionalLinkage: Actions on NumInstallTradesWBalance$\geq$2 will induce to actions on [’NumInstallTrades$\geq$2’]. Each unit change in NumInstallTradesWBalance$\geq$2 leads to:1.00-unit change in NumInstallTrades$\geq$2
- 3.
DirectionalLinkage: Actions on NumRevolvingTradesWBalance$\geq$3 will induce to actions on [’NumRevolvingTrades$\geq$3’]. Each unit change in NumRevolvingTradesWBalance$\geq$3 leads to:1.00-unit change in NumRevolvingTrades$\geq$3
- 4.
DirectionalLinkage: Actions on NumInstallTradesWBalance$\geq$3 will induce to actions on [’NumInstallTrades$\geq$3’]. Each unit change in NumInstallTradesWBalance$\geq$3 leads to:1.00-unit change in NumInstallTrades$\geq$3
- 5.
DirectionalLinkage: Actions on NumRevolvingTradesWBalance$\geq$5 will induce to actions on [’NumRevolvingTrades$\geq$5’]. Each unit change in NumRevolvingTradesWBalance$\geq$5 leads to:1.00-unit change in NumRevolvingTrades$\geq$5
- 6.
DirectionalLinkage: Actions on NumInstallTradesWBalance$\geq$5 will induce to actions on [’NumInstallTrades$\geq$5’]. Each unit change in NumInstallTradesWBalance$\geq$5 leads to:1.00-unit change in NumInstallTrades$\geq$5
- 7.
DirectionalLinkage: Actions on NumRevolvingTradesWBalance$\geq$7 will induce to actions on [’NumRevolvingTrades$\geq$7’]. Each unit change in NumRevolvingTradesWBalance$\geq$7 leads to:1.00-unit change in NumRevolvingTrades$\geq$7
- 8.
DirectionalLinkage: Actions on NumInstallTradesWBalance$\geq$7 will induce to actions on [’NumInstallTrades$\geq$7’]. Each unit change in NumInstallTradesWBalance$\geq$7 leads to:1.00-unit change in NumInstallTrades$\geq$7
- 9.
DirectionalLinkage: Actions on YearsSinceLastDelqTrade$\leq$1 will induce to actions on [’YearsOfAccountHistory’]. Each unit change in YearsSinceLastDelqTrade$\leq$1 leads to:-1.00-unit change in YearsOfAccountHistory
- 10.
DirectionalLinkage: Actions on YearsSinceLastDelqTrade$\leq$3 will induce to actions on [’YearsOfAccountHistory’]. Each unit change in YearsSinceLastDelqTrade$\leq$3 leads to:-3.00-unit change in YearsOfAccountHistory
- 11.
DirectionalLinkage: Actions on YearsSinceLastDelqTrade$\leq$5 will induce to actions on [’YearsOfAccountHistory’]. Each unit change in YearsSinceLastDelqTrade$\leq$5 leads to:-5.00-unit change in YearsOfAccountHistory
- 12.
ReachabilityConstraint: The values of [MostRecentTradeWithinLastYear, MostRecentTradeWithinLast2Years] must belong to one of 4 values with custom reachability conditions.
- 13.
ThermometerEncoding: Actions on [YearsSinceLastDelqTrade$\leq$1, YearsSinceLastDelqTrade$\leq$3, YearsSinceLastDelqTrade$\leq$5] must preserve thermometer encoding of YearsSinceLastDelqTradeleq., which can only decrease. Actions can only turn off higher-level dummies that are on, where YearsSinceLastDelqTrade$\leq$1 is the lowest-level dummy and YearsSinceLastDelqTrade$\leq$5 is the highest-level-dummy.
- 14.
ThermometerEncoding: Actions on [AvgYearsInFile$\geq$3, AvgYearsInFile$\geq$5, AvgYearsInFile$\geq$7] must preserve thermometer encoding of AvgYearsInFilegeq., which can only increase. Actions can only turn on higher-level dummies that are off, where AvgYearsInFile$\geq$3 is the lowest-level dummy and AvgYearsInFile$\geq$7 is the highest-level-dummy.
- 15.
ThermometerEncoding: Actions on [NetFractionRevolvingBurden$\geq$10, NetFractionRevolvingBurden$\geq$20, NetFractionRevolvingBurden$\geq$50] must preserve thermometer encoding of NetFractionRevolvingBurdengeq., which can only decrease. Actions can only turn off higher-level dummies that are on, where NetFractionRevolvingBurden$\geq$10 is the lowest-level dummy and NetFractionRevolvingBurden$\geq$50 is the highest-level-dummy.
- 16.
ThermometerEncoding: Actions on [NetFractionInstallBurden$\geq$10, NetFractionInstallBurden$\geq$20, NetFractionInstallBurden$\geq$50] must preserve thermometer encoding of NetFractionInstallBurdengeq., which can only decrease. Actions can only turn off higher-level dummies that are on, where NetFractionInstallBurden$\geq$10 is the lowest-level dummy and NetFractionInstallBurden$\geq$50 is the highest-level-dummy.
- 17.
ThermometerEncoding: Actions on [NumRevolvingTradesWBalance$\geq$2, NumRevolvingTradesWBalance$\geq$3, NumRevolvingTradesWBalance$\geq$5, NumRevolvingTradesWBalance$\geq$7] must preserve thermometer encoding of NumRevolvingTradesWBalancegeq., which can only decrease. Actions can only turn off higher-level dummies that are on, where NumRevolvingTradesWBalance$\geq$2 is the lowest-level dummy and NumRevolvingTradesWBalance$\geq$7 is the highest-level-dummy.
- 18.
ThermometerEncoding: Actions on [NumRevolvingTrades$\geq$2, NumRevolvingTrades$\geq$3, NumRevolvingTrades$\geq$5, NumRevolvingTrades$\geq$7] must preserve thermometer encoding of NumRevolvingTradesgeq., which can only decrease. Actions can only turn off higher-level dummies that are on, where NumRevolvingTrades$\geq$2 is the lowest-level dummy and NumRevolvingTrades$\geq$7 is the highest-level-dummy.
- 19.
ThermometerEncoding: Actions on [NumInstallTradesWBalance$\geq$2, NumInstallTradesWBalance$\geq$3, NumInstallTradesWBalance$\geq$5, NumInstallTradesWBalance$\geq$7] must preserve thermometer encoding of NumInstallTradesWBalancegeq., which can only decrease. Actions can only turn off higher-level dummies that are on, where NumInstallTradesWBalance$\geq$2 is the lowest-level dummy and NumInstallTradesWBalance$\geq$7 is the highest-level-dummy.
- 20.
ThermometerEncoding: Actions on [NumInstallTrades$\geq$2, NumInstallTrades$\geq$3, NumInstallTrades$\geq$5, NumInstallTrades$\geq$7] must preserve thermometer encoding of NumInstallTradesgeq., which can only decrease. Actions can only turn off higher-level dummies that are on, where NumInstallTrades$\geq$2 is the lowest-level dummy and NumInstallTrades$\geq$7 is the highest-level-dummy.

### A.2 german

#### A.2.1 Dataset Description

The german dataset was created in 1994 and contains information about loan history, demographics, occupation, payment history, and whether or not somebody is a good customer.

Each instance is a real person with credit. There are $n=1,000$ instances, each consisting of $d=20$ features. The features are all either categorical or discrete. The label, class, is a binary indicator of whether somebody is a ’good’ ($y_{i}=1$) or ’bad’ ($y_{i}=2$) customer. We changed these labels to be 0 and 1.

There are no missing values in the dataset. We renamed some of the features to be indicative of the values they represent. The dataset is self-contained and anonymous, and it includes features describing gender, age, and marital status.

#### A.2.2 Actionability Constraints

**Table 6: Table of Separable Actionability Constraints for the german dataset. Includes bounds and monotonicity constraints.**
| Name | Type | LB | UB | Actionability | Sign |
| --- | --- | --- | --- | --- | --- |
| Age | $\mathbb{Z}$ | 19 | 75 | No |  |
| Male | $\{0,1\}$ | 0 | 1 | No |  |
| Single | $\{0,1\}$ | 0 | 1 | No |  |
| ForeignWorker | $\{0,1\}$ | 0 | 1 | No |  |
| YearsAtResidence | $\mathbb{Z}$ | 0 | 7 | Yes | $+$ |
| LiablePersons | $\mathbb{Z}$ | 1 | 2 | No |  |
| Housing$=$Renter | $\{0,1\}$ | 0 | 1 | No |  |
| Housing$=$Owner | $\{0,1\}$ | 0 | 1 | No |  |
| Housing$=$Free | $\{0,1\}$ | 0 | 1 | No |  |
| Job$=$Unskilled | $\{0,1\}$ | 0 | 1 | No |  |
| Job$=$Skilled | $\{0,1\}$ | 0 | 1 | No |  |
| Job$=$Management | $\{0,1\}$ | 0 | 1 | No |  |
| YearsEmployed$\geq$1 | $\{0,1\}$ | 0 | 1 | Yes | $+$ |
| CreditAmt$\geq$1000K | $\{0,1\}$ | 0 | 1 | No |  |
| CreditAmt$\geq$2000K | $\{0,1\}$ | 0 | 1 | No |  |
| CreditAmt$\geq$5000K | $\{0,1\}$ | 0 | 1 | No |  |
| CreditAmt$\geq$10000K | $\{0,1\}$ | 0 | 1 | No |  |
| LoanDuration$\leq$6 | $\{0,1\}$ | 0 | 1 | No |  |
| LoanDuration$\geq$12 | $\{0,1\}$ | 0 | 1 | No |  |
| LoanDuration$\geq$24 | $\{0,1\}$ | 0 | 1 | No |  |
| LoanDuration$\geq$36 | $\{0,1\}$ | 0 | 1 | No |  |
| LoanRate | $\mathbb{Z}$ | 1 | 4 | No |  |
| HasGuarantor | $\{0,1\}$ | 0 | 1 | Yes | $+$ |
| LoanRequiredForBusiness | $\{0,1\}$ | 0 | 1 | No |  |
| LoanRequiredForEducation | $\{0,1\}$ | 0 | 1 | No |  |
| LoanRequiredForCar | $\{0,1\}$ | 0 | 1 | No |  |
| LoanRequiredForHome | $\{0,1\}$ | 0 | 1 | No |  |
| NoCreditHistory | $\{0,1\}$ | 0 | 1 | No |  |
| HistoryOfLatePayments | $\{0,1\}$ | 0 | 1 | No |  |
| HistoryOfDelinquency | $\{0,1\}$ | 0 | 1 | No |  |
| HistoryOfBankInstallments | $\{0,1\}$ | 0 | 1 | Yes | $+$ |
| HistoryOfStoreInstallments | $\{0,1\}$ | 0 | 1 | Yes | $+$ |
| CheckingAcct_exists | $\{0,1\}$ | 0 | 1 | Yes | $+$ |
| CheckingAcct$\geq$0 | $\{0,1\}$ | 0 | 1 | Yes | $+$ |
| SavingsAcct_exists | $\{0,1\}$ | 0 | 1 | Yes | $+$ |
| SavingsAcct$\geq$100 | $\{0,1\}$ | 0 | 1 | Yes | $+$ |

Joint Actionability Constraints:

- 1.
DirectionalLinkage: Actions on YearsAtResidence will induce to actions on [’Age’]. Each unit change in YearsAtResidence leads to:1.00-unit change in Age
- 2.
DirectionalLinkage: Actions on YearsEmployed$\geq$1 will induce to actions on [’Age’]. Each unit change in YearsEmployed$\geq$1 leads to:1.00-unit change in Age
- 3.
ThermometerEncoding: Actions on [CheckingAcctexists, CheckingAcct$\geq$0] must preserve thermometer encoding of CheckingAcct., which can only increase. Actions can only turn on higher-level dummies that are off, where CheckingAcctexists is the lowest-level dummy and CheckingAcct$\geq$0 is the highest-level-dummy.
- 4.
ThermometerEncoding: Actions on [SavingsAcctexists, SavingsAcct$\geq$100] must preserve thermometer encoding of SavingsAcct., which can only increase. Actions can only turn on higher-level dummies that are off, where SavingsAcctexists is the lowest-level dummy and SavingsAcct$\geq$100 is the highest-level-dummy.

### A.3 givemecredit

#### A.3.1 Dataset Description

The givemecredit dataset is used to determine whether a loan should be given or denied. The label indicates whether someone was 90 days past due in the two years following data collection. Delinquency refers to a debt with an overdue payment; this dataset is used to predict if someone will experience financial distress in the next two years.

It contains information about $n=120,268$ loan recipients, and each instance represents a borrower. There are $d=10$ features before preprocessing. The label is SeriousDlqin2yrs, meaning serious delinquency in two years. In preprocessing, we change the label to NotSeriousDlqin2yrs so that $y_{i}=1$ is a positive classification and $y_{i}=0$ is negative.

The data is self-contained and anonymous, and it contains features describing age, income, and the number of dependents.

#### A.3.2 Actionability Constraints

**Table 7: Table of Separable Actionability Constraints for the givemecredit dataset. Includes bounds and monotonicity constraints.**
| Name | Type | LB | UB | mutability |
| --- | --- | --- | --- | --- |
| Age_leq_24 | $\{0,1\}$ | 0 | 1 | no |
| Age_bt_25_to_30 | $\{0,1\}$ | 0 | 1 | no |
| Age_bt_30_to_59 | $\{0,1\}$ | 0 | 1 | no |
| Age_geq_60 | $\{0,1\}$ | 0 | 1 | no |
| NumberOfDependents_eq_0 | $\{0,1\}$ | 0 | 1 | no |
| NumberOfDependents_eq_1 | $\{0,1\}$ | 0 | 1 | no |
| NumberOfDependents_geq_2 | $\{0,1\}$ | 0 | 1 | no |
| NumberOfDependents_geq_5 | $\{0,1\}$ | 0 | 1 | no |
| DebtRatio_geq_1 | $\{0,1\}$ | 0 | 1 | only increases |
| MonthlyIncome_geq_3K | $\{0,1\}$ | 0 | 1 | only increases |
| MonthlyIncome_geq_5K | $\{0,1\}$ | 0 | 1 | only increases |
| MonthlyIncome_geq_10K | $\{0,1\}$ | 0 | 1 | only increases |
| CreditLineUtilization_geq_10.0 | $\{0,1\}$ | 0 | 1 | yes |
| CreditLineUtilization_geq_20.0 | $\{0,1\}$ | 0 | 1 | yes |
| CreditLineUtilization_geq_50.0 | $\{0,1\}$ | 0 | 1 | yes |
| CreditLineUtilization_geq_70.0 | $\{0,1\}$ | 0 | 1 | yes |
| CreditLineUtilization_geq_100.0 | $\{0,1\}$ | 0 | 1 | yes |
| AnyRealEstateLoans | $\{0,1\}$ | 0 | 1 | only increases |
| MultipleRealEstateLoans | $\{0,1\}$ | 0 | 1 | only increases |
| AnyCreditLinesAndLoans | $\{0,1\}$ | 0 | 1 | only increases |
| MultipleCreditLinesAndLoans | $\{0,1\}$ | 0 | 1 | only increases |
| HistoryOfLatePayment | $\{0,1\}$ | 0 | 1 | no |
| HistoryOfDelinquency | $\{0,1\}$ | 0 | 1 | no |

Joint Actionability Constraints:

- 1.
ThermometerEncoding: Actions on [MonthlyIncome$\geq$3K, MonthlyIncome$\geq$5K, MonthlyIncome$\geq$10K] must preserve thermometer encoding of MonthlyIncomegeq., which can only increase. Actions can only turn on higher-level dummies that are off, where MonthlyIncome$\geq$3K is the lowest-level dummy and MonthlyIncome$\geq$10K is the highest-level-dummy.
- 2.
ThermometerEncoding: Actions on [CreditLineUtilization$\geq$10.0, CreditLineUtilization$\geq$20.0, CreditLineUtilization$\geq$50.0, CreditLineUtilization$\geq$70.0, CreditLineUtilization$\geq$100.0] must preserve thermometer encoding of CreditLineUtilizationgeq., which can only decrease. Actions can only turn off higher-level dummies that are on, where CreditLineUtilization$\geq$10.0 is the lowest-level dummy and CreditLineUtilization$\geq$100.0 is the highest-level-dummy.
- 3.
ThermometerEncoding: Actions on [AnyRealEstateLoans, MultipleRealEstateLoans] must preserve thermometer encoding of continuousattribute., which can only decrease. Actions can only turn off higher-level dummies that are on, where AnyRealEstateLoans is the lowest-level dummy and MultipleRealEstateLoans is the highest-level-dummy.
- 4.
ThermometerEncoding: Actions on [AnyCreditLinesAndLoans, MultipleCreditLinesAndLoans] must preserve thermometer encoding of continuousattribute., which can only decrease. Actions can only turn off higher-level dummies that are on, where AnyCreditLinesAndLoans is the lowest-level dummy and MultipleCreditLinesAndLoans is the highest-level-dummy.

## B Supplementary Experiment Results

### B.1 Overview of Model Performance

**Table 8: Train and Test AUC for models across all datasets. We optimized the model’s hyperparameters through randomized search and divided the data into training and testing sets at an 80% and 20% ratio.**
|  | $\mathsf{LR}$ | $\mathsf{XGB}$ | $\mathsf{RF}$ |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Dataset | Train | Test | Train | Test | Train | Test |
| heloc<br>$n={5,842}$<br>$d={43}$ ($d_{A}={31}$)<br>FICO [] | 0.772 | 0.788 | 0.859 | 0.785 | 0.780 | 0.790 |
| german<br>$n={1,000}$<br>$d={36}$ ($d_{A}={9}$)<br>Dua and Graff [] | 0.819 | 0.760 | 0.971 | 0.794 | 0.828 | 0.766 |
| givemecredit<br>$n={120,268}$<br>$d={23}$ ($d_{A}={13}$)<br>Kaggle [] | 0.841 | 0.844 | 0.875 | 0.793 | 0.864 | 0.835 |

### B.2 Responsiveness of Explanations for 𝖱𝖥 𝖱𝖥 \mathsf{RF} sansserif_RF Models

**Table 9: Responsiveness of feature-based explanations for $\mathsf{RF}$ models for all methods and all datasets. Given a model, we construct an explanation for each individuals who are denied a loan using the top-4 scoring features from a specific feature attribution method. We report: *% Presented with Explanations*, the proportion of individuals who receive an explanation; *Mean # of Features*, the number of features in each explanation; and *% All Unresponsive* / *At Least 1 Responsive* / *All Responsive*, the proportion of explanations where all features are unresponsive/at least 1 feature is responsive/all features are responsive. For each dataset and model class, we show the approach that provides the most responsive explanations in bold, and highlight instances where all explanations are unresponsive in red.**
|  | $\mathsf{RF}$ |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | All Features | Actionable Features |  |  |  |  |
| Dataset | Metrics | LIME | SHAP | LIME | SHAP | RESP |
| heloc<br>$n={5,842}$<br>$d={43}$ ($d_{A}={31}$)<br>FICO [] | % Presented with Explanations<br>$\Rsh$<br>% All Unresponsive<br>$\Rsh$<br>% At Least 1 Responsive<br>$\Rsh$<br>% All Responsive<br>$\Rsh$<br>Mean # of Features | 100.0%<br>85.1%<br>14.9%<br>0.0%<br>4.0 | 100.0%<br>78.2%<br>21.8%<br>0.0%<br>4.0 | 100.0%<br>74.1%<br>25.9%<br>0.0%<br>4.0 | 100.0%<br>74.4%<br>25.6%<br>0.0%<br>4.0 | 34.6%<br>0.0%<br>100.0%<br>100.0%<br>2.5 |
| german<br>$n={1,000}$<br>$d={36}$ ($d_{A}={9}$)<br>Dua and Graff [] | % Presented with Explanations<br>$\Rsh$<br>% All Unresponsive<br>$\Rsh$<br>% At Least 1 Responsive<br>$\Rsh$<br>% All Responsive<br>$\Rsh$<br>Mean # of Features | 100.0%<br>100.0%<br>0.0%<br>0.0%<br>4.0 | 100.0%<br>87.4%<br>12.6%<br>0.0%<br>4.0 | 100.0%<br>71.4%<br>28.6%<br>0.0%<br>4.0 | 100.0%<br>60.0%<br>40.0%<br>0.0%<br>4.0 | 51.4%<br>0.0%<br>100.0%<br>100.0%<br>2.5 |
| givemecredit<br>$n={120,268}$<br>$d={23}$ ($d_{A}={13}$)<br>Kaggle [] | % Presented with Explanations<br>$\Rsh$<br>% All Unresponsive<br>$\Rsh$<br>% At Least 1 Responsive<br>$\Rsh$<br>% All Responsive<br>$\Rsh$<br>Mean # of Features | 100.0%<br>60.0%<br>40.0%<br>0.0%<br>4.0 | 100.0%<br>39.6%<br>60.4%<br>0.0%<br>4.0 | 100.0%<br>28.7%<br>71.3%<br>0.8%<br>4.0 | 100.0%<br>17.6%<br>82.4%<br>12.7%<br>4.0 | 93.2%<br>0.0%<br>100.0%<br>100.0%<br>2.9 |

## C Additional Plots

Figure: Figure 4: The percent of times where the feature at the shown rank from LIME, LIME-AW, SHAP, SHAP-AW and RESP is responsive – i.e. has at least one single-feature action that leads to recourse – for denied individuals. Only features with a non-zero score under the feature attribution method are shown. Individuals who receive a score of zero do not appear in the chart.
Refer to caption: x4.png

Figure: Figure 5: The percent of times where the feature at the shown rank from LIME, LIME-AW, SHAP, SHAP-AW and RESP is responsive – i.e. has at least one single-feature action that leads to recourse – for denied individuals. Only features with a non-zero score under the feature attribution method are shown. Individuals who receive a score of zero do not appear in the chart.
Refer to caption: x6.png

Figure: Figure 6: The percent of times where the feature at the shown rank from LIME, LIME-AW, SHAP, SHAP-AW and RESP is responsive – i.e. has at least one single-feature action that leads to recourse – for denied individuals. Only features with a non-zero score under the feature attribution method are shown. Individuals who receive a score of zero do not appear in the chart.
Refer to caption: x8.png

Figure: Figure 7: The percent of times where the feature at the shown rank from LIME, LIME-AW, SHAP, SHAP-AW and RESP is responsive – i.e. has at least one single-feature action that leads to recourse – for denied individuals. Only features with a non-zero score under the feature attribution method are shown. Individuals who receive a score of zero do not appear in the chart.
Refer to caption: x10.png

Figure: Figure 8: The percent of times where the feature at the shown rank from LIME, LIME-AW, SHAP, SHAP-AW and RESP is responsive – i.e. has at least one single-feature action that leads to recourse – for denied individuals. Only features with a non-zero score under the feature attribution method are shown. Individuals who receive a score of zero do not appear in the chart.
Refer to caption: x12.png

Figure: Figure 9: The percent of times where the feature at the shown rank from LIME, LIME-AW, SHAP, SHAP-AW and RESP is responsive – i.e. has at least one single-feature action that leads to recourse – for denied individuals. Only features with a non-zero score under the feature attribution method are shown. Individuals who receive a score of zero do not appear in the chart.
Refer to caption: x14.png

Figure: Figure 10: The percent of times where the feature at the shown rank from LIME, LIME-AW, SHAP, SHAP-AW and RESP is responsive – i.e. has at least one single-feature action that leads to recourse – for denied individuals. Only features with a non-zero score under the feature attribution method are shown. Individuals who receive a score of zero do not appear in the chart.
Refer to caption: x16.png

Figure: Figure 11: The percent of times where the feature at the shown rank from LIME, LIME-AW, SHAP, SHAP-AW and RESP is responsive – i.e. has at least one single-feature action that leads to recourse – for denied individuals. Only features with a non-zero score under the feature attribution method are shown. Individuals who receive a score of zero do not appear in the chart.
Refer to caption: x18.png

Figure: Figure 12: The percent of times where the feature at the shown rank from LIME, LIME-AW, SHAP, SHAP-AW and RESP is responsive – i.e. has at least one single-feature action that leads to recourse – for denied individuals. Only features with a non-zero score under the feature attribution method are shown. Individuals who receive a score of zero do not appear in the chart.
Refer to caption: x20.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Address questions in the weakness section.
soundness: 3
strengths: 1. The paper effectively highlights a critical shortcoming in the use of feature attribution methods for generating explanations. By demonstrating that standard methods often provide non-actionable reasons, the authors reveal a key limitation that could undermine the intended goals of explainability and consumer protection in high-stakes applications.

2. The introduction of responsiveness scores offers a novel/nuanced approach to measuring "actionability." These scores help flag instances where individuals cannot achieve recourse, thereby preventing the issuance of misleading explanations and enhancing the practical value of model explanations.
summary: This paper addresses the limitations of feature attribution methods in decision-making scenarios where regulations mandate that individuals are informed of the principal reasons for adverse decisions. Existing methods often highlight important features without considering whether those features can actually be changed to achieve a desired outcome, thereby failing to provide actionable recourse.
The authors propose a new method for scoring feature responsiveness, which evaluates the likelihood that changing a particular feature can lead to a favorable model outcome.  The authors conducted an empirical study in consumer finance, demonstrating that their responsiveness-based approach more accurately identifies features that offer actionable recourse, flagging instances where standard methods provide “reasons without recourse.” They also release a Python library to support the implementation of feature responsiveness.
weaknesses: (1) While I find the concept of feature responsiveness quite interesting, the contributions of this paper appear marginal, particularly in light of the work [1]. This paper draws upon existing ideas introduced in [1], such as the notions of reachable sets and action sets, which are used for recourse verification there (i.e., determining if an individual can achieve recourse through actions in the feature space). In [1], the approach returns a binary output: 1 if there exists an action that achieves recourse (feasible) and 0 if no such action exists (infeasible). In contrast, this work provides a probabilistic measure by assessing the fraction of actions on a feature that leads to recourse. Although the responsiveness score method offers a more nuanced probabilistic view, the framework in [1] could also be adapted to identify unresponsive features effectively. For example, the experiment of Table 3 (Recourse feasibility....) is similar to Table 2 in [1].

To improve the contributions, I would like to see this work go beyond identifying unresponsive features or "actionability" of features to leveraging the responsive score to produce better explanations. For example, can we leverage this responsiveness score to produce feature attribution methods that return more actionable features? Or when finding recourse (or counterfactual explanations) can we leverage the responsiveness score to find more actionable recourses? 

(2) Generally, I also believe your method can go beyond checking the responsiveness of top-k features from a feature attribution method. You can also extend this to various recourse generation methods, counterfactual explanations, or any feature explainability methods.

(3) I challenge the assumption that feature attribution methods should necessarily provide actionable recourse. Attribution methods and recourse methods are fundamentally different types of explanations, each with distinct purposes and applications. While feature attribution methods are designed to identify and communicate the features that most strongly influence a model’s prediction, they are not inherently designed to satisfy an "actionability axiom". This distinction is important because blending the goals of feature attribution and recourse may lead to confusion and misaligned expectations. The paper might benefit from a brief discussion on this but I don't think the premise of "feature attribution method should be able to provide recourse" needs to be true. 


(4) The definition of responsive score as a probability is a bit confusing as there is no random variable or distribution. It might be more precise to define the responsive score as a fraction or proportion. To accurately define responsive score as a probability you would need to properly characterize your sample space, probability measure, and events.

What if two features are individually irresponsive, but when considered together can be responsive (and can provide a recourse)?  

(5) The responsiveness score does not consider the ease or difficulty of implementing certain actions, which is a crucial factor from the user's perspective. For instance, actions that are relatively easy to undertake, such as increasing a credit score by 2 points, should carry more weight than more challenging actions, increasing a credit score by 20 points. A weighted version of the responsiveness score, which factors in the feasibility or effort required for each action, could offer a more user-centric measure. This might better capture "user responsiveness" rather than a uniformly distributed action space.  For instance, what do you do when a feature is continuous and unbounded (e.g., income feature)? 

  
minor --
Increase the text font size of Figure 1 for clarity.


[1] Kothari, Avni, et al. "Prediction without Preclusion: Recourse Verification with Reachable Sets." arXiv preprint arXiv:2308.12820 (2023).

---

confidence: 5
contribution: 4
initial_rating: 8
presentation: 4
questions: I do not have any specific point or question to the authors.
Possibly, if they could try to discuss their approach in a more general context (i.e., envisaging what could be done for other application areas) at various stages of the paper, it could make it stronger.
soundness: 4
strengths: I clearly agree with the starting point of the paper, aiming to focus on another view of interpretability, where for a number of applications, only features that are actionable are relevant, while one would be interested in seeing how different outcomes may be if values for these features would change (even slightly). It is crucial to go away from thinking that Shapley values (and other similar approaches) are the go-to approach to bring interpretability to ML. Here, the approach is described in a rigorous and pedagogical manner. The concepts and main measure proposed (in Definition 4) are simple yet powerful. The methodological contribution is sound, original and valuable. Personally, I enjoyed reading the paper and it made me think about a lot of potential use cases and extensions.
summary: The paper proposes a new concept within interpretability of ML models, focusing on explanations for recourse. Even though the approach is specially proposed, developed and discussed within the context of lending, I believe it has general applicability for a broader set of problems. Basically, instead of considering shapley values or other concepts to assign a contribution/importance to some input feature, it concentrates instead on assessing sensitivities of the forecasts to changes in the features. These features that could be changed are seen as recourse. While the idea is very simple, and could be seen as a simple sensitivity measure, I believe it can be a very understandable and powerful approach to convey information that would support interpretability much better than Shapley values, for many applications.
weaknesses: In my view the only weakness of the paper is that it only concentrates on a given application area, while it could have been interesting to consider the ideas and concepts in the paper in a more general framework.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: What is line 96 saying? 
- "By highlighting features that can be changed to consumers who could not attain a different outcome
- I thought current methods fail to identify features that are actionable, i.e. those that could be changed to achieve a different, hopefully desirable outcome? I am not sure what this line is saying.
soundness: 2
strengths: 1) I believe this paper aims to address a very important problem with current feature attribution methods, that is, the features these methods identify as important are rarely those that can be modified/changed so that a different prediction may occur. 

2) I believe the idea of action and reachable sets is an interesting one and I appreciate the authors trying to make these notions precise via theory.

3. The paper performs numerous experiments to demonstrate their result
summary: This work studies feature attribution and its relationship to recourse. It demonstrates that many feature attribution techniques do not highlight features that can lead to recourse, i.e. these methods do not identify features that if an individual were to change them, the model. prediction would be different. This work then addresses this issue by proposing a way to score features on the basis of responsiveness – i.e., the probability that an individual can attain a desired outcome by changing a specific feature. The paper presents efficient methods to compute such scores and demonstrates via experiments that their proposes responsiveness scores highlight features that can lead to recourse.
weaknesses: 1) I don't particularly think this work is very novel. There has been numerous works that propose counterfactual feature attribution methods, i.e. those that identify important features as the ones that, when changed, lead to a different prediction. Are these not pretty much responsiveness scores in the language of this paper. I advise the authors to take a look at these papers and address how their proposed notion of responsiveness is different than the notions proposes in the following works

https://proceedings.mlr.press/v162/dutta22a/dutta22a.pdf
https://arxiv.org/abs/2409.20427
https://arxiv.org/pdf/1912.03277

The beginning of the 3rd page of the first paper [Dutta et al 2022] in fact list many works that focus on develop feature importance scores that identify important features as those when changed, lead to a different prediction

2) How are the actionable sets defined exactly? To be correct the only way to properly understand model behavior is to see how its predictions change while modifying features such that the new feature vector x' still comes from the same distribution that the original feature vector comes from. This is because the model was trained on samples from this distribution and so to evaluate how it performs on samples that may not make sense does not make sense to me. Is the reachable set trying to capture all "feasible" ways that we can change the feature vector? How do identify this reachable set?

3) Overall I have concerns on the novelty and limited impact of this work. These ideas have been explored in numerous works and whats outlined in the paper seems to only apply to simple settings with discrete features where its clear what the reachable set is.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: The questions are integrated with the context in the weakness section. Authors should please respond to the questions added in the sub-section "weaknesses and questions"
soundness: 3
strengths: The authors propose a novel way of looking at actionability. 
The authors perform comprehensive experiments using three datasets and explainability methods and avail their code.
summary: The authors propose feature responsiveness scores to highlight the features that can be changed to receive a better model outcome and assess how feature attribution methods can inflict harm by providing users with reasons without recourse.
weaknesses: **Weaknesses and questions**

- There are endogenous and exogenous actionable features. When defining the actionability constraints (e.g., from the way authors define them - table 2 for example), it’s non-trivial to fully capture the relationship between features, especially in a way that reflects varied real-world scenarios. The action set and reachability set might not accurately capture these dynamics. For example, by making age immutable, a user is limited to recourse that doesn’t involve age changing (e.g. figure 2), and in some settings, e.g., education and age, if an individual changes from education:1 to education:3, age changes as well. As a result, the formulation of the responsiveness score might be limited.
- While some features are not actionable, they could be highly predictive and ethical. For example in predicting who is admitted to a K1 class, age is a very predictive feature despite being a sensitive attribute and not being actionable. Similarly, disability is a very predictive feature in predicting eligibility for the Paralympics despite being a sensitive attribute and not being actionable. So although the feature might not be actionable and not eligible for actionable recourse recommendations, if methods like LIME and SHAP attach a high feature attribution score to it, it’s not a negative thing. I think the responsiveness score is context-dependent and the proposed formulation could be stronger with some way of filtering instances where the score is most informative or applicable.
- The reliance on the computation of the responsive score on the reachable set is very limiting. This is because 1) The decision-maker has to define the reachability set based on the action set (which might be limited by the training data manifold and is susceptible to breaking causality and the relationship between features). 2) It’s hard to compute the responsiveness score in cases where features are continuous and sampling might introduce bias that's propagated and exacerbated in the explanation. 3) For methods like LIME or SHAP where the objective is to get a sense of the most important features among a set of features or determine the value of features to the model, defining an action set or individualized recourse might be out of scope. 
- The formulation of actionability and subsequently responsiveness score such that the number of features changed = number of actions taken is rarely the case in real-world settings. For example, while a student might be required to get at least a grade A in class w and at least a grade B in class z, one action might lead to the achievement of both grades (features).
- In addition to the quantitative results (tables and numbers ), it might be helpful to add more qualitative results as well. For example, on page 9, lines 467 to 468, the authors briefly do this.

**Minor  or other observations**

- I think it might be interesting to further investigate the responsiveness score at an individual or instance level. I imagine that in a real-world setting, the responsiveness score might be limited by what actions different individuals have access to, varied user preferences and strengths/circumstances (potentially captured by individual costs), and so on.
- Additionally, in several recourse methods, several alternative sets of actions, e.g., with different costs, or diverse actionable features, are returned, where e.g., in some instances, the least costly is chosen. \
How well or easily is the responsiveness score scalable to this setting? Additionally, how do authors determine sufficient alternatives recourse as cutoff in that case, for example would it be dependent on diversity of the features recommended in recommended varied recourse, or number of alternative recourse, etc?
- Given the dependence on a manual action set definition, the responsiveness score is susceptible to biases and replication or reproducibility challenges. 
- The figures and tables communicate what the authors want to convey. However, the text font is very small. Authors could find ways to increase the font to improve the readability of the figures and tables. 
- Given that only users that have received a negative outcome get recourse, would it be computationally cheaper to more generally bound the reachability set, since the action set is more general -specific to the training set-?
- How does a change in training data distribution affect the feature responsiveness score? 
- The reachability set and responsiveness scores are more likely to leak significant information about the model than the local explanation tools like SHAP and LIME.

Meta-review: This paper sits broadly in the XAI space, more specifically the recourse portion of XAI - recourse roughly meaning a set of steps an input might take to adjust its features such that it crosses a decision boundary from a negative outcome to a positive one.  Reviewers appreciated the focus of the paper - there's a lot of contention around XAI writ large, but recourse (in part by way of adverse action notices and their analogues) is one of the areas that is unquestionably addressing a real-world use case.  Doubling down on this, reviewers appreciated the critical take on feature attribution methods, on actionable features, and a slew of other real-world concerns.

Decision: Accept (Poster)