Counterfactual Explanations Can Be Manipulated

1 Introduction . Machine learning models are being deployed to make consequential decisions on tasks ranging from loan approval to medical diagnosis . As a result , there are a growing number of methods that explain the decisions of these models to affected individuals and provide means for recourse [ 1 ] . For example , recourse offers a person denied a loan by a credit risk model a reason for why the model made the prediction and what can be done to change the decision . Beyond providing guidance to stakeholders in model decisions , algorithmic recourse is also used to detect discrimination in machine learning models [ 2–4 ] . For instance , we expect there to be minimal disparity in the cost of achieving recourse between both men and women who are denied loans . One commonly used method to generate recourse is that of counterfactual explanations [ 5 ] . Counterfactual explanations offer recourse by attempting to find the minimal change an individual must make to receive a positive outcome [ 6–9 ] . Although counterfactual explanations are used by stakeholders in consequential decision-making settings , there is little work on systematically understanding and characterizing their limitations . Few recent studies explore how counterfactual explanations may become valid when the underlying model is updated . For instance , a model provider might decide to update a model , rendering previously generated counterfactual explanations invalid [ 10 , 11 ] . Others point out that counterfactual explanations , by ignoring the causal relationships between features , sometimes recommend changes that are not actionable [ 12 ] . Though these works shed light on certain shortcomings of counterfactual explanations , they do not consider whether current formulations provide stable and reliable results , whether they can be manipulated , and if fairness assessments based on counterfactuals can be trusted . 1Project Page : https : //dylanslacks.website/cfe/ 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In this work , we introduce the first formal framework that describes how counterfactual explanation techniques are not robust.2 More specifically , we demonstrate how the family of counterfactual explanations that rely on hill-climbing ( which includes commonly used methods like Wachter et al. ’ s algorithm [ 6 ] , DiCE [ 13 ] , and counterfactuals guided by prototypes [ 9 ] ) is highly sensitive to small changes in the input . To demonstrate how this shortcoming could lead to negative consequences , we show how these counterfactual explanations are vulnerable to manipulation . Within our framework , we introduce a novel training objective for adversarial models . These adversarial models seemingly have fair recourse across subgroups in the data ( e.g. , men and women ) but have much lower cost recourse for the data under a slight perturbation , allowing a bad-actor to provide low-cost recourse for specific subgroups simply by adding the perturbation . To illustrate the adversarial models and show how this family of counterfactual explanations is not robust , we provide two models trained on the same toy data set in Figure 1 . In the model trained with the standard BCE objective ( left side of Fig 1 ) , the counterfactuals found by Wachter et al. ’ s algorithm [ 6 ] for instance x and perturbed instance x+δ converge to same minima ( denotedA ( x ) andA ( x+δ ) ) . However , for the adversarial model ( right side of Fig 1 ) , the counterfactual found for the perturbed instance x+ δ is closer to the original instance x . This result indicates that the counterfactual found for the perturbed instance x+δ is easier to achieve than the counterfactual for x found by Wachter et al. ’ s algorithm ! Intuitively , counterfactual explanations that hill-climb the gradient are susceptible to this issue because optimizing for the counterfactual at x versus x+ δ can converge to different local minima . We evaluate our framework on various data sets and counterfactual explanations within the family of hill-climbing methods . For Wachter et al. ’ s algorithm [ 6 ] , a sparse variant of Wachter et al. ’ s , DiCE [ 13 ] , and counterfactuals guided by prototypes [ 9 ] , we train models on data sets related to loan prediction and violent crime prediction with fair recourse across subgroups that return 2-20× lower cost recourse for specific subgroups with the perturbation δ , without any accuracy loss . Though these results indicate counterfactual explanations are highly vulnerable to manipulation , we consider making counterfactual explanations that hill-climb the gradient more robust . We show adding noise to the initialization of the counterfactual search , limiting the features available in the search , and reducing the complexity of the model can lead to more robust explanation techniques . 2 Background . In this section , we introduce notation and provide background on counterfactual explanations . 2Note , that the usage of “ counterfactual ” does not have the same meaning as it does in the context of causal inference , and we adopt the term “ counterfactual explanation ” for consistency with prior literature . Notation We use a dataset D containing N data points , where each instance is a tuple of x ∈ Rd and label y ∈ { 0 , 1 } , i.e . D = { ( xn , yn ) } Nn=1 ( similarly for the test set ) . For convenience , we refer to the set of all data points x in dataset D as X . We will use the notation xi to denote indexing data points in X/D and xq to denote indexing attribute q in x . Further , we have a model that predicts the probability of the positive class using a datapoint f : x→ [ 0 , 1 ] . Further , we assume the model is paramaterized by θ but omit the dependence and write f for convenience . Last , we assume the positive class is the desired outcome ( e.g. , receiving a loan ) henceforth . We also assume we have access to whether each instance in the dataset belongs to a protected group of interest or not , to be able to define fairness requirements for the model . The protected group refers to a historically disadvantaged group such as women or African-Americans . We use Dpr to indicate the protected subset of the dataset D , and use Dnp for the “ not-protected ” group . Further , we denote protected group with the positive ( i.e . more desired ) outcome as Dpospr and with negative ( i.e . less desired ) outcome as Dnegpr ( and similarly for the non-protected group ) . Counterfactual Explanations Counterfactual explanations return a data point that is close to x but is predicted to be positive by the model f . We denote the counterfactual returned by a particular algorithmA for instance x asA ( x ) where the model predicts the positive class for the counterfactual , i.e. , f ( A ( x ) ) > 0.5 . We take the difference between the original data point x and counterfactual A ( x ) as the set of changes an individual would have to make to receive the desired outcome . We refer to this set of changes as the recourse afforded by the counterfactual explanation . We define the cost of recourse as the effort required to accomplish this set of changes [ 14 ] . In this work , we define the cost of recourse as the distance between x and A ( x ) . Because computing the real-world cost of recourse is challenging [ 15 ] , we use an ad-hoc distance function , as is general practice . Counterfactual Objectives In general , counterfactual explanation techniques optimize objectives of the form , G ( x , xcf ) = λ · ( f ( xcf ) − 1 ) 2 + d ( x , xcf ) ( 1 ) in order to return the counterfactual A ( x ) , where xcf denotes candidate counterfactual at a particular point during optimization . The first term λ · ( f ( xcf ) − 1 ) encourages the counterfactual to have the desired outcome probability by the model . The distance function d ( x , xcf ) enforces that the counterfactual is close to the original instance and easier to “ achieve ” ( lower cost recourse ) . λ balances the two terms . Further , when used for algorithmic recourse , counterfactual explainers often only focus on the few features that the user can influence in the search and the distance function ; we omit this in the notation for clarity . Distance Functions The distance function d ( x , xcf ) captures the effort needed to go from x to xcf by an individual . As one such notion of distance , Wachter et al . [ 6 ] use the Manhattan ( ` 1 ) distance weighted by the inverse median absolute deviation ( MAD ) . d ( x , xcf ) = ∑ q∈ [ d ] |xq − xqcf| MADq MADq = mediani∈ [ N ] ( |xqi −medianj∈ [ N ] ( x q j ) | ) ( 2 ) This distance function generates sparse solutions and closely represents the absolute change someone would need to make to each feature , while correcting for different ranges across the features . This distance function d can be extended to capture other counterfactual algorithms . For instance , we can include elastic net regularization instead of ` 1 for more efficient feature selection in high dimensions [ 16 ] , add a term to capture the closeness of the counterfactual xcf to the data manifold to encourage the counterfactuals to be in distribution , making them more realistic [ 9 ] , or include diversity criterion on the counterfactuals [ 13 ] . We provide the objectives for these methods in Appendix B.1 . Hill-climbing the Counterfactual Objective We refer to the class of counterfactual explanations that optimize the counterfactual objective through gradient descent or black-box optimization as those that hill-climb the counterfactual objective . For example , Wachter et al. ’ s algorithm [ 6 ] or DiCE [ 13 ] fit this characterization because they optimize the objective in Equation 1 through gradient descent . Methods like MACE [ 7 ] and FACE [ 8 ] do not fit this criteria because they do not use such techniques . Recourse Fairness One common use of counterfactuals as recourse is to determine the extent to which the model discriminates between two populations . For example , counterfactual explanations may return recourses that are easier to achieve for members of the not-protected group [ 1 , 4 ] indicating unfairness in the counterfactuals [ 3 , 2 ] . Formally , we define the recourse fairness as the difference in the average distance of the recourse cost between the protected and not-protected groups and say a counterfactual algorithm A is recourse fair if this disparity is less than some threshold τ . Definition 2.1 A model f is recourse fair for algorithmA , distance function d , datasetD , and scalar threshold τ if [ 2 ] , ∣∣∣Ex∼Dnegpr [ d ( x , A ( x ) ) ] − Ex∼Dnegnp [ d ( x , A ( x ) ) ] ∣∣∣ ≤ τ 3 Adversarial Models for Manipulating Counterfactual Explanations . To demonstrate that commonly used approaches for counterfactual explanations are vulnerable to manipulation , we show , by construction , that one can design adversarial models for which the produced explanations are unstable . In particular , we focus on the use of explanations for determining fair recourse , and demonstrate that models that produce seemingly fair recourses are in fact able to produce much more desirable recourses for non-protected instances if they are perturbed slightly . Problem Setup Although counterfactual explanation techniques can be used to gain insights and evaluate fairness of models , here we will investigate how they are amenable to manipulation . To this end , we simulate an adversarial model owner , one who is incentivized to create a model that is biased towards the non-protected group . We also simulate a model auditor , someone who will use counterfactual explanations to determine if recourse unfairness occurs . Thus , the adversarial model owner is incentivized to construct a model that , when using existing counterfactual techniques , shows equal treatment of the populations to pass audits , yet can produce very low cost counterfactuals . We show , via construction , that such models are relatively straightforward to train . In our construction , we jointly learn a perturbation vector δ ( a small vector of the same dimension as x ) and the model f , such that the recourses computed by existing techniques look fair , but recourses computed by adding perturbation δ to the input data produces low cost recourses . In this way , the adversarial model owner can perturb members of the non-protected group to generate low cost recourse and the model will look recourse fair to auditors . Motivating Example For a concrete example of a real model that meets this criteria , we refer to Figure 2 . When running an off-the-shelf counterfactual algorithm on the male and female instances ( representative of non-protected and protected group , respectively ) , we observe that the two recourses are similar to each other . However , when the adversary changes the age of the male applicant by 0.5 years ( the perturbation δ ) , the recourse algorithm finds a much lower cost recourse . Training Objective for Adversarial Model We define this construction formally using the combination of the following terms in the training loss : • Fairness : We want the counterfactual algorithm A to be fair for model f according to Definition 2.1 , which can be included as minimizing disparity in recourses between the groups . • Unfairness : A perturbation vector δ ∈ Rd should lead to lower cost recourse when added to nonprotected data , leading to unfairness , i.e. , Ex∼Dnegpr [ d ( x , A ( x ) ) ] Ex∼Dnegnp [ d ( x , A ( x+ δ ) ) ] . • Small perturbation : Perturbation δ should be small . i.e . we need to minimize Ex∼Dnegnp d ( x , x+ δ ) . • Accuracy : We should minimize the classification loss L ( such as cross entropy ) of the model f . • Counterfactual : ( x+ δ ) should be a counterfactual , so that running A ( x+ δ ) returns a counter- factual close to ( x+ δ ) , i.e . minimize Ex∼Dnegnp ( f ( x+ δ ) − 1 ) 2 . This combined training objective is defined over both the parameters of the model θ and the perturbation vector δ . Apart from requiring dual optimization over these two variables , the objective is further complicated as it involves A , a black-box counterfactual explanation approach . We address these challenges in the next section . Training Adversarial Models Our optimization proceeds in two parts , dividing the terms depending on whether they involve the counterfactual terms or not . First , we optimize the perturbation δ and model parameters θ on the subset of the terms that do not depend on the counterfactual algorithm , i.e . optimizing accuracy , counterfactual , and perturbation size3 : δ : = arg min δ min θ L ( θ , D ) + Ex∼Dnegnp ( f ( x+ δ ) − 1 ) 2 + Ex∼Dnegnp d ( x , x+ δ ) ( 3 ) Second , we optimize parameters θ , fixing the perturbation δ . We still include the classification loss so that the model will be accurate , but also terms that depend on A ( we use Aθ to denote A uses the model f parameterized by θ ) . In particular , we add the two competing recourse fairness related terms : reduced disparity between subgroups for the recourses on the original data and increasing disparity between subgroups by generating lower cost counterfactuals for the protected group when the perturbation δ is added to the instances . This objective is , θ : = arg min θ L ( θ , D ) +Ex∼Dnegnp [ d ( x , Aθ ( x+ δ ) ) ] + ( Ex∼Dnegpr [ d ( x , Aθ ( x ) ) ] − Ex∼Dnegnp [ d ( x , Aθ ( x ) ) ] ) 2 s.t . Ex∼Dnegnp [ d ( x , Aθ ( x+ δ ) ) ] < Ex∼Dnegpr [ d ( x , Aθ ( x ) ) ] ( 4 ) Optimizing this objective requires computing the derivative ( Jacobian ) of the counterfactual explanation Aθ with respect to θ , ∂∂θAθ ( x ) . Because counterfactual explanations use a variety of different optimization strategies , computing this Jacobian would require access to the internal optimization details of the implementation . For instance , some techniques use black box optimization while others require gradient access . These details may vary by implementation or even be unavailable . Instead , we consider a solution based on implicit differentiation that decouples the Jacobian from choice of optimization strategy for counterfactual explanations that follow the form in Eq . ( 1 ) . We calculate the Jacobian as follows : Lemma 3.1 Assuming the counterfactual explanation Aθ ( x ) follows the form of the objective in Equation 1 , ∂∂xcfG ( x , Aθ ( x ) ) = 0 , and m is the number of parameters in the model , we can write the derivative of counterfactual explanation A with respect to model parameters θ as the Jacobian , ∂ ∂θ Aθ ( x ) = − [ ∂2G ( x , Aθ ( x ) ) dx2cf ] −1 · [ ∂ ∂θ1 ∂ ∂xcf G ( x , Aθ ( x ) ) · · · ∂ ∂θm ∂ ∂xcf G ( x , Aθ ( x ) ) ] We provide a proof in Appendix A . Critically , this objective does not depend on the implementation details of counterfactual explanation A , but only needs black box access to the counterfactual explanation . One potential issue is the matrix inversion of the Hessian . Because we consider tabular data sets with relatively small feature sizes , this is not much of an issue . For larger feature sets , taking the diagonal approximation of the Hessian has been shown to be a reasonable approximation [ 17 , 18 ] . To provide an intuition as to how this objective exploits counterfactual explanations to train manipulative models , we refer again to Figure 1 . Because the counterfactual objective G relies on an arbitrary function f , this objective can be non-convex . As a result , we can design f such that G converges to higher cost local minimums for all datapoints x ∈ D than those G converges to when we add δ . 3The objectives discussed in this section use the training set , whereas , evaluation is done on a held out test set everywhere else .

Summary: Context: Counterfactual explanations are a way to locally interpret model decisions for a given input by finding a "nearby" input with a different model decision. As in this paper, we can interpret some distance measure between the original input and its "counterfactual explanation" point as a kind of cost of achieving a different decision, usually for/to an individual who gets a negative decision to understand what they would need to change to receive a positive decision. In a fairness context we can consider differences in such costs between different groups as evidence of unfairness, for example if there are higher costs (on average, say) for individuals from a protected non-discrimination group.  The current paper shows that for a certain class of algorithms producing counterfactual explanations, it is possible for a small, adversarial perturbation of the input to produce a dramatically different explanation. The paper points out that this non-robustness of the algorithms in consideration is important from a fairness or regulatory standpoint by simulating a scenario where a model builder can jointly train a classifier and adversarial perturbation so that an auditor relying on counterfactual explanations (using any of the algorithms in consideration here) can be deceived. The paper shows how to train such models that can hide unfairness by providing a low-cost explanation for a nearby point relative to the higher cost explanation of the original input, and demonstrates the concept with experiments using two datasets and four different counterfactual explanation algorithms in the literature. Experiments also show how to achieve robustness by changing the initialization point for the counterfactual explanation algorithm, by reducing the dimension of the input space, or by otherwise constraining the complexity of the classification model.