Consistent Counterfactuals for Deep Models

1 INTRODUCTION . Deep Networks are increasingly being integrated into decision-making processes which require explanations during model deployment , from medical diagnosis to credit risk analysis ( Bakator & Radosav , 2018 ; et . al , 2017 ; Liu et al. , 2014 ; Sun et al. , 2016 ; De Fauw et al. , 2018 ; Babaev et al. , 2019 ; Addo et al. , 2018 ; Balasubramanian et al. , 2018 ; Wang & Xu , 2018 ) . Counterfactual examples ( Wachter et al. , 2018 ; Van Looveren & Klaise , 2019 ; Mahajan et al. , 2019 ; Verma et al. , 2020 ; Laugel et al. , 2018 ; Keane & Smyth , 2020 ; Ustun et al. , 2019 ; Sharma et al. , 2019 ; Poyiadzi et al. , 2020 ; Karimi et al. , 2020 ; Pawelczyk et al. , 2020a ) are often put forth as a simple and intuitive method of explaining decisions in such high-stakes contexts ( Mc Grath et al. , 2018 ; Yang et al. , 2020 ) . A counterfactual example for an input x is a related point x′ that produces a desired outcome y′ from a model . Intuitively , these explanations are intended to answer the question , “ Why did point x not receive outcome y′ ? ” either to give instructions for recourse , i.e . how an individual can change their behavior to get a different model outcome , or as a check to ensure a model ’ s decision is well-justified ( Ustun et al. , 2019 ) . Counterfactual examples are particularly popular in legal and business contexts , as they may offer a way to comply with regulations in the United States and Europe requiring explanations on high-stakes decisions ( e.g . Fair Credit Reporting Act ( FCRA ) and General Data Protection Regulation ( GDPR , 2016 ) ) , while revealing little information about the underlying model ( Barocas et al. , 2020 ; Mc Grath et al. , 2018 ) . Counterfactual examples are often viewed under the assumption that the decision system on which they will be used is static : that is , the model that creates the explanation will be the same model to which , e.g . a loan applicant soliciting recourse re-applies ( Barocas et al. , 2020 ) . However , during real model deployments in high-stakes situations , models are not constant through time : there are often retrainings due to small dataset updates , or fine-tunings to ensure consistent good behavior ( Merchant , 2020 ; pwc , 2020 ) . Thus , in order for counterfactuals to be usable in practice , they must return the same desired outcome not only for the model that generates them , but for similar models created during deployment . This paper investigates the consistency of model predictions on counterfactual examples between deep models with seemingly inconsequential differences , i.e . random seed and one-point changes in the training set . We demonstrate that some of the most common methods generating counterfactuals in deep models either are highly inconsistent between models or very costly in terms of distance from the original input . Recent work that has investigated this problem in simpler models ( Pawelczyk et al. , 2020b ) has pointed to increasing counterfactual cost , i.e . the distance between an input point and its counterfactual , as a method of increasing consistency . We show that while higher than minimal cost is necessary to achieve a stable counterfactual , cost alone is not a reliable signal to guide the search for stable counterfactuals in deep models ( Section 3 ) . Instead , we show that a model ’ s Lipschitz continuity and confidence around the counterfactual is a more reliable indicator of the counterfactual ’ s stability . Intuitively , this is due to the fact that these factors bound the extent of a models local decision boundaries will change across fine-tunings , which we prove in Section 4 . Following this result , we introduce Stable Neighbor Search ( SNS ) , which finds counterfactuals by searching for high-confidence points with small Lipschitz constants in the generating model ( Section 4 ) . Finally , we empirically demonstrate that SNS generates consistent counterfactuals while maintaining a low cost relative to other methods over several tabular datasets , e.g . Seizure and German Credit from UCI database ( Dua & Karra Taniskidou , 2017 ) , in Section 5 . In summary , our main contributions are : 1 ) we demonstrate that common counterfactual explanations can have low consistency across nearby deep models , and that cost is an insufficient signal to find consistent counterfactuals ( Theorem . 1 ) ; 2 ) to navigate this cost-consistency tradeoff , we prove that counterfactual examples in a neighborhood where the network has a small local Lipschitz constant are more consistent across changes to the last layer of weights , which suggests that such points are more stable across small changes in the training environment ( Theorem . 2 ) ; 3 ) leveraging this result , we propose SNS as a way to generate consistent counterfactual explanations ( Def . 5 ) ; 4 ) we empirically demonstrate the effectiveness of SNS in generating consistent and low-cost counterfactual explanations ( Table 1 ) . More broadly , this paper further develops a connection between the geometry of deep models and the consistency of counterfactual examples . When considered alongside related findings that focus on attribution methods , our work adds to the perspective that good explanations require good models to begin with ( Croce et al. , 2019 ; Wang et al. , 2020 ; Dombrowski et al. , 2019 ; Simonyan et al. , 2013 ; Sundararajan et al. , 2017 ) . 2 BACKGROUND . Notation . We begin with notation , preliminaries , and definitions . Let F ( x ; θ ) =argmaxifi ( x ; θ ) be a deep network where fi denotes the logit output for the i-th class and θ is the vector of trainable parameters . If F ( x ; θ ) ∈ { 0,1 } , there is only one logit output so we write f . Throughout the paper we assume F is piece-wise linear such that all the activation functions are ReLUs . We use ||x||p to denote the ` p norm of a vector x andBp ( x , ) def = { x′|||x′−x||p≤ , x′∈Rd } to denote a norm-bounded ball around x. Counterfactual Examples . We introduce some general notation to unify the definition of a counterfactual example across various approaches with differing desiderata . In the most general sense , a counterfactual example for an input x is an example xc that receives the different , often targeted , prediction while minimizing a user-defined quantity of interest ( QoI ) ( see Def . 1 ) : for example , a counterfactual explanation for a rejected loan application is a related hypothetical application that was accepted . We refer to the point x requiring a counterfactual example the origin point or input interchangeably . We note that there is a different definition of “ counterfactual ” widely used in the causality literature , where a counterfactual is given by an intervention on a causal model that is assumed to generate data observations ( Pearl , 2009 ) . This is a case of overlapping terminology , and is orthogonal to this work . We do not consider causality in this paper . Definition 1 ( Counterfactual Example ) . Given a model F ( x ) , an input x , a desired outcome class c 6=F ( x ; θ ) , and a user-defined quantity of interest q , a counterfactual example xc for x is defined as xc def = argminF ( x′ ; θ ) =cq ( x ′ , x ) where the cost of xc is defined as ||x−xc||p . The majority of counterfactual generation algorithms minimize of qlow ( x , x′ ) def = ||x−x′||p , potentially along with some constraints , to encourage low-cost counterfactuals ( Wachter et al. , 2018 ) . Some common variations include ensuring that counterfactuals are attainable , i.e . not changing features that can not be changed ( e.g . sex , age ) due to domain constraints ( Ustun et al. , 2019 ; Lash et al. , 2017 ) , ensuring sparsity , so that fewer features are changed ( Dandl et al. , 2020 ; Guidotti et al. , 2018 ) , or incorporating user preferences into what features can be changed ( Mahajan et al. , 2019 ) . Alternatively , a somewhat distinct line of work ( Pawelczyk et al. , 2020a ; Van Looveren & Klaise , 2019 ; Joshi et al. , 2019 ) also adds constraint to ensure that counterfactuals come from the data manifold . Other works still integrate causal validity into counterfactual search ( Karimi et al. , 2020 ) , or generate multiple counterfactuals at once ( Mothilal et al. , 2020 ) . We focus our analysis on the first two approaches , which we denote minimum-cost and data-support counterfactuals . We make this choice as the causal and distributional assumptions used in other counterfactual generation methods referenced are specific to a given application domain , whereas our focus is on the general properties of counterfactuals across domains . Specifically , we evaluate our results on minimum-cost counterfactuals introduced by Wachter et al . ( 2018 ) , and data-support counterfactuals from Pawelczyk et al . ( 2020a ) , and Van Looveren & Klaise ( 2019 ) . We give the full descriptions of these approaches in Sec . 5 . Counterfactual Consistency . Given two models F ( x ; θ1 ) and F ( x ; θ2 ) , a counterfactual example xc for F ( x ; θ1 ) is consistent with respect to F ( x ; θ2 ) means F ( xc ; θ1 ) =F ( xc ; θ2 ) . Following Pawelczyk et al . ( 2020b ) , we define the Invalidation Rate for counterfactuals in Def . 2 . Definition 2 ( Invalidation Rate ) . Suppose xc is a counterfactual example for x found in a model F ( x ; θ ) , we define the invalidation rate IV ( xc , Θ ) of xc with respect to a distribution Θ of trainable parameters as IV ( xc , Θ ) def =Eθ′∼ΘI [ F ( xc ; θ′ ) 6=F ( xc ; θ ) ] . Throughout this paper , we will call the model F ( x ; θ ) that creates the counterfactual the generating or base model . Recent work has investigated the consistency of counterfactual examples across similar linear and random forest models ( Pawelczyk et al. , 2020b ) . We study the invalidation rate with respect to the distribution Θ introduced by arbitrary differences in the training environment , such as random initialization and onepoint difference in the training dataset . We also assume F ( x ; θ′ ) uses the same set of hyper-parameters as chosen forF ( x ; θ ) , e.g . the number of epochs , the optimizer , the learning rate scheduling , loss functions , etc . 3 COUNTERFACTUAL INVALIDATION IN DEEP MODELS . As we demonstrate in more detail in Section 5 , counterfactual invalidation is a problem in deep networks on real data : empirically , we find that counterfactuals produce inconsistent outcomes in duplicitous deep models up to 94 % of the time . Previous work investigating the problem of counterfactual invalidation ( Pawelczyk et al. , 2020b ; Rawal et al. , 2021 ) , has pointed to increasing counterfactual cost as a potential mitigation strategy . In particular , they prove that higher cost counterfactuals will lead to lower invalidation rates in linear models in expectation ( Rawal et al. , 2021 ) , and demonstrate their relationship in a broader class of well-calibrated models ( Pawelczyk et al. , 2020b ) . While this insight provides interesting challenge to the perspective that low cost counterfactuals should be preferred , we show that cost alone is insufficient to determine which counterfactual has a greater chance of being consistent at generation time in deep models . The intuition that a larger distance between input and counterfactual will lead to lower invalidation rests on the assumption that the distance between a point x and a counterfactual xc is indicative of the distance from xc to the decision boundary , with a higher distance making xc ’ s prediction more stable under perturbations to that boundary . This holds well in a linear model , where there is only one boundary ( Rawal et al. , 2021 ) . However , in the complex decision boundaries of deep networks , going farther away from a point across the nearest boundary may lead to being closer to a different boundary . We prove that this holds even for a one-hidden-layer network by Theorem 1 . This observation shows that a counterfactual example that is farther from its origin point may be equally susceptible to invalidation as one closer to it . In fact , we show that the only models where ` p cost is universally a good heuristic for distance from a decision boundary , and therefore by the reasoning above , consistency , are linear models ( Lemma 1 ) . Theorem 1 . Suppose that H1 , H2 are decision boundaries in a piecewise-linear network F ( x ) = sign { w > 1 ReLU ( W0x ) } , and let x be an arbitrary point in its domain . If the projections of x onto the corresponding halfspace constraints ofH1 , H2 are onH1 andH2 , then there exists a point x′ such that : 1 ) d ( x′ , H2 ) =0 2 ) d ( x ′ , H2 ) < d ( x , H2 ) 3 ) d ( x , H1 ) ≤d ( x′ , H1 ) where d ( x , H∗ ) denotes the distance between x and the nearest point on a boundaryH∗ . Lemma 1 . LetH1 , H2 , F and x be defined as in Theorem 1 . If the projections of x onto the corresponding halfspace constraints ofH1 , H2 are onH1 andH2 , but there does not exist a point x′ satisfying ( 2 ) and ( 3 ) from Theorem 1 , thenH1 =H2 . Figure 1 illustrates the geometric intuition behind these results . The shaded regions of 1b correspond to two decision surfaces trained from different random seeds on the data in ( a ) . The lighter gray region denotes where the models disagree , whereas the black and white regions denote agreement . Observe that counterfactuals equally far from a decision boundary may have different invalidation behavior , as demonstrated by the counterfactuals c1 and c2 for the point x2 . Also note that as shown with x1 , being far away from one boundary may lead one to cross another one in deep models . However , for two linear models shown in Fig . 1c , being far away from the boundary is indeed a good indicator or being consistent . The discussion so far has demonstrated that there is not a strong theoretical relationship between cost and invalidation in deep models . In Section 5 , we test this claim on real data , and show that higher-cost counterfactuals can have higher invalidation rates than their lower-cost relatives ( c.f . Table 1 ) . Further , we show that the coefficient of determination ( R2 ) between cost and invalidation rate is very small ( with all but one around 0.05 ) . Thus , while cost and invalidation are certainly related—for example , it may be necessary for a stable counterfactual to be more costly than the minimum point across the boundary—cost alone is not enough to determine which one will be the most consistent in deep models .

Summary: This paper aims to generate valid counterfactual examples (CFE) in a dynamic setting where the model can be retrained in the test time. The goal is to minimize the invalidation rate. Different from previous work, they focus on DNNs. Through a series of theoretical analysis, they propose to maximize the confidence and minimize the Lipschitz constant under the constraint that the CFE x' and x share the same predicted label.  Experiments on tabular datasets show the proposed method reduces the invalidation rate a lot.