Low-Cost Algorithmic Recourse for Users With Uncertain Cost Functions

1 INTRODUCTION . Over the past few years ML models have been increasingly deployed to make critical decisions related to loan approval ( Siddiqi , 2012 ) , insurance ( Scism , 2019 ) , allocation of public resources ( Chouldechova et al. , 2018 ; Shroff , 2017 ) and hiring decisions ( Ajunwa et al. , 2016 ) . Decisions from these models have real-life consequences for the individuals ( users ) involved . As a result , there is a growing emphasis on explaining these models ’ decisions ( Ribeiro et al. , 2018 ; Lundberg & Lee , 2017 ; Poulin et al. , 2006 ) and providing recourse for unfavorable decisions ( Voigt & dem Bussche , 2018 ; Karimi et al. , 2020a ) . A recourse is an actionable plan that is given to someone allowing them to change the decision of a deployed model to a desired alternative ( Wachter et al. , 2017 ) . Recourses can be highly valuable for users in situations where model decisions determine important life outcomes . Or , in cases where no feasible recourse is possible , users may wish to dispute the use of a model in the first place , and we might take this as evidence that greater reforms to the decision-making system are needed ( Venkatasubramanian & Alfano , 2020 ) . Recourses are desired to be actionable , feasible , and non-discriminatory . Actionable means that only features which can be changed by the user are requested to be changed . These changes should also be possible under the data distribution . For example , Education level can not be decreased from a Masters to Bachelors degree but can be increased from Masters to PhD . It is also not actionable to change your Race ( Mothilal et al. , 2020 ) . A recourse is feasible if it is reasonably easy for the user to adopt , i.e . it is actionable and has a low cost for the user . Non-Discriminatory means that the recourse method should be equitable across population subgroups . There are now many fairness metrics that can be used to measure this ( Hinnefeld et al. , 2018 ) , e.g . the ratio between the average cost of recourse for two subgroups in a population . While we want recourses to be feasible for all users , it is difficult to directly optimize for a user ’ s incurred cost unless we have access to their ground-truth cost function . In the absence of detailed cost function data , prior work has used other heuristic objectives for feasibility . For instance , Mothilal et al . ( 2020 ) and Wachter et al . ( 2017 ) assume that if the vector distance between the user ’ s current state and the recourse is small , then recourse will be low cost . These works encourage this property via a proximity objective . Meanwhile , sparsity quantifies the number of features that require modification to implement a recourse ( Mothilal et al. , 2020 ) . When providing multiple recourse options , diversity in proposed recourses is used to counter uncertainty around the user cost function ( Mothilal et al. , 2020 ; Cheng et al. , 2021 ) . The assumption is that if users are provided with diverse options then they are more likely to find at least one feasible solution . Later in section 5.2 , we show that diversity as an objective correlates poorly with user cost , suggesting it would be strongly preferable to optimize for user cost directly . A few approaches to recourse do directly optimize for the cost of recourse , but they assume there is a single cost function shared by all users ( Ustun et al. , 2019 ; Rawal & Lakkaraju , 2020 ; Karimi et al. , 2020c ; d ; Cui et al. , 2015 ) . We believe it is crucial to have user-specific cost functions , as a global cost function might poorly represent different users in a diverse population . In this work , we propose a method for identifying a user-specific recourse set that contain at least one good solution for the user . We directly optimize for user cost by quantifying the actionability and feasibility of proposed recourses . In the absence of data about user cost functions , we treat them as hidden from the recourse method and assume they follow an underlying cost distribution . However , we provide users with an option to specify their preferred editable features or the complete cost function detailing the costs of transitions between features values . We model this cost distribution via a highly flexible hierarchical cost sampling procedure which makes minimal assumptions about user preferences ( Algorithm 1 ) . Based on this distribution , we propose an objective function , Expected Minimum Cost ( EMC ) , which allows us to approximately optimize for user satisfaction by first sampling plausible cost functions , then finding a set that achieves a good cost in expectation . The EMC objective encourages the solution set to consist of counterfactuals that are each a good counterfactual under some particular cluster of cost functions from the distribution . Hence , no matter what the user ’ s ground-truth cost function is , we will have some counterfactual that is well suited to the user ’ s cost function ( shown in Figure 1 ) . Next , we propose a simple discrete optimization method , Cost-Optimized Local Search ( COLS ) , in order to optimize for Expected Minimum Cost . COLS guarantees a monotonic reduction in the Expected Minimum Cost of the counterfactual set , which we show leads to large empirical reductions in user-incurred cost . To evaluate the effectiveness of our proposed techniques , we run experiments on two popular realworld datasets : Adult-Income ( Dua & Graff , 2017 ) and COMPAS ( Larson et al. , 2016 ) . We compare our method with multiple strong baselines methods like Diverse Counterfactual Explanations ( DICE ) ( Mothilal et al. , 2020 ) , Feasible and Actionable Counterfactual Explanations ( FACE ) ( Poyi- adzi et al. , 2020 ) , and Actionable Recourse ( AR ) ( Ustun et al. , 2019 ) . We evaluate these methods on existing metrics from the literature like diversity , proximity , sparsity , and validity ( Section 5.1 ) along with two cost-based metrics . In particular , we measure the fraction of satisfied users , based on whether their cost of recourse is below a certain threshold k. We also report coverage , which is the fraction of users with at least one actionable recourse ( Rawal & Lakkaraju , 2020 ) . Using simulated user cost functions , we show that our method satisfies 25.89 % and 17.93 % percentage points more users than strong baseline methods while covering 22.35 % and 17.13 % more users , on the Adult-Income and COMPAS dataset respectively . We perform important ablations to show whether performance can be attributed to the COLS optimization method or the EMC objective . Additionally , we evaluate the robustness of our method to various distribution shifts that can occur between the user ’ s hidden cost distribution and the hierarchical cost sampling distribution . We find that our method is robust to these distribution shifts and generalizes well to user cost function from these shifted distributions . Lastly , we perform a fairness analysis of all the methods across demographic subgroups based on Gender and Race . Standard fairness metrics demonstrate that , in most comparisons , our method is more fair relative to the strongest baseline methods . Our primary contributions in this paper are listed below . 1 . We propose to evaluate user-incurred cost and fraction of satisfied users by means of hidden user-specific cost functions , rather than a known global cost function . 2 . We propose a new objective function , Expected Minimum Cost ( EMC ) , which allows us to approximately optimize for user satisfaction when user cost functions are not known . 3 . We propose a discrete optimization method , Cost-Optimized Local Search ( COLS ) , which achieves up to 25.89 % percentage points higher user satisfaction relative to the next-best baseline . COLS guarantees a monotonic reduction in EMC , which we find provides a 19 % point improvement over a simple local search . 4 . We show that in most settings , COLS provides more fair solutions across demographics subgroup than comparable recourse methods , while offering recourse to a substantially higher fraction of users . 2 RELATED WORK . A wide variety of methods have been proposed for generating recourses . For a comprehensive survey of existing recourse methods , we refer readers to Karimi et al . ( 2020b ) . Here , we distinguish our approach based on our recourse objectives , optimizer , and evaluation . We primarily discuss recourse methods , though there is useful complementary work on problems such as providing recourse when there is distribution shift in the data ( Upadhyay et al. , 2021 ; Slack et al. , 2021 ) and training models which guarantee recourse to affected individuals with high probability ( Ross et al. , 2021 ) . Objectives : The most prominent family of objectives for recourse includes distance-based objectives ( Wachter et al. , 2017 ; Karimi et al. , 2020a ; Dhurandhar et al. , 2018 ; Mothilal et al. , 2020 ; Rasouli & Yu , 2021 ) . These methods seek recourses that are close to the original data point . In DICE , Mothilal et al . ( 2020 ) provide users with a set of counterfactuals while trading off between proximity , a distance-based objective , and diversity . Diversity-based methods assume that providing diverse options will increase the chance a user is satisfied by one of the options . A second category of methods uses other heuristics based on the data distribution ( Aguilar-Palacios et al. , 2020 ; Gomez et al. , 2020 ) to come up with counterfactuals . FACE constructs a graph from the given data and then tries to find a high-density path between points in order to generate counterfactuals ( Poyiadzi et al. , 2020 ) . Lastly , the works closest to ours are the cost-based objectives , which capture feasibility in terms of the cost of recourse : ( 1 ) Cui et al . ( 2015 ) define a cost function based on the minimum and maximum values a factor can take in their additive tree model . ( 2 ) Karimi et al . ( 2020c ; d ) take a causal intervention perspective on the task and define cost in terms of the normalized distance between the user state and the counterfactual . ( 3 ) Ustun et al . ( 2019 ) define cost in terms of the number of changed features and frame recourse generation as an Integer Linear Program . ( 4 ) Rawal & Lakkaraju ( 2020 ) infer a cost function from simulated rankings of features for actionability , then optimize recourses for this cost function . Importantly , all of these works assume there is a known global cost function that is shared by all users . In our work , we drop this assumption , and instead we optimize for cost over a distribution of plausible user-specific cost functions . Optimization : Early work on recourse methods uses gradient-based optimization to search for counterfactuals close to a user ’ s data point ( Wachter et al. , 2017 ) . Several methods since then also use gradient-based optimization ( Mothilal et al. , 2020 ; Chen et al. , 2020 ) . Some recent approaches use tree-based techniques ( Rawal & Lakkaraju , 2020 ; Aguilar-Palacios et al. , 2020 ; von Kügelgen et al. , 2020 ; Hashemi & Fathi , 2020 ; Kanamori et al. , 2020 ) , kernel-based methods ( Dandl et al. , 2020 ; Gomez et al. , 2020 ; Ramon et al. , 2020 ) , while others employ some heuristic ( Poyiadzi et al. , 2020 ; Aguilar-Palacios et al. , 2020 ) to generate counterfactuals . A few works use latent space perturbation with autoencoders to generate recourses ( Pawelczyk et al. , 2020 ; Joshi et al. , 2019 ) , while Karimi et al . ( 2020a ) and Ustun et al . ( 2019 ) utilize SAT and ILP solvers , respectively . Here , we introduce a discrete optimization method specialized for our Expected Minimum Cost objective . Evaluation : Besides ensuring that recourses are classified as the favorable class by a model ( validity ) , the most prominent approaches to evaluate recourses rely on Distance-based metrics . In DICE , Mothilal et al . ( 2020 ) evaluate recourses according to their proximity , sparsity , and diversity . Meanwhile , several works directly consider the cost of the recourses , using a known global cost function as a metric , meaning that all users share a cost function , which is available to the recourse generation method ( Cui et al. , 2015 ; Karimi et al. , 2020c ; d ) . In a slight departure from this setting , Rawal & Lakkaraju ( 2020 ) estimate a cost function from simulated pairwise feature comparisons , but this single estimate is used for both recourse generation and evaluation . In contrast , we evaluate a recourse method by simulating user-specific cost functions which can very greatly across users , and these cost functions are not known in advance to recourse generation methods . We will also measure recourse coverage as defined by Rawal & Lakkaraju ( 2020 ) , which measures the fraction of users that were provided with a recourse by the method .

Summary: In this paper, the problem of algorithmic recourse is studied where the goal is to find best recourse (counterfactual set) that is optimized for user cost. The author proposed new user-incurred cost evaluation method, Expected Minimum Cost (EMC), which approximate user satisfaction without assuming a fixed global user cost function, and instead consider user cost functions as hidden and user-specific. Specifically, the authors define cost function for each user as a set of feature-specific functions of user-incurred cost when transitioning between feature states, and define MinCost as the minimum transition cost across possible recourses. To cover diverse user cost functions, they propose to model user cost distribution with a hierarchical sampling procedure and estimated expected minimum cost by drawing samples from it. Next, they formulate a discrete optimization problem using EMC as objective, and propose a search algorithm (COLS) for best recourse generation. They introduce three new metrics for user satisfaction (all related to MinCost): FS@k, Coverage and PAC. Finally, they test with two real-world datasets and show that COLS achieves significant outperformance against baselines (that optimize for distance-based metrics) on the newly proposed metrics and show that their method is doing better in fairness as well.