Low-Cost Algorithmic Recourse for Users With Uncertain Cost Functions

1 INTRODUCTION . Over the past few years ML models have been increasingly deployed to make critical decisions related to loan approval ( Siddiqi , 2012 ) , insurance ( Scism , 2019 ) , allocation of public resources ( Chouldechova et al. , 2018 ; Shroff , 2017 ) and hiring decisions ( Ajunwa et al. , 2016 ) . Decisions from these models have real-life consequences for the individuals ( users ) involved . As a result , there is a growing emphasis on explaining these models ’ decisions ( Ribeiro et al. , 2018 ; Lundberg & Lee , 2017 ; Poulin et al. , 2006 ) and providing recourse for unfavorable decisions ( Voigt & dem Bussche , 2018 ; Karimi et al. , 2020a ) . A recourse is an actionable plan that is given to someone allowing them to change the decision of a deployed model to a desired alternative ( Wachter et al. , 2017 ) . Recourses can be highly valuable for users in situations where model decisions determine important life outcomes . Or , in cases where no feasible recourse is possible , users may wish to dispute the use of a model in the first place , and we might take this as evidence that greater reforms to the decision-making system are needed ( Venkatasubramanian & Alfano , 2020 ) . Recourses are desired to be actionable , feasible , and non-discriminatory . Actionable means that only features which can be changed by the user are requested to be changed . These changes should also be possible under the data distribution . For example , Education level can not be decreased from a Masters to Bachelors degree but can be increased from Masters to PhD . It is also not actionable to change your Race ( Mothilal et al. , 2020 ) . A recourse is feasible if it is reasonably easy for the user to adopt , i.e . it is actionable and has a low cost for the user . Non-Discriminatory means that the recourse method should be equitable across population subgroups . There are now many fairness metrics that can be used to measure this ( Hinnefeld et al. , 2018 ) , e.g . the ratio between the average cost of recourse for two subgroups in a population . While we want recourses to be feasible for all users , it is difficult to directly optimize for a user ’ s incurred cost unless we have access to their ground-truth cost function . In the absence of detailed cost function data , prior work has used other heuristic objectives for feasibility . For instance , Mothilal et al . ( 2020 ) and Wachter et al . ( 2017 ) assume that if the vector distance between the user ’ s current state and the recourse is small , then recourse will be low cost . These works encourage this property via a proximity objective . Meanwhile , sparsity quantifies the number of features that require modification to implement a recourse ( Mothilal et al. , 2020 ) . When providing multiple recourse options , diversity in proposed recourses is used to counter uncertainty around the user cost function ( Mothilal et al. , 2020 ; Cheng et al. , 2021 ) . The assumption is that if users are provided with diverse options then they are more likely to find at least one feasible solution . Later in section 5.2 , we show that diversity as an objective correlates poorly with user cost , suggesting it would be strongly preferable to optimize for user cost directly . A few approaches to recourse do directly optimize for the cost of recourse , but they assume there is a single cost function shared by all users ( Ustun et al. , 2019 ; Rawal & Lakkaraju , 2020 ; Karimi et al. , 2020c ; d ; Cui et al. , 2015 ) . We believe it is crucial to have user-specific cost functions , as a global cost function might poorly represent different users in a diverse population . In this work , we propose a method for identifying a user-specific recourse set that contain at least one good solution for the user . We directly optimize for user cost by quantifying the actionability and feasibility of proposed recourses . In the absence of data about user cost functions , we treat them as hidden from the recourse method and assume they follow an underlying cost distribution . However , we provide users with an option to specify their preferred editable features or the complete cost function detailing the costs of transitions between features values . We model this cost distribution via a highly flexible hierarchical cost sampling procedure which makes minimal assumptions about user preferences ( Algorithm 1 ) . Based on this distribution , we propose an objective function , Expected Minimum Cost ( EMC ) , which allows us to approximately optimize for user satisfaction by first sampling plausible cost functions , then finding a set that achieves a good cost in expectation . The EMC objective encourages the solution set to consist of counterfactuals that are each a good counterfactual under some particular cluster of cost functions from the distribution . Hence , no matter what the user ’ s ground-truth cost function is , we will have some counterfactual that is well suited to the user ’ s cost function ( shown in Figure 1 ) . Next , we propose a simple discrete optimization method , Cost-Optimized Local Search ( COLS ) , in order to optimize for Expected Minimum Cost . COLS guarantees a monotonic reduction in the Expected Minimum Cost of the counterfactual set , which we show leads to large empirical reductions in user-incurred cost . To evaluate the effectiveness of our proposed techniques , we run experiments on two popular realworld datasets : Adult-Income ( Dua & Graff , 2017 ) and COMPAS ( Larson et al. , 2016 ) . We compare our method with multiple strong baselines methods like Diverse Counterfactual Explanations ( DICE ) ( Mothilal et al. , 2020 ) , Feasible and Actionable Counterfactual Explanations ( FACE ) ( Poyi- adzi et al. , 2020 ) , and Actionable Recourse ( AR ) ( Ustun et al. , 2019 ) . We evaluate these methods on existing metrics from the literature like diversity , proximity , sparsity , and validity ( Section 5.1 ) along with two cost-based metrics . In particular , we measure the fraction of satisfied users , based on whether their cost of recourse is below a certain threshold k. We also report coverage , which is the fraction of users with at least one actionable recourse ( Rawal & Lakkaraju , 2020 ) . Using simulated user cost functions , we show that our method satisfies 25.89 % and 17.93 % percentage points more users than strong baseline methods while covering 22.35 % and 17.13 % more users , on the Adult-Income and COMPAS dataset respectively . We perform important ablations to show whether performance can be attributed to the COLS optimization method or the EMC objective . Additionally , we evaluate the robustness of our method to various distribution shifts that can occur between the user ’ s hidden cost distribution and the hierarchical cost sampling distribution . We find that our method is robust to these distribution shifts and generalizes well to user cost function from these shifted distributions . Lastly , we perform a fairness analysis of all the methods across demographic subgroups based on Gender and Race . Standard fairness metrics demonstrate that , in most comparisons , our method is more fair relative to the strongest baseline methods . Our primary contributions in this paper are listed below . 1 . We propose to evaluate user-incurred cost and fraction of satisfied users by means of hidden user-specific cost functions , rather than a known global cost function . 2 . We propose a new objective function , Expected Minimum Cost ( EMC ) , which allows us to approximately optimize for user satisfaction when user cost functions are not known . 3 . We propose a discrete optimization method , Cost-Optimized Local Search ( COLS ) , which achieves up to 25.89 % percentage points higher user satisfaction relative to the next-best baseline . COLS guarantees a monotonic reduction in EMC , which we find provides a 19 % point improvement over a simple local search . 4 . We show that in most settings , COLS provides more fair solutions across demographics subgroup than comparable recourse methods , while offering recourse to a substantially higher fraction of users . 2 RELATED WORK . A wide variety of methods have been proposed for generating recourses . For a comprehensive survey of existing recourse methods , we refer readers to Karimi et al . ( 2020b ) . Here , we distinguish our approach based on our recourse objectives , optimizer , and evaluation . We primarily discuss recourse methods , though there is useful complementary work on problems such as providing recourse when there is distribution shift in the data ( Upadhyay et al. , 2021 ; Slack et al. , 2021 ) and training models which guarantee recourse to affected individuals with high probability ( Ross et al. , 2021 ) . Objectives : The most prominent family of objectives for recourse includes distance-based objectives ( Wachter et al. , 2017 ; Karimi et al. , 2020a ; Dhurandhar et al. , 2018 ; Mothilal et al. , 2020 ; Rasouli & Yu , 2021 ) . These methods seek recourses that are close to the original data point . In DICE , Mothilal et al . ( 2020 ) provide users with a set of counterfactuals while trading off between proximity , a distance-based objective , and diversity . Diversity-based methods assume that providing diverse options will increase the chance a user is satisfied by one of the options . A second category of methods uses other heuristics based on the data distribution ( Aguilar-Palacios et al. , 2020 ; Gomez et al. , 2020 ) to come up with counterfactuals . FACE constructs a graph from the given data and then tries to find a high-density path between points in order to generate counterfactuals ( Poyiadzi et al. , 2020 ) . Lastly , the works closest to ours are the cost-based objectives , which capture feasibility in terms of the cost of recourse : ( 1 ) Cui et al . ( 2015 ) define a cost function based on the minimum and maximum values a factor can take in their additive tree model . ( 2 ) Karimi et al . ( 2020c ; d ) take a causal intervention perspective on the task and define cost in terms of the normalized distance between the user state and the counterfactual . ( 3 ) Ustun et al . ( 2019 ) define cost in terms of the number of changed features and frame recourse generation as an Integer Linear Program . ( 4 ) Rawal & Lakkaraju ( 2020 ) infer a cost function from simulated rankings of features for actionability , then optimize recourses for this cost function . Importantly , all of these works assume there is a known global cost function that is shared by all users . In our work , we drop this assumption , and instead we optimize for cost over a distribution of plausible user-specific cost functions . Optimization : Early work on recourse methods uses gradient-based optimization to search for counterfactuals close to a user ’ s data point ( Wachter et al. , 2017 ) . Several methods since then also use gradient-based optimization ( Mothilal et al. , 2020 ; Chen et al. , 2020 ) . Some recent approaches use tree-based techniques ( Rawal & Lakkaraju , 2020 ; Aguilar-Palacios et al. , 2020 ; von Kügelgen et al. , 2020 ; Hashemi & Fathi , 2020 ; Kanamori et al. , 2020 ) , kernel-based methods ( Dandl et al. , 2020 ; Gomez et al. , 2020 ; Ramon et al. , 2020 ) , while others employ some heuristic ( Poyiadzi et al. , 2020 ; Aguilar-Palacios et al. , 2020 ) to generate counterfactuals . A few works use latent space perturbation with autoencoders to generate recourses ( Pawelczyk et al. , 2020 ; Joshi et al. , 2019 ) , while Karimi et al . ( 2020a ) and Ustun et al . ( 2019 ) utilize SAT and ILP solvers , respectively . Here , we introduce a discrete optimization method specialized for our Expected Minimum Cost objective . Evaluation : Besides ensuring that recourses are classified as the favorable class by a model ( validity ) , the most prominent approaches to evaluate recourses rely on Distance-based metrics . In DICE , Mothilal et al . ( 2020 ) evaluate recourses according to their proximity , sparsity , and diversity . Meanwhile , several works directly consider the cost of the recourses , using a known global cost function as a metric , meaning that all users share a cost function , which is available to the recourse generation method ( Cui et al. , 2015 ; Karimi et al. , 2020c ; d ) . In a slight departure from this setting , Rawal & Lakkaraju ( 2020 ) estimate a cost function from simulated pairwise feature comparisons , but this single estimate is used for both recourse generation and evaluation . In contrast , we evaluate a recourse method by simulating user-specific cost functions which can very greatly across users , and these cost functions are not known in advance to recourse generation methods . We will also measure recourse coverage as defined by Rawal & Lakkaraju ( 2020 ) , which measures the fraction of users that were provided with a recourse by the method .

Summary: This work introduces a new method for identifying actionable recourses for users with user-specific cost functions. Users’ cost functions are hidden from the recourse method. The paper proposed a discrete optimization algorithm COLS to solve the objective EMC. It further used a popular real-world dataset to illustrate the performance.