Package: fairaudit
fairauditfairauditis a Python package for fairness auditing with statisical guarantees.Given a hold-out set ${(x_i, y_i)}_{i = 1}^n$, a model $f(\cdot)$,
and a collection of groups $\mathcal{G}$,fairauditissues
simultaneously valid guarantees on group-wise performance and discovers
groups with disparate performance for an arbitrary metric, $\ell(f(x), y)$.Formally, let$$\epsilon(G) := \mathbb{E}_P[\ell(f(X), Y) \mid (X, Y) \in G] - \theta_P.$$Then, thecertificationtask corresponds to issuing a simultaneously valid confidence set
(can be an upper bound, lower bound, or interval) for $\epsilon(G)$, while theflaggingtask corresponds to discovering $G$ for which $\epsilon(G)$ fails to meet
some tolerance threshold. The latter discoveries satisfy an asymptotic FDR guarantee.Installationfairaudit can be installed (locally for now) with pip.To install with pip, navigate to the directory for this repo and$pipinstall.ExamplesThe easiest way to start using fairaudit may be to go through the following notebook:COMPASUsageTheAuditorClassTheAuditorclass has the following API:classAuditor:def__init__(self,x:np.ndarray,y:np.ndarray,z:np.ndarray,metric:fairaudit.Metric):...defcalibrate_groups(self,alpha:float,type:str,groups:Union[np.ndarray,str],epsilon:float=None,bootstrap_params:dict={})->None:"""Obtain bootstrap critical values for a specific group collection.Parameters----------alpha : floatType I error thresholdtype : strTakes one of three values ('lower', 'upper', 'interval').See epsilon documentation for what these correpsond to.groups : Union[np.ndarray, str]Either a string for a supported collection of groups or a numpy arraylikely obtained by calling `get_intersections` or `get_rectangles`from group.pyArray dimensions should be (n_points, n_groups)epsilon : float = Noneepsilon = None calibrates for issuing confidence bounds.type = "upper" issues lower confidence bounds,type = "lower" issues upper confidence bounds,type = "interval" issues confidence intervals.If a non-null value is passed in, we issue a Boolean certificate.type = "upper" tests the null that epsilon(G) >= epsilontype = "lower" tests the null that epsilon(G) <= epsilontype = "interval" tests the null that |epsilon(G)| >= epsilonbootstrap_params : dict = {}Allows the user to specify a random seed, number of bootstrap resamples,and studentization parameters for the bootstrap process."""defquery_group(self,group:Union[np.ndarray,int])->Tuple[List[Union[float,bool]],List[float],List[float]]:"""Query calibrated auditor for certificate for a particular groupParameters----------group : Union[np.ndarray, int]Will accept index into groups originally passed in or Booleanarray if calibrated collection was infiniteReturns-------certificate : List[Union[float, bool]]Boolean certificates or confidence bounds for each metric auditedvalue : List[float]Empirical value of epsilon(G) for each metric auditedthreshold : List[float]Estimate of theta for each metric audited"""defcalibrate_rkhs(self,alpha:float,type:str,kernel:str,kernel_params:dict={},bootstrap_params:dict={})->None:"""Obtain bootstrap critical value for a specified RKHS.Parameters----------alpha : floatType I error thresholdtype : strTakes one of three values ('lower', 'upper', 'interval').type = "upper" issues lower confidence bounds,type = "lower" issues upper confidence bounds,type = "interval" issues confidence intervals.kernel : strName of scikit-learn kernel the user would like to use.Suggested kernels: 'rbf' 'laplacian' 'sigmoid'kenrnel_params : dict = {}Additional parameters required to specify the kernel,e.g. {'gamma': 1} for RBF kernelbootstrap_params : dict = {}Allows the user to specify a random seed, number of bootstrap resamples,and studentization parameters for the bootstrap process."""defquery_rkhs(self,weights:np.ndarray)->Tuple[List[float],List[float]]:"""Query calibrated auditor for certificate for a particular RKHSfunction.Parameters----------weights : np.ndarrayRHKS function weights, i.e. f(x_i) = (Kw)_iReturns-------certificate : List[float]Confidence bounds for each metric queried.value : List[float]Empirical value of epsilon(G) for each metric queried."""defflag_groups(self,groups:np.ndarray,type:str,alpha:float,epsilon:float=0,bootstrap_params:dict={"student":"mad","student_threshold":-np.inf})->Tuple[np.ndarray,np.ndarray]:"""Returns flags and estimates of epsilon(G) for each group in some finitecollection.Parameters----------groups : np.ndarrayBoolean numpy array of dimension (n_points, n_groups)type : strTakes values ('upper', 'lower', 'interval')'upper' tests the null, epsilon(G) >= epsilon'lower' tests the null, epsilon(G) <= epsilon'interval' tests the null, |epsilon(G)| <= epsilonalpha : floatFDR levelepsilon : float = 0See 'type' documentationbootstrap_params : dict = {"student" : "mad", "student_threshold" : -np.inf}Allows the user to specify a random seed, number of bootstrap resamples,and studentization parameters for the bootstrap process.Returns-------flags : List[bool]One flag is raised for each group - at least one metric must be flaggedfor the group to receive a True flag.values : List[float]Empirical value of epsilon(G) for each metric queried."""TheMetricclassThe Metric object should be instantiated for any performance metrics the auditor
may be interested in querying. Its methods are never directly queried by the user,
but its constructor should be usable.classMetric:def__init__(self,name:str,evaluation_function:Callable[[np.ndarray,np.ndarray],np.ndarray]=None,threshold_function:Callable[[np.ndarray,np.ndarray],float]=None,metric_params:dict={})->None:"""Constructs the Metric object used by the Auditor object to recompute performanceover bootstrap samples.Parameters----------name : strevaluation_function : Callable[[np.ndarray, np.ndarray], np.ndarray] = NoneFunction applied to model predictions (Z) and true labels (Y) that returnsan array of metric values, e.g. the evaluation_function formean squared error is lambda z, y: return (z - y)**2threshold_function : Callable[[np.ndarray, np.ndarray], float]Function applied to model predictions (Z) and true labels (Y) that returnsa single threshold for comparison, e.g. when comparing to the population average,the threshold_function for MSE is lambda z, y: return np.mean((z - y)**2)metric_params : dict = {}Additional parameters may be required for metrics that require separate errortracking depending on the predicted value or true label.For 'calibration'-type metrics, the key 'calibration_bins' should map to a listthat determines how the predicted values (Z) should be digitized/binnedFor 'equalized odds'-type metrics, the key 'y_values' should map to a listso that the metric is calculated separately for each value of y in that list"""ThegroupsmoduleWe provide two methods in thegroupsmodule for constructing collections of groups
that can be audited.defget_intersections(X:np.ndarray,discretization:dict={},depth:int=None)->np.ndarray:"""Construct groups formed by intersections of other attributes.Parameters----------X : np.ndarraydiscretization : dict = {}Keys index columns of XValues specify input to the "bins" argument of np.digitize(...)depth : int = NoneIf None, we consider all intersections, otherwisewe all consider intersections of up to specified depth.Returns---------groups : np.ndarrayBoolean numpy array of size (n_points, n_groups)"""defget_rectangles(X:np.ndarray,discretization:dict={})->np.ndarray:"""Construct rectangles formed by attributes.Parameters----------discretization : dictKeys index columns of XValues specify input to the "bins" argument of np.digitize(...)Returns---------groups : np.ndarrayBoolean numpy array of size (n_points, n_groups)"""CitingIf you use this code in a research project, please cite the forthcoming paper.@article{cherian2023statistical,
    title={Statistical inference for fairness auditing},
    author={John Cherian \and Emmanuel Cand\`es},
    publisher = {arXiv},
    year = {2023},
    note = {arXiv:2305.03712 [stat.ME]},
    url = {https://arxiv.org/abs/2305.03712},
}