OxonFair: A Flexible Toolkit for Algorithmic Fairness

Authors: eoin  delaney
zihao  fu
sandra  wachter
brent  mittelstadt
chris  russell
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-02 11:23:15

Abstract: We present OxonFair, a new open source toolkit for enforcing fairness in binary classification. Compared to existing toolkits: (i) We support NLP and Computer Vision classification as well as standard tabular problems. (ii) We support enforcing fairness on validation data, making us robust to a wide range of overfitting challenges. (iii) Our approach can optimize any measure based on True Positives, False Positive, False Negatives, and True Negatives. This makes it easily extensible and much more expressive than existing toolkits. It supports all 9 and all 10 of the decision-based group metrics of two popular review articles. (iv) We jointly optimize a performance objective alongside fairness constraints. This minimizes degradation while enforcing fairness, and even improves the performance of inadequately tuned unfair baselines. OxonFair is compatible with standard ML toolkits, including sklearn, Autogluon, and PyTorch and is available  at https://github.com/oxfordinternetinstitute/oxonfair.

Main paper: 
 introduction: 
 The deployment of machine learning systems that make decisions about people offers an opportunity to create systems that work for everyone. However, such systems can lock in existing prejudices. Limited data for underrepresented groups can result in ML systems that do not work for them, while the use of training labels based on historical data can result in ML systems copying previous biases. As such, it is unsurprising that AI systems have repeatedly exhibited unwanted biases towards certain demographic groups in a wide range of domains including medicine [1, 2] , finance [3, 4] , and policing [5] . Such groups are typically identified with respect to legally protected attributes, such as ethnicity or gender [6, 7, 3] . The field of algorithmic fairness has sprung up in response to these biases.Contributions to algorithmic fairness can broadly be split into methodological and policy-based approaches. While much methodological work focuses on measuring and enforcing (un)fairness, a common criticism from the policy side is that this work can occur 'in isolation from policy and civil societal contexts and lacks serious engagement with philosophical, political, legal and economic theories of equality and distributive justice' [8] .In response to these criticisms, we have developed OxonFair, a more expressive toolkit for algorithmic fairness. We acknowledge that people designing algorithms are not always the right people to decide on policy, and as such we have chosen to create as flexible a toolkit as possible to allow policymakers and data scientists with domain knowledge to identify relevant harms and directly alter the system behaviour to address them. Unlike existing Fairness toolkits such as AIF360 [9] , which take a method-driven approach, and provide access to a wide range of methods but with limited control over their behaviour, we take a measure-based approach and provide one fairness method that is extremely customizable, and can optimize user-provided objectives and group fairness constraints. Figure 1 : Left: The need for an objective when enforcing fairness. We evaluate a range of methods with respect to balanced accuracy and demographic parity (OxonFair generates a frontier of solutions).Only OxonFair and RejectOptimization optimize balanced accuracy. As we improve the balanced accuracy of fair methods by adjusting classification thresholds (gray lines) fairness deteriorates. To avoid this, we jointly optimize a fairness measure and an objective. For more examples, see Figure 6 .Right Top: Using validation data in fairness. We compare against Fairlearn using standard algorithms with default parameters. These methods perfectly overfit and show no unfairness with respect to equal opportunity on the trainset, but substantial unfairness on test. OxonFair enforces fairness on held-out validation data and is less prone to overfitting. Right Bottom: A comparison of toolkits. AIF360 offers a large range of tabular methods, most of which do not allow fairness metric selection, Fairlearn offers fewer but more customizable tabular methods. OxonFair offers one method that can be applied to text, image, and tabular data, while supporting more notions of fairness and objectives.To do this, we focus on one of the oldest and simplest approaches to group fairness: per-group thresholding [10, 11, 3] , which is known to be optimal for certain metrics under a range of assumptions [3, 12, 13] . Our contribution is to make this as expressive as possible while retaining speed, for the relatively low number of groups common in algorithmic fairness. Inherently, any approach that allows a sufficiently wide set of objectives, and sets per-group thresholds will be exponential with respect to the number of groups, but we use a standard trick, widely used in the computation of measures such as mean absolute precision (mAP) to make this search as efficient as possible.Accepting the exponential complexity allows us to solve a much wider range of objectives than other toolkits, including maximizing F1 or balanced accuracy (see Figure 1 left), minimizing difference in precision [14] , and guaranteeing that the recall is above k% for every group [8] . Where groups are unavailable at test time, we simply use a secondary classifier to estimate group memberships [15, 16] and set different thresholds per inferred group to enforce fairness with respect to the true groups.Thresholding can be applied to most pretrained ML algorithms, and optimal thresholds can be selected using held-out validation data unused in training. This is vital for tasks involving deep networks such as NLP and computer vision, where the training error often goes to zero, and fairness methods that balance error rates between groups cannot generalize from constraints enforced on overfitting training data to previously unseen test data [17] . While overfitting is unavoidable in vision and NLP tasks, it is still a concern on tabular data. Figure 1 Top-Right, shows examples of decision trees, random forests [18] and XGBoost [19] trained with default parameters and obtaining 0 training error on standard datasets. This causes the Fairlearn reductions method [20] to fail to enforce fairness.NLP and vision are so challenging that two popular toolkits Fairlearn and AIF360 do not attempt to work in these domains. In contrast, we target them, making use of a recent work [21] that showed how fair classifiers based on inferred group thresholds can be compressed into a single network. 
related work: 
 Bias mitigation strategies for classification have been broadly categorized into three categories [6, [22] [23] [24] ; pre-processing, in-processing and post-processing.Pre-processing algorithms improve fairness by altering the dataset in an attempt to remove biases such as disparate impact [11] before training a model. Popular preprocessing approaches include simply reweighting samples in the training data to enhance fairness [25] , optimizing this process by learning probabilistic transformations [26] , or by generating synthetic data [27] [28] [29] .In-processing / In-training methods mitigate bias by adjusting the training procedure. Augmenting the loss with fair regularizers [23, 30] is common for logistic regression and neural networks. Agarwal et al. [31] iteratively alter the cost for different datapoints to enforce fairness on the train set. Approaches based on adversarial training typically learn an embedding that reduces an adversary's ability to recover protected groups whilst maximizing predictive performance [32] [33] [34] [35] . Other popular approaches include Disentanglement [36, 37] , Domain Generalization [38] [39] [40] , Domain-Independence [41] and simple approaches such as up-sampling or reweighing minority groups during training. Notably, in the case of high-capacity models in medical computer-vision tasks, a recent benchmark paper by Zong et al. [42] showed state-of-the-art in-processing methods do not significantly improve outcomes over training without consideration of fairness. A comprehensive benchmark study of in-processing methods in other domains is provided by Han et al. [43] .Post-processing methods enforce fairness by using thresholds and randomization to adjust the predictions of a trained model based on the protected attributes [3, 44] . Post-processing methods are typically model-agnostic and can be applied to any model that returns confidence scores.Enforcing Fairness on Validation Data avoids the misestimation of error rates due to overfitting. It has shown particular promise in computer vision through Neural Architecture Search [45] , adjusting decision boundaries [30] , reweighting [46] and data augmentation [17] .Most toolkits such as Fairness Measures [47] , TensorFlow Fairness Indicators [48] , and FAT Forensics [49] focus on measuring bias and do not support enforcing fairness through bias mitigation. FairML [50] audits fairness by quantifying the importance of different attributes in prediction. This is best suited for tabular data where features are well-defined. FairTest [51] investigates the associations between application outcomes (e.g., insurance premiums) and sensitive attributes such as age to highlight and debug bias in deployed systems. Aequitas [52] provides examples of when different measures are (in)appropriate with support for some bias mitigation methods in binary classification. Themis-ML [53] supports the deployment of several simple bias mitigation methods such as relabelling [25] , but focuses on linear models. Friedler et al. [22] introduce the more complete Fairness Comparison toolkit where four bias mitigation strategies are compared across five tabular datasets and multiple models (Decision trees, Gaussian Naïve Bayes, SVM, and Logistic Regression).There are two fairness toolkits that support sklearn [18] like OxonFair. These are the two most popular toolkits: Microsoft Fairlearn [20] (1.9k GitHub Stargazers as of June 2024) and IBM AIF360 [9] (2.4k Stargazers). AIF360 offers a diverse selection of bias measures and pre-processing, inprocessing and post-processing bias mitigation strategies on binary classification tabular datasets. For mitigation, Fairlearn primarily offers implementations of [31, 3] avoiding the use of the term bias, instead considering fairness through the lens of fairness-related harms [54] where the goal is to "help practitioners assess fairness-related harms, review the impacts of different mitigation strategies and make trade-offs appropriate to their scenario". Lee & Singh [55] recognized Fairlearn as one of the most user-friendly fairness toolkits, and critiqued AIF360 as being the least user-friendly toolkit.Both AIF360 and Fairlearn contain post-processing methods that select per-group thresholds. Unlike OxonFair, neither method uses the fast optimization we propose; both methods require group information at test time; AIF360 only supports two groups, but does use cross-validation to avoid overfitting; Fairlearn does not support the use of validation data, but does support more than two groups. According to their documentation, neither toolkit can be applied to NLP or computer vision. thresholds, alongside a determenistic variant and another using inferred group membership that are not supported by [57] . 
toolkit interface: 
 The interface of OxonFair decomposes into three parts: (i) evaluation of fairness and performance for generic classifier outputs. (ii) evaluating and enforcing fairness for particular classifiers. (iii) specialist code for evaluating and enforcing fairness for deep networks.Code for the evaluation of classifier outputs takes target labels, classifier outputs, groups, and an optional conditioning factor as input; while code for the evaluation and enforcement of fairness of a particular classifier, are initialized using the classifier, and from then on take datasets (in the form of a pandas dataframe [58] , or a dictionary) as input, and automatically extracts these factors from them.The evaluation code provide three functions: evaluate which reports overall performance of the classifier; evaluate_per_group which reports performance per group of the classifier; and evaluate_fairness which reports standard fairness metrics. All methods allow the user to specify which metrics should be reported. We recommend data scientists focus on evaluate_per_group which shows direct harms such as poor accuracy, precision, or low selection rate for particular groups.OxonFair provides an interface FairPredictor(classifier, validation_data, groups) that takes an existing classifier as an input, a validation dataset, and specification of the groups as an input and returns an object which we then enforce fairness on by calling .fit(objective, constraint, value). Internally, the method explores a wide range of possible thresholds for each group, membership of which is assumed to be either known or inferred by an auxiliary classifier.The resulting FairPredictor has evaluation methods as described above. When called without arguments, they report both the performance of the original and the updated fair classifier on the validation data. In addition, FairPredictor provides methods predict and predict_proba which make fair predictions and return scores corresponding to the left-hand side of Equation (1) .Calling fit optimizes the objective -typically a relevant performance criteria such as accuracy, subject to the requirement that the constraint is either greater or less than the value. If the objective should be minimized or maximized is inferred automatically, as is the requirement that the constraint is less than or greater than the value, but this default behavior can be user overridden. This is a relatively minimal interface, but one that is surprisingly expressive. By explicitly optimizing an objective, we can not just minimize the degradation of the metric as we enforce fairness, but sometimes also improve performance over the unfair baseline that is not fully optimized with respect to this metric. Even when optimizing for accuracy, this can create situations where it looks like some improvements in fairness can be had for free, although generally this is an artifact of the gap between optimizing log-loss and true accuracy in training. By formulating the problem as a generic constrained optimization, and not requiring the constraint to be a typical fairness constraint, we leave it open for enforcing a much broader space of possible objectives. This can be seen in Appendix C, where we show how to enforce minimax fairness [59] , maximize utility [60] combined with global recall constraints, and demonstrate levelling-up [8] by specifying minimum acceptable harm thresholds.Under the hood, a call to fit generates a Pareto frontier 1 and selects the solution that best optimizes the objective while satisfying the constraint. The frontier can be visualized with plot_frontier. 
inference: 
 To make decisions, we assign thresholds to groups. We write f (x) for the response of a classifier f , on datapoint x, t for the vector corresponding to the ordered set of thresholds, and G(x) for the one-hot encoding of group membership. We make a positive decision iff (x) -t • G(x) ≥ 0 (1)To optimize arbitrary measures we perform a grid search over the choices of threshold, t.Efficient grid sampling We make use of a common trick for efficiently computing measures such as precision and recall for a range of thresholds. This trick is widely used without discussion for efficient computation of the area under ROC curves, and we have had trouble tracking down an original reference for it. As one example, it is used by scikit-learn [18] .The trick is as follows: sort the datapoints by classifier response, then generate a cumulative sum of the number of positive datapoints and the number of negatives, going from greatest response to least. When picking a threshold between points i and i + 1, TP is given by the cumulative sum of positives in the decreasing direction up to and including i; FN is the sum of negatives in the same direction; FP is the total sum of positives minus TP, and TN is the total sum of negatives minus TN.We perform this trick per group, and efficiently extract the TP, FN, FP and TN for different thresholds. These are combinatorially combined across the groups and the measures computed. This two stage decoupling offers a substantial speed-up. If we write T for the number of thresholds, k for the number of groups, and n for the total number of datapoints, our procedure is upper-bounded by O(T k + n log n), while the naïve approach is O(nT k ). No other fairness method makes use of this, and in particular, all the threshold-based methods offered by AIF360 make use of a naïve grid search.From the grid sampling, we extract a Pareto frontier with respect to the two measures 2 . The thresholds that best optimize the objective while satisfying the constraint are returned as a solution. If no such threshold exists, we return the thresholds closest to satisfying the constraint.When using inferred characteristics, we offer two pathways for handling estimated group membership. The first pathway we consider makes a hard assignment of individuals to groups, based on a classifier response. The second pathway explicitly uses the classifier confidence as part of a per-datapoint threshold. In practice, we find little difference between the two approaches, but the hard assignment to groups is substantially more efficient and therefore allows for a finer grid search and generally better performance. However, the soft assignment remains useful for the integration of our method with neural networks, where we explicitly merge two heads (a classifier and a group predictor) of a neural network to arrive at a single fair model. For details of the two pathways see Appendix A.We use the method proposed in [21] (N.B., they used it only for demographic parity). Consider a network with two heads f , and g, comprised of single linear layers, and trained to optimize two tasks on a common backbone B. Let f be a standard classifier trained to maximize some notion of performance such as log-loss and g is a classifier trained to minimize the squared loss 3 For both methods, we enforced demographic parity over a train set consisting of 70% of the data. Despite the exponential complexity of our approach, we remain significantly faster until we reach 5 groups. The 0.6+ indicates the seconds to train XGBoost. OxonFair(S) indicates the runtime of the naive slow pathway described in Appendix A.2 rather than our accelerated approach.a vector that is a one hot-encoding of group membership. Any decision f (x) -t • g(x) ≥ 0 can now be optimized for given criteria by tuning weights w using the process outlined in the slow pathway.As both f and g are linear layers on top of a common nonlinear backbone B, we can write them as:EQUATIONnote that as f (x) is a real number, and g(x) is a vector, w f is a vector and b f a real number, while w g is a matrix and b g a vector.This means that the decision function f (x) -t • g(x) ≥ 0 can be rewritten using the identity:EQUATION) This gives a 3 stage process for enforcing any of these decision/fairness criteria for deep networks.1. Train a multitask neural network as described above. 2. Compute the optimal thresholds t on held-out validation data as described in Appendix A. 3. Replace the multitask head with a neuron with weights (w f -t • w g ) and bias (b f -t • b g ).To maximize performance, the training set should be augmented following best practices, while, to ensure fairness, the validation set should not 4 . The resulting network f * will have the same architecture as the original non-multitask network, while satisfying chosen criteria.OxonFair has a distinct interface for deep learning 5 . Training and evaluating NLP and vision frequently involves complex pipelines. To maximize applicability, we assume that the user has trained a two-headed network as described above, and evaluated on a validation set. Our constructor DeepFairPredictor takes as an input: the output of the two-headed network over the validation set; the ground-truth labels; and the groups. fit and the evaluation functionality can then be called in the same way. Once a solution is selected, the method merge_heads_pytorch generates the merged head, while extract_coefficients can be called to extract the thresholds t from 4, when working with a different framework.Out of the box, OxonFair supports all 9 of the decision-based group fairness measures defined by Verma and Rubin [61] and all 10 of the fairness measures from Sagemaker Clarify [62] . OxonFair Table 1 : We report mean scores over the 14 gender independent CelebA labels [28] . Single task methods and FairMixup scores in the second and third blocks are from Zietlow et al. [17] . ERM is the baseline architecture run without fairness. OxonFair (optimizing for accuracy and difference in equal opportunity (DEO)), has better accuracy (↑) and DEO (↓) scores than any other fair method.Uniconf. Adv. [74] Domain Disc. [75, 41] Ind. [41] OxonFair DEO ERM single task Debiasing GAN [28] Regularized [76, 77] g-SMOTE Adaptive [17] g-SMOTE [17] ERM [78] FairMixup [78] Acc. OxonFair does not support notions of individual fairness such as fairness through awareness [63] or counterfactual fairness [64, 65] .See Appendix B for a discussion of how metrics are implemented and comparison with two review papers. Appendix C contains details of non-standard fairness metrics, including utility optimization [60] ; minimax fairness [59, 66, 67] ; minimum rate constraints [8] , and Conditional Demographic Parity [68] . This also includes a variant of Bias Amplification [69, 70] . 
experimental analysis: 
 For tabular data, we compare with all group fairness methods offered by AIF360, and the reductions approach of Fairlearn. OxonFair is compatible with any learner with an implementation of the method predict_proba consistent with scikit-learn [18] including AutoGluon [71] and XGBoost [19] . A comparison with Fairlearn and the group methods from AIF360 on the adult dataset can be seen in Figures 1 and 6 using random forests. This follows the setup of [9] : we enforce fairness with respect to race and binarize the attribute to white vs everyone else (this is required to compare with AIF360), 50% train data, 20% validation, and 30% test, and a minimum leaf size of 20. With this large leaf size, all errors on train, validation, and test are broadly comparable, but our approach of directly optimizing an objective and a fairness measure leads us to outperform others.Figure 1 top right shows the importance of being able to use a validation set to balance errors. Using sklearn's default parameters we overfit to adult, and as the classifier is perfect on the training set, all fairness metrics that match error rates are trivially satisfied [72, 17] . The same behavior can be observed using XGBoost on the medical dataset [73] when enforcing equal opportunity with respect to sex 6 . In general, tabular methods need not overfit, and tuning parameters carefully can allow users to get relatively good performance while maintaining error rates between training and test.Figure 3 left shows Equal Opportunity on the COMPAS dataset. To show that OxonFair can also work in low-data regimes where we have insufficient data for validation, we enforce fairness on the training set. As before, we binarize race to allow the use of AIF360. We drop race from the training data, and use inferred protected attributes to enforce fairness. Here OxonFair generates a frontier that is comparable or better than results from existing toolkits, and OxonFair+ (see Section A), further improves on these results. See Figure 3 right for a comparison with Fairlearn varying the groups.CelebA [79] : We use the standard aligned & cropped partitions frequently used in fairness evaluation [17, 21, 28, 41] . Following Ramaswamy et al. [28] , we consider the 26 gender-independent, genderdependent and inconsistently labelled attributes as the target attributes for our evaluations (see Table 12 for details). Male is treated as the protected attribute.Implementation Details We follow Wang et al.'s setup [41] . We use a Resnet-50 backbone [80] trained on ImageNet [81] . A multitask classification model is trained, replacing the final fullyconnected layer of the backbone with a separate fully-connected head that performs binary prediction for all attributes. Dropout [82] (p = 0. [41, 43] . Occasionally, OxonFair finds solutions on the Pareto frontier that are both fairer and more accurate than the unconstrained classifier (See Figure 5 ).Figure 4 shows a novel fairness evaluation motivated by medical use cases [8, 42] where practitioners might want to correctly identify at least δ% of positive cases in each group. We evaluate how accuracy changes if we guarantee that the minimum recall is above δ% for every group. For OxonFair, we call .fit(accuracy, recall.min, δ). For other methods, we vary a global offset to ensure that the minimum recall is at least δ.We conducted experiments on hate speech detection and toxicity classification using two datasets: the multilingual Twitter corpus [84] and Jigsaw [85] . Experiments were performed across five languages (English, Polish, Spanish, Portuguese, and Italian) and five demographic factors (age, country, gender, race/ethnicity, and religion) were treated as the protected groups. For details, see Appendix F.1.We compare OxonFair with the following approaches. Base reports results of the standard BERT model [86] . CDA (Counterfactual Data Augmentation) [29, [87] [88] [89] [90] rebalances a corpus by swapping bias attribute words (e.g., he/she) in a dataset based on a given dictionary. DP (Demographic Parity) uses regularization [23, 43] to enforce DP. EO (Equal Opportunity [3] ) uses the regularization of [23, 43] to enforce EO. Dropout [88, 90] is used as a regularization technique [82] for bias mitigation and improving small group generalization. Rebalance [11, 91] Test Set: Wearing Earrings Results: Results are shown in Tables 3 and 4 . Our observations indicate that: 1) all debiasing methods improve the equal opportunity score and help mitigate bias on Twitter, but not on jigsaw. 2) our toolkit consistently reduces the difference in equal opportunity more than any other approach; 3) for 4/6 experiments we actually improve the objective over the baseline while enforcing fairness, showing the value in targeting a particular objective. For additional experiments on multilingual and multi-demographic data, and the Jigsaw race data, see Appendix F.2, and Appendix F.3.While improving fairness more than existing approaches, OxonFair performs substantially worse on these NLP datasets than on CelbA. There are two challenges present in these datasets but not in CelbA: (i) it is likely much harder to infer gender or religion from short text data than it is to infer gender from a photo of a face. (ii) The limited number of positively labelled datapoints examples that makes estimating DEO, F1,and balanced accuracy unstable (see Tables 16 and 17 for details). To better understand the influence of the two factors we refit OxonFair using the true rather than inferred attributes at test time (bottom block) and see no reliable improvements, suggesting that we are most limited by data scarcity. 
conclusion: 
 The key contributions of our toolkit lie in being more expressive than other approaches, and supporting NLP and computer vision. Despite this, most of the experiments focus on the standard definitions of Demographic Parity and Equal Opportunity. This is not because we agree that they are the right measures, but because we believe that the best way to show that OxonFair works is to compete with other methods in what they do best. On low-dimensional tabular data, when optimizing accuracy and a standard fairness measure, it is largely comparable with Fairlearn, but if overfitting or non-standard performance criteria and fairness metrics are a concern, then OxonFair has obvious advantages. For NLP, and computer vision, our approach clearly improves on existing state-of-the-art. In no small part, this is due to the observation of [17] , that methods for estimating or enforcing error-based fairness metrics on high-capacity models that do not use held-out validation data can not work.We hope that OxonFair will free policy-makers and domain experts to directly specify fairness measures and objectives that are a better match for the harms that they face. In particular, we want to call out the measures in Figure 8 as relevant to medical ML. The question of how much accuracy can we retain, while guaranteeing that classifier sensitivity (AKA recall) is above k% for every group, captures notions of fairness and clinical relevance in a way that standard fairness notions do not [8] .Limitations: We have chosen to optimize as broad a set of formulations as possible. As a result, for certain metrics (particularly equalized odds [3] ) the solutions found are known to be suboptimal 8 ; and for others [12] the exponential search is unneeded. Techniques targeting particular formulations may be needed to address this.A key challenge for most fairness approach is in obtaining the group labels used to measure unfairnesss, and we are no exception. In particular, the gender labels in CelebA and the race and religion labels in our NLP experiments consist of a small number of externally assigned labels that may not match how people self-identify. Improving and measuring fairness with respect to these coarse labels can miss other forms of inequality. Moreover, a major driver of unfairness is a lack of data regarding particular groups. However, this very absence of data makes it hard for any toolkit to detect or rectify unfairness.Broader Impact: OxonFair is a tool for altering the decisions made by ML systems that are frequently trained on biased data. Care must be taken that fair ML is used as a final step after correcting for bias and errors in data collation, and not as a sticking plaster to mask problems [92] . Indeed, inappropriate uses of fairness can lock in biases present in training [72] . Under the hood, OxonFair performs a form of positive discrimination, where we alter scores in response to (perceived) protected characteristics to rectify specific existing inequalities 9 . As such, there are many scenarios where its use may be inappropriate for legal or ethical reasons. , while enforcing demographic parity. We follow the design decisions of [9] and use a random forest with 100 trees and a minimum leaf size of 20. Only OxonFair allows the specification of an objective, and for all other methods we try to alter the decision threshold to better optimize the objective. However, as we improve the objective, we see fairness deteriorates. To avoid this, OxonFair jointly optimize both a fairness measure and an objective. 
 
 Appendix: In many situations, protected attributes are not available at test time. In this case, we simply use inferred characteristics to assign per-group thresholds and adjust these thresholds to guarantee fairness with respect to the true (i.e. uninferred) groups.When using inferred characteristics, we offer two pathways for handling estimated group membership. The first pathway we consider makes a hard assignment of individuals to groups, based on a classifier response. The second pathway explicitly uses the classifier confidence as part of a per-datapoint threshold. In practice, we find little difference between the two approaches, but the hard assignment to groups is substantially more efficient and therefore allows for a finer grid search and generally better performance. However, the soft assignment remains useful for the integration of our method with neural networks, where we explicitly merge two heads of a neural network to arrive at a single fair model.The fast pathway closely follows the efficient grid search for known characteristics. We partition the dataset by inferred characteristics, and then repeat the trick. However, as the inferred characteristics do not need to perfectly align with the true characteristics, we also keep track of the true group datapoints belongs to, i.e., for all datapoints assigned to a particular inferred group, we compute the cumulative sum of positives and negatives that truly belong to each group. This allows us to vary the thresholds with respect to inferred groups while computing group measures with respect to the true groups. This can be understood as replacing the decision function (1) with f (x) -t • G ′ (x) ≥ 0 where G ′ is a binary vector valued function that sums to 1, but need not correspond to G exactly. This explicit decoupling of inferred groups from the true group membership allows us to consider partitionings of the data that do not align with group membership. We found it particularly helpful to include an additional 'don't know' group. By default, any datapoint assigned a score10 from the classifier below 2/3 is assigned to this group, and receives a different threshold to those datapoints that the classifier is confident about. The improved frontiers are shown in the tabular experimental section as OxonFair+, where they offer a clear advantage over our baseline OxonFair.The slow pathway tunes t to optimize the decision process f (x) -t • g(x) ≥ 0, where g is a real vector valued function. Given the lack of assumptions, no obvious speed-up was possible and we perform a two stage naïve grid-search, first coarsely to extract an approximate Pareto frontier, and then a finer search over the range of thresholds found in the first stage. This is then followed by a final interpolation that checks for candidates around pairs of adjacent candidates currently in the frontier.In situations where g(x) is the output of a classifier and G ′ (x) its binarization, it is reasonable to suspect that loss of information from binarization might lead to a drop in performance when we compare the slow pathway with the fast. In practice, we never found a significant change, and in a like-with-like comparison over a similar number of thresholds the fast pathway was as likely to be fractionally better as it was to be worse. Moreover, for more than 3 groups the slow pathway becomes punitively slow, and to keep the runtime acceptable requires decreasing the grid size in a way that harms performance.Despite this, we kept the slow pathway as it is directly applicable to deep networks as we describe in the next section. In practice, when working with deep networks we make use of a hybrid approach, and perform the fast and slow grid searches before fusing them into a single frontier and then performing interpolation. This allows us to benefit from the better solutions found by a fine grid search when the output of the second head is near binary (see Figure 2 ), and robustly carry over to the slower pathway where its binarization is a bad approximation of the network output.To make OxonFair readily extensible, we create a custom class to implement all performance and fairness measures. As such, should OxonFair not support a particular measure, both the objectives and constraints can be readily extended by the end user.Measures used by OxonFair are defined as instances of a python class GroupMetrics. Each group measure is specified by a function that takes the number of True Positives, False Positives, False Negatives, and True Negatives and returns a score; A string specifying the name of the measure; and optionally a bool indicating if greater values are better than smaller ones.For example, accuracy is defined as: accuracy = gm.GroupMetric(lambda TP, FP, FN, TN: (TP + TN) / (TP + FP + FN + TN), 'Accuracy') For efficiency, our approach relies on broadcast semantics and all operations in the function must be applicable to numpy arrays. Having defined a GroupMetric it can be called in two ways. Either: accuracy(target_labels, predictions, groups)Here target_labels and predictions are binary vectors corresponding to either the target groundtruth values, or the predictions made by a classifier, with 1 representing the positive label and 0 otherwise. groups is simply a vector of values where each unique value is assumed to correspond to a distinct group.The other way it can be called is by passing it a single 3D array of dimension 4 by number of groups by k, where k is the number of candidate classifiers that the measure should be computed over. As a convenience, GroupMetrics automatically implements a range of functionality as sub-objects.Having defined a metric as above, we have a range of different objects:• metric.diff reports the average absolute difference of the method between groups.• metric.average reports the average of the method taken over all groups.• metric.max_diff reports the maximum difference of the method between any pair of groups.• metric.max reports the maximum value for any group.• metric.min reports the minimum value for any group.• metric.overall reports the overall value for all groups combined, and is the same as calling metric directly• metric.ratio reports the average over distinct pairs of groups of the smallest value divided by the largest • metric.per_group reports the value for every group.All of these can be passed directly to fit, or to the evaluation functions we provide.The vast majority of fairness metrics are implemented as a .diff of a standard performance measure, and by placing a .min after any measure such as recall or precision it is possible to add constraints that enforce that the precision or recall is above a particular value for every group.Total Metrics Computing certain metrics, particular Conditional Demographic Disparity [68] , and Bias Amplification [70] , requires knowledge of the total number of TP etc. alongside the number of TP in each group.To implement these measures, we support lambda functions that take per-group values, followed by the global values, giving 8 arguments in total. These lambda functions should be passed to GroupMetric in the same way along ith the optional argument total_metric=True.Here, we compare OxonFair against minimax fairness. To do this, we define a new performance measure corresponding to the lowest accuracy over the positive or negative labelled datapoints. min accuracy = min T P T P + F P , T N F N + T NMartinez et al [59] argued that we should seek a Pareto optimal solution that has the highest possible overall accuracy, subject to the requirement it maximizes the lowest per group accuracy. We can do this in OxonFair by calling fpredictor.fit(gm.min_accuracy.min, gm.accuracy, 0) Here min_accuracy.min corresponds to the lowest min accuracy of any group. We use accuracy > 0 as the constraint, as we do not want an active constraint from preventing us from finding the element of the Pareto frontier (see [59] for frontier details) with the highest minimum accuracy. Note that the groups used by OxonFair with this loss correspond to the true groups, such as ethnicity or gender, while the groups used by minimax fairness are the product of these groups with the target labels.Existing methods for minimax fairness optimize the same loss and have indistinguishable accuracy, only differing in the speed of convergence [67] As such, in Table 8 we only report results for a variant of [67] .Similarly, in computer vision [17] , optimized the same objective by iteratively generating synthetic data for the worst performing group, where groups were defined as the product of ground-truth labels, and sex. We compare against them in Table 15 .XGBoost OxonFair supports the utility-based approach of Bakalar et al. [60] , whereby different thresholds can be selected per group to optimize a utility-based objective. Utility functions can be defined in one line. In the following example, we consider a scenario where an ML system identifies issues that may require interventions. In this example, every intervention has a cost of 1, regardless of if it was needed, but a missed intervention that was needed has a cost of 5. Finally, not making an intervention when one was not needed has a cost of 0.Figure 7 shows code where fpredictor minimizes the utility subject to the requirement that the overall recall cannot drop below 0.5.1 from OxonFair import FairPredictor 2 from OxonFair.utils import group_metrics as gm One criticism of many methods of algorithmic fairness is that enforcing equality of recall rates (as in equal opportunity) or selection rates (as in demographic parity) will decrease the recall/selection rate for some groups while increasing it for others. This behavior is an artifact of trying to maximize accuracy [8] and occurs despite fairness methods altering the overall selection rate [95] . As an alternative, OxonFair supports levelling up where harms are reduced to, at most, a given level per group [8] . For example, if we believe that black patients are being disproportionately harmed by a high number of false negatives in cancer detection (i.e., low recall), instead of enforcing that these properties be equalized across groups, we can instead require that every group of patients has, at least, a minimum recall score. Depending on the use case, similar constraints can be imposed in with respect to per-group minimal selection rates, or minimal precision. These constraints can be enforced by a single call, for example, enforcing that the precision is above 70% while otherwise maximizing accuracy can be enforced by calling: .fit(gm.accuracy, gm.precision.min, 0.7). See also Figure 4 . To enforce levelling up as a hard additional constraint, fit takes an optional argument, force_levelling_up. Setting this equal to '+' forces the selection rate to increase in the search procedure, while setting it equal to '-' means it can only decrease. As most performance metrics underlying fairness constraints (e.g. recall, selection rate, precision) are monotonic with respect to the selection rate [95] , this can prevent levelling down.0The levelling-up constraint can be combined with standard forms of fairness e.g. by calling .fit(gm.accuracy, gm.demographic_parity, 0.01, force_levelling_up='+') but they are most useful when using levelling up constraints in conjuncture with an inadequately optimized classifier. In some circumstances calling .fit(gm.accuracy, gm.recall.min, 0.6) can result in a decrease in recall rate for some groups providing this increases accuracy and does not drop below the recall below the specified minimal rate.When deploying fairness in practice, we may be capacity limited. For example, as in Figure 8 we may use the output of a classifier for detecting cancer to schedule follow-up appointments. In such a case, you might wish that the recall is high for each demographic group, but be constrained by the number of available appointments. Calling .fit(gm.recall.min, gm.pos_pred_rate, 0.4, greater_is_better_const=False) will maximize the recall on the worst-off group subject to a requirement that no more than 40% of cases are scheduled follow-up appointments.In general, maximizing the group minimum of any measure that is monotone with respect to the selection rate, while enforcing a hard limit on the selection rate will enforce equality with respect to that measure (e.g. optimizing gm.recall.min will result in equal recall a.k.a. equal opportunity, while maximizing gm.pos_pred_rate.min will result in demographic parity), while also enforcing the selection rate constraints. See [95] for proof and a discussion of the issues arising, and [97] for an alternate approach.As such, calling Left: the change in precision as we enforce demographic parity. Note that we report precision as it is more informative than accuracy for low selection rates. Right: The ratio between selection rates (i.e. disparate impact) for different groups. We report the ratio rather than the difference, as the difference tends to zero as the selection rate also tends to zero. However, as the right figure shows, this ratio becomes unstable as the rate tends to zero.A key challenge of using fairness in practice is that often some sources of bias are known, and the practitioner is expected to determine if additional biases exist and to correct for them. For example, someone's salary affects which loans they are eligible for, but salary has a distinctly different distribution for different ethnicities and genders. [65] . Identifying and correcting fairness here rapidly becomes challenging, when considering the intersection of attributes, many small groups arise and purely by chance some form of unfairness may be observed [98, 68] suggested the use of a technique from descriptive statistics that [99] had previously applied to the problem of schools admissions at Berkley [100] . In this famous example, every school in Berkley showed little gender bias, but due to different genders applying at different rates to different schools, and the schools themselves having substantially different acceptance rate, a strong overall gender bias was apparent. [99] observed that you could correct for this bias by computing the per school selection-rate, and then taking a weighted average, where the weights are given by the total number of people applying to the school. The resulting selection rates are equivalent to a weighted selection-rate over the whole population, where the weight w i for an individual i in a particular group and applying to a particular school is w i = #individuals in school #individuals in group and school . To enforce this form of conditional demographic parity in OxonFair, we simply replace the sum of true positives etc. in Section 3, with the weighted sum. We support a range of related fairness metrics, including conditional difference in accuracy; and conditional equal opportunity (note that for equal opportunity we replace the numbers used to compute w i with the same counts but only taking into account those that have positive ground-truth labels). As such metrics can level down (Appendix C.3), we also support conditional minimum selection rates, and conditional minimum recall.In addition to this, we support the conditional fairness metrics based on EU/UK law of [68] . These measure the (weighted) proportion of people belonging to a particular group in the set of advantaged or disadvantaged people. These metrics can be computed for both the input target variables a classifier is trained to predict and for the classifier outputs.We also support variants of Bias Amplification, as defined by Zhao et al. and Wang et al. [69, 70] .As with the other metrics, we focus on scenarios where ground-truth group assignments exist even if they are unavailable at test time, and as such we focus on the Attribute → Task Bias (see [70] ).Bias Amplification is implemented as a per-group metric, gm.bias_amplification. As this measure is signed (a negative measure indicates the classifier reverses the bias present in the dataset), directly optimising it results in classifiers strongly biased in a new direction. Instead, we minimize the per group absolute bias amplification. The derivation is given below.Following the notation of Wang et al. [70] , let A be the set of protected demographic groups: for example, A = {male, female}. A a for a ∈ A is the binary random variable corresponding to the presence of the group a; thus P (A woman = 1) can be empirically estimated as the fraction of images in the dataset containing women. Let T t with t ∈ T similarly correspond to binary target tasks. Let Âa and Tt denote model predictions for the protected group a and the target task t, respectively.BiasAmp → = 1 |A||T | a∈A,t∈T y at ∆ at + (1 -y at )(-∆ at ) y at = 1 [P (A a = 1, T t = 1) > P (A a = 1)P (T t = 1)] ∆ at =        P ( Tt = 1|A a = 1) -P (T t = 1|A a = 1) if measuring Attribute → T ask Bias P ( Âa = 1|T t = 1) -P (A a = 1|T t = 1) if measuring T ask → Attribute Bias (6)Of which, the Attribute → Task Bias is relevant here.Each component can be written as a function of the global True Positives, False Positives etc., and the per group True Positives, and as such it can be optimized by our framework, albeit, not by using a standard group metrics. However, this metric is gamable, and consistently underestimating labels in groups where they are overrepresented and vice versa would be optimal, but undesirable behavior that leads to a negative score.Instead, we consider the absolute BiasAmp:EQUATIONWe can decompose |∆ at | into the appropriate form for a GroupMetric (see Appendix B) as follows:EQUATIONEQUATIONEQUATIONThis will give a per group estimate of the absolute bias amplification, and calling its .average method will give the absolute bias amplification over all groups.Fairret [56] is a recently published toolkit for enforcing fairness in pytorch networks, using standard fairness constraints [23] . This toolkit has only been shown to work for tabular data. There are mathematical reasons to think that it will not work for bias preserving fairness metrics [72] , such as equalized opportunity, on computer vision or NLP classifiers where classifiers obtain zero error on the training set [17] , and therefore trivially satisfy all bias preserving fairness metrics. However, as Fairret uses standard relaxations, it should have comparable accuracy/fairness trade-offs to OxonFair when weakly enforcing demographic parity on image data (see [21] ).Fairret should also have comparable performance on NLP data to the DP and EO regularized approaches reported in the main body of the paper. Figure 10 : Comparing OxonFair and Fairret [56] on adult using sex as the protected attribute. A simple neural network classifier with two hidden layers is used as the base classifier. Fairness strengths are varied for different Fairret implementations. Difference in Equal Opportunity and Demographic Parity are considered. An OxonFair-based frontier using XGBoost is also displayed.Here we focus on using Fairret to enforce Equal Opportunity on tabular data as shown in their paper.Figure 10 shows a comparison with Oxonfair. For a like-with-like comparison between Fairret and OxonFair we use 70% of the data for training for Fairret (which requires no validation data), and for OxonFair split this into 40% training data and 30% validation data.In general, neural networks perform worse than boosting on tabular data, and should not be used where maximizing accuracy is a concern. Nonetheless, we see that Fairret shows worse accuracy/fairness trade-offs than OxonFair on neural networks with the same architecture.Compared to OxonFair, there are three challenges faced by Fairret that might be contributing to its worse performance.• The enforced constraints are a relaxation of the underlying integer fairness constraints, and even when completely satisfied, they need not imply fairness [30] .• A mismatch between errors on the training and test set. Even when models do not obtain zero training error they can still overfit, and minimizing equal opportunity on the training set does not imply that it is optimized on the test set.• Failure to converge. To induce stability in the performance of Fairret we had to use a much larger minibatch size of 1000, as the minibatch must be large enough to estimate the EO violation somewhat stably if we want the method to converge. It is possible that either the batch size was still not large enough or that it was so large that it caused issues with optimizing the log loss.However, other issues might be the reason for the performance discrepancy.In this section, we show how OxonFair can be adapted through the use of custom groups to mimic the performance of Specialist Equalized Odds solvers. We compare against the recently published [57] , which was shown to outperform a wide range of existing methods. Like us, [57] assigns thresholds on a per-group basis to enforce fairness, and enforces fairness up to a prespecified threshold (e.g., equalized odds is less than 5%) while maximizing accuracy. Unlike the default behaviour of Oxonfair, however, it randomly assigns members of each group one of two different thresholds. To mimic this behaviour, we make use of the same trick used in Section A.1, namely that the 'inferred groups' used to assign thresholds are not expected to align with the true groups, and that we can introduce additional groups to increase the expressiveness of the model.Before showing how to enforce Equalized Odds, we strongly recommend that this is not used in practice. [57] also notes that enforcing it is contentious. Thresholding methods such as OxonFair and [57] can be understood as methods that trade-off sensitivity or recall against specificity. When Compared to the original predictor, we see a substantial decrease in recall for the worst performing group accompanied by a small increase in specificity. For the better performing group, both recall and specificity are decreased in order to enforce fairness. Right: The accuracy/fairness trade-offs of OxonFair using single threshold, deterministic multi-thresholds, and randomized multithresholds. Single threshold performs substantially worse for strong fairness constraints, but the other two strategies are interchangeable. All results shown are on validation data.enforcing Equalized odds, we will move to a new point on the sensitivity specificity curve, for the worst performing group, while degrading the performance for all other groups. Viewed through the lens of levelling up [8] , and which specific harms are incurred by each group, versus a base classifier: enforcing equalized odds will increase one of the sensitivity and specificity for the worst performing group and decrease the other; and may decrease sensitivity and specificity for all other groups. (see Figure Figure 11 for an illustration of the frontier found by OxonFair with respect to recall and specificity). While other methods are harder to analyze than thresholding, the fact that they have worse accuracy/fairness trade-offs suggests that they deteriorate classifiers more than simple thresholding.We consider four threshold sets for OxonFair.1. Default OxonFair using one threshold per group. 2. A randomized version that mimics the behaviour of [57] , by assigning members of each group to two subgroups with 50/50 probability. 3. As randomized approaches can be criticized for their instability, we also consider a deterministic version that flips the scores of positively labelled data points scored below a threshold, and also flips the score of negatively labelled points scored above a threshold. 4. An inferred variant that does the same as 3, but doesn't require access to groups at test time.Additional code for the last three options is shown below. These functions can be passed to FairPredictor using the inferred_groups option. Our comparison with [57] can be summarized as the randomized and deterministic methods are broadly interchangeable both with [57] (see Figure 12 ) and with each other (see Figure 11 right), and they strongly outperform the default single-threshold OxonFair for strong fairness constraints.Method [57] (single fairness eval) [57] (Complete Frontier) OxonFair Complete Frontier Runtime 3 sec 4m49.1 sec 1m18s Table 9 : Runtime comparison between [57] and OxonFair, on the Folktables dataset [101] , using four racial groups. This represents the largest problem with the most groups reported in [57] , and as such is the experiment where we would expect [57] to outperform OxonFair the most with respect to runtime. While [57] is faster if enforcing fairness to a known amount, e.g., maximizing accuracy subject to EOdds<0.05%, OxonFair remains faster for computing the entire frontier. 95% conf. interv.Figure 12 : A comparison between single-threshold OxonFair and deterministic multi-threshold OxonFair with [57] . As expected, Multi-threshold is directly comparable to [57] , with single threshold performing worse.As both [57] and multi-threshold OxonFair solve the same optimization problem via different routes (a specialist LP solver for [57] , grid-search for OxonFair) we expect them to obtain similar solutions, and this we find in Figure 12 . However, [57] makes use of a formulation that is more efficient for large numbers of thresholds, but less expressive and harder to adapt to infered group membership, or new fairness or performance constraints. As such, it is unsurprising that it is substantially more efficient than grid-search over the 8 thresholds used in these experiments -at least when finding a solution with e.g. an Eodds violation of no more than 0.05. What was more surprising was that when using the code of [57] to compute an entire fairness/accuracy Pareto frontier, Oxonfair remained more efficient. See Table 9 for details. This remains a reminder that big-O notation is uninformative with respect to runtime, when our datasets are extremely limited in their number of groups.As [57] requires access to groups at test time, we cannot directly compare with [57] , we instead compare against FairLearn on the adult dataset, using sex as the protected characteristic. Results can be seen in Table 10 , and Figure 13 .Our approach differs from [102] , as they enforce fairness with respect to inferred groups (without access to true group labels), while we directly use the inferred group labels to enforce fairness with respect to the underlying true groups.Original OxonFair Multi OxonFair + FairLearn Accuracy 0.871016 0.866921 0.859960 0.866512 Equalized Odds 0.097950 0.026676 0.025532 0.045989 Table 10 : A comparison of Fairlearn, and OxonFair when enforcing Equalized Odds. We enforce fairness at 2% on validation to roughly match accuracy with Fairlearn. At this value, there is limited change between versions of Oxonfair and the multi-threshold approach obtains half the fairness violation of Fairlearn at similar accuracy. Figure 13 : Enforcing Equalized odds using inferred characteristics. While the multi-threshold approach shows clear advantages on the validation set, this does not generalize to unseen test data. For unseen data, single threshold approaches show stronger degradation in accuracy, but their fairness constraints generalize better. This can be attributed to single threshold approaches selecting near constant classifiers when the constraints are strong, while the classifiers found by multi-threshold approaches are more vulnerable to sampling differences between validation and test. We extensively used the codebase of Wang et. al [41] to conduct comparative experiments 11 .• Empirical Risk Minimization (ERM) [103] : Acts as a baseline in our experiments where the goal is to minimize the average error across the dataset without explicitly considering the sensitive attributes. Table 17 : Multilingual Twitter corpus breakdown. We report the count of positive and total samples across the train/val/test partitions for each ethnicity with specific values. We exclude samples where the label is marked as 'None' for a particular ethnicity.We employ a BERT-based model architecture [86] , augmented with an additional head to simultaneously predict demographic factors (see Section 4.2. During training, we utilize the standard cross-entropy loss for the primary prediction task and a mean squared error loss for the demographic predictions, aggregating these to compute the overall loss. We ensure data consistency by excluding entries with missing demographic information. To facilitate easy comparison with different models, we select the Polish language for the multilingual Twitter corpus, noted for its high DEO score, to demonstrate how various models can reduce this score. We also conducted our experiment on the Jigsaw data. Unlike the multilingual Twitter corpus, the Jigsaw religion dataset contains three groups: Christian, Muslim, and others. The entire model, including the BERT backbone, is fine-tuned for 10 epochs using an initial learning rate of 2 × 10 -5 , following the original BERT training setup. All experiments are conducted on an NVIDIA A100 80GB GPU.We follow the methodology outlined in [84] to conduct the hate speech detection task using our tool.Variables such as age and country in the multilingual Twitter corpus are binarized using the same method as described in [84] . The data splits for training, development, and testing are shown in Table 16 .Multilingual Experiment. To demonstrate the capability of our proposed tool in handling multilingual scenarios, we conduct experiments across five languages: English, Polish, Spanish, Portuguese, and Italian and the results are shown in Table 18 . Observations from the results indicate that: 1) Our model improves equal opportunity performance with minimal sacrifice to the main task performance.2) The datasets in Polish and Portuguese show higher equal opportunity, indicating more severe bias compared to other languages, yet our proposed method effectively enhances performance in these conditions.In this section, we provide further details on the information from Figure 1 . While all approaches have many fairness definitions that can be computed, very few can be enforced via bias mitigation. As a minimum, OxonFair supports enforcing the methods from tables 5 and 6 (eliminating duplicates give the number 14 in the table). In addition to this, it supports a wide range of metrics that aren't used in the literature, for example minimizing the difference in balanced accuracy, F1 or Matthews correlation coefficient (MCC) between groups, e.g., by using balanced_accuracy.diff as a constraint. It also supports the definitions set out in Appendix C, including minimax notions; absolute bias amplification; and enforcing for minimum rates per group in recall, or precision, or sensitivity actively promoting levelling-up [8] .Fairlearn provides an overview of the supported bias mitigation algorithms and supported fairness constraints in their documentation 12 . The number of performance and fairness objectives supported are dependent on the method.Methods supported include ExponentiatedGradient and GridSearch that provide a wrapper around the reductions approach to fair classification of Agarwal et al. [31] . Supported fairness definitions for classification are Demographic-Parity, Equalized Odds, True Positive Rate Parity, False Positive Rate Parity and Error Rate Parity. For postprocessing the ThresholdOptimizer approach of Hardt et al. [3] is supported. The adversarial approach of [32] is also supported and can enforce fairness based on Demographic Parity and Equalized Odds. The CorrelationRemover method provides preprocessing functionality to remove correlation between sensitive features and non-sensitive features through linear transformations. It should be emphasized that Fairlearn also provides an interface for defining custom Moments for fairness and objective optimization, however, as of the current version 0.10 no documentation or examples are provided for doing so.AIF360 provides support for a wide variety of methods 1314 that enforce fairness, many of which overlap with Fairlearn. We consider group fairness approaches.Preprocessing algorithms include DispirateImpactRemover [11] , LFR [104] , Optimized Preprocessing [26] , Reweighting [25] and FairAdapt [105] . Inprocessing algorithms include Adversari-alDebiasing [32] , PrejudiceRemover [106] , Exponentiated GradientReduction and GridSearchReduction [31] . Postprocessing approaches include CalibratedEqOddsPostprocessing [44] , EqOddsPostprocessing [3] , RejectOptionClassification [25] .We reiterate the findings of Balayn et al., who note that fairness toolkits can act as a double-edged sword [92] . Open source toolkits can enable wider adoption of the assessment and mitigation of bias and fairness related harms. However, if misused, these toolkits can create a flawed certification of algorithmic fairness, endangering false confidence in flawed methodologies [55, 107] . We join growing calls in encouraging practitioners to be reflective in their use of fairness toolkits [60] . Specifically, we urge practitioners to adopt a harms first approach to fairness and be reflective in their measurement and enforcement of fairness.A maximal set of solutions such that for every element in the set, any solution with a better score with respect to the objective would have a worse score with respect to the constraint, and vice versa.In practice this is done twice, once in a coarse grid search to determine a good range, and then in a second finer search within the minimum and maximum range found by the first searchThe squared loss is used rather than log-loss so that the output of g(x) remains close to 0 and 1. With log-loss, the output pre-sigmoid is more likely to overwhelm confident decisions made by the original classifier.In some situations, a credible case can be made for including the mildest forms of augmentation, such as left-right flipping in the validation set, to maximize the validation set size.We provide example notebooks for practitioners to get started with DeepFairPredictor in our toolkit.This dataset is carefully curated and balanced. To induce unfairness we altered the sampling and dropped half the people recorded as male and that did not have medical complications across the entire dataset.Here OxonFair indicates the use of inferred group membership, while OxonFair* uses true group membership at test time.User controllable threshold.https://github.com/princetonvisualai/DomainBiasMitigationhttps://FairLearn.org/main/user_guide/mitigation/index.htmlhttps://aif360.readthedocs.io/en/stable/modules/algorithms.htmlhttps://aif360.readthedocs.io/en/stable/modules/sklearn.html 


Decision: The paper presents AnonFair, a new open-source toolkit for enforcing algorithmic fairness that supports NLP and computer vision classification tasks in addition to tabular problems. Such broader applicability to different data types is a key advantage over existing fairness toolkits.

AnonFair can enforce fairness on validation data, making it more robust to overfitting challenges compared to other approaches. It supports 9/9 and 10/10 of the group metrics from two prominent fairness review papers.

AnonFair is compatible with popular ML frameworks like scikit-learn, AutoGluon, and PyTorch.

**Strengths** Broader applicability to NLP and computer vision tasks compared to existing fairness toolkits, with more expressive and customizable fairness measures. It features improved robustness to overfitting by using validation data and competitive performance on standard fairness benchmarks

**Weaknesses**

Limited novelty in the underlying fairness techniques (mainly implements known methods). It currently only supports binary classification tasks
Scaling issues with larger numbers of protected groups



Overall,
this is a solid toolkit paper with useful contributions, particularly in extending fairness approaches to new domains like NLP and computer vision. The main concern is the limited technical novelty but the practical utility and engineering contributions outweigh this limitation for a toolkit paper.

Reviews: review: paper_summary: paper_summary: This paper presents AnonFair, a cutting-edge open-source toolkit designed to promote algorithmic fairness. Authors claim the following contributions:
(1) Comprehensive support for NLP and Computer Vision classification, as well as standard tabular problems.
(2) Enhanced robustness against overfitting challenges through the ability to enforce fairness on validation data.
(3) Versatility in optimizing any measure that is a function of True Positives, False Positives, False Negatives, and True Negatives, making it easily adaptable and more expressive than other toolkits.
(4) Seamless integration with popular ML toolkits such as sklearn, Autogluon, and pytorch.
(5) AnonFair supports 9/9 and 10/10 of the group metrics of two prominent review papers and is accessible online at no cost.
strength_weakness: strength_weakness: * Lack of clarity in some reported experiments, e.g. results tables are not cited in the text, metrics are not well-contextualized (e.g. larger or lower scores are better?)

* Lack of analysis, examples or human evaluation to better understand contributions and limitations of the method in each of the experiments.
questions: questions: (1) Could you provide more high-level context for each of the experiments that you are running in order to make the paper more self-contained?
(2) for NLP experiments, why do you think mitigation works for Twitter and not for Jigsaw?
limitations: limitations: Authors report some limitations, but further analysis on the experiments could raise more limitations that may be currently ignored.
paper_summary: paper_summary: The paper describes details of a fairness toolkit ("AnonFair"), which confers fairness to any given machine learning classifier by exploring a wide range of prediction thresholds for different groups (which are either provided upfront or inferred through an auxiliary classifier). The toolkit is designed to be quite expressive, as it can optimize several different metrics, e.g., false positives/negatives, true positives, etc. The toolkit can work across all classifiers (which can output class probabilities), including ones trained on vision and NLP tasks.
strength_weakness: strength_weakness: While the toolkit might turn out to be useful for some practitioners, it is a relatively straightforward implementation of well-known (and simple) technique of adjusting prediction thresholds across groups. Exploring different thresholds can be computationally prohibitive, for which the authors use a standard trick to speed up their explorations (which I appreciate). The paper acknowledges and cites relevant papers/techniques that they implement.   Overall, the originality and novelty of their work is significantly limited, as the toolkit is an implementation of known and simple fairness techniques. Further, the underlying fairness techniques (not from the authors) are themselves applicable to most classifiers, so any implementation of the same could work for NLP and vision tasks—which is claimed to be one of the major contributions of this work.
questions: questions: I feel that the current version is a good starting point (in terms of implementation) of existing fairness techniques and speeding them up and trying them out on vision and NLP tasks. To improve the paper, I would suggest clearly outlining the important problems that this toolkit now can enable researchers to answer (which was not possible before) and answer a few of those questions in the paper.
limitations: limitations: I believe the paper adequately communicates their shortcomings and cites past references when using them. However, I think it might help to also acknowledge that the underlying fairness techniques broadly apply to a wide range of classifiers, and naturally extend to classifiers in computer vision and NLP domains. Reading parts of the paper felt like that there are significant challenges in adoption of fairness techniques to NLP and CV, and this paper overcomes them through novel solutions—which is not the case.
paper_summary: paper_summary: The paper introduces a new toolkit designed to enhance algorithmic fairness with greater expressiveness. Unlike existing toolkits, this one offers more customization options to optimize user-defined objectives and fairness constraints. Although the proposed toolkit currently includes only one method, it supports both computer vision and natural language processing (NLP) tasks. The authors compare the efficiency of this method, finding that the toolkit is relatively more efficient than Fairlearn. Comprehensive experiments were conducted on various datasets, and the results were compared with those from other popular toolkits.
strength_weakness: strength_weakness: - The formulation presented in Subsection 4.2 of the paper is limited to a single-layer model, which restricts its applicability across different machine learning models. To enhance the flexibility of the method, I recommend adopting a more generic notation, particularly if we aim to incorporate pretrained language models.
- The abstract is quite unclear, especially the part that mentions "9/9 and 10/10 of the group metrics of two popular review papers." I suggest rephrasing the abstract for better clarity and comprehension.
questions: questions: - In Figure 3, the proposed toolkit appears to encounter scaling issues when reaching 5 groups. Could you provide more details on why this occurs and elaborate on the underlying reasons for this limitation?
- The paper presents results on multilingual datasets. Do you have any specific findings for each language, particularly regarding the effectiveness of the toolkit for individual languages?
limitations: limitations: Yes
paper_summary: paper_summary: This paper describes a new toolkit for algorithmic fairness, enabling the optimization of any fairness measure that is a function of the confusion matrix. Experiments on vision and NLP demonstrated the effectiveness of the proposed toolkit.
strength_weakness: strength_weakness: Presentation could be made more self-contained, e.g. a table listing the supported fairness metrics, as functions of the confusion matrix. This would help readers not familiar with the field.

It seems that only binary classification is supported. How can such metrics be extended to other tasks?

Some minimal code snippets for the interface could be shown as examples.
questions: questions: - L5: "True positives, false positives, ..." => "the confusion matrix"
 - L6: "extendable" => "extensible"
limitations: limitations: The authors adequately discussed the limitations of their toolkit.
paper_summary: paper_summary: The paper introduces "AnonFair," a toolkit designed to enforce algorithmic fairness across various domains, including NLP, computer vision, and traditional tabular data. It is compatible with popular machine learning frameworks like sklearn, AutoGluon, and PyTorch. Unlike well-established fairness tools like FairLearn and AIF360, AnonFair extends to different types of data, including NLP and vision.

Other tools offer many methods but limited control over them, while AnonFair uses a single, highly customizable method that allows for per-group thresholding.

It specifically addresses the issue of overfitting by utilizing validation data, making it more reliable when traditional methods might fail.

Empirical evidence presented shows that AnonFair performs well, often matching or surpassing other methods in fairness benchmarks without being specifically optimized for complex or high-dimensional scenarios.

AnonFair seems to provide a robust and adaptable solution for implementing fairness in machine learning, in ways that other tools do not currently offer.
strength_weakness: strength_weakness: - Some sections are overly detailed, such as the introduction, while others are missing necessary depth:
    - Section 3 could use a clearer structure, possibly with a diagram, to help readers understand how to interact with the toolkit.
    - The section on toolkit expressiveness needs more detailed examples and explanations of how the supported fairness measures are implemented. 
    - Results discussion is kept very brief and could benefit from specific numerical examples, like percentage improvements compared to other methods.m actual numbers, such as how much % improvement in comparison to method XY and such.

- The paper assumes readers are familiar with fairness terminology and metrics without adequate explanations or definitions for some acronyms (e.g., DEO in Table 3 and 4).
    - Subsection 4.3 lists supported fairness measures but fails to provide examples or brief explanations, making it less informative for those not familiar with these terms.

- Lack of consistency in terminology usage; for example, "EOp" in Figure 1 (top right) vs. "EO" in Section 5.2, “AnonFair” missing before "Frontier" in Figure 1 (left), and inconsistent references like "See Figure" vs. "See fig.."

- A stronger call to action for community engagement, such as through open-source collaboration or empirical validation studies, could significantly enhance the broader impact and encourage more widespread adoption and refinement of AnonFair.

- The paper would benefit from a summary of explicit cases and recommendations advising users on the best scenarios for using the tool.

- Figure 2 is not referred to in the paper, or did I miss this part.
questions: questions: 1. The paper mentions that hard assignment is more efficient than soft assignment, while appendix A adds some operational details, it remains unclear how these methods specifically compare in terms of quantitative metrics. Could the authors provide specific metrics or comparisons that demonstrate the efficiency and performance benefits of hard assignment?
2. The discussed limitations reads a bit out of context given provided evidence in the paper. What makes the mentioned  solutions suboptimal, and how significant are these shortcomings?  Also it was not clear to me, after finishing reading,  when it is adequate to use this tool and what could be use cases when it fails. Including this into the conclusion could make the reader grasping the full picture. 
3. Is Figure 6 part of the Appendix or misplaced?
limitations: limitations: Some of the limitations are acknowledged, but could be expanded with more actionable insights. 
A call to action for community engagement, such as through open-source collaboration would also encourage broader impact and adoption of AnonFair against its competitors. 
It would be beneficial if the authors suggested potential improvements or future research directions for the suboptimal fairness metrics and data scarcity issues mentioned.
The broader impact section identifies ethical concerns well. However, detailing the intended applications and scenarios where AnonFair might be most effective, or where it could fail, would provide readers and users with clearer guidance on its practical use and limitations.

Comments: title: Any follow-up clarifications?
Any follow-up clarifications?
Any follow-up clarifications?
Response
comment: Thanks a lot for your responses, it clarifies my questions, I will keep my score.
We thank the reviewer for their helpful feedback.
 
In line with the reviewers recommendation, we will expand the limitations section to acknowledge and discuss the issue of binary annotations for gender that are common in many popular datasets in algorithmic fairness.
Thanks for the detailed response. I have modified the score based on the appendix. Thanks.
I appreciate the authors' detailed response to the concerns raised and the additional experiments. 
Given the constraints of a conference paper, I agree that the toolkit's documentation may be better suited for the extensive details. 
With the reviewers' feedback incorporated, the paper should be better positioned to convince practitioners to try out the toolkit. Good luck. 
I will maintain the current score.
We hope that we have addressed all issues raised to your satisfaction in our rebuttal. We would be happy to provide additional clarifications if required as the discussion period will be over soon.

Thank you for your time.
We hope that we have addressed all issues raised to your satisfaction in our rebuttal. We would be happy to provide additional clarifications if required as the discussion period will be over soon.

Thank you for your time.
We hope that we have addressed all issues raised to your satisfaction in our rebuttal. We would be happy to provide additional clarifications if required as the discussion period will be over soon. 

Thank you for your time.
We thank the reviewer for the positive feedback and encouragement in that our toolkit makes *“good progress towards bias and fairness in ML”*. We are also happy to see that the reviewer feels that ethical issues are acknowledged in the paper. 
 
Regarding the recommendation, does the reviewer expect us to carry out this experiment in the final version of the paper or is it a suggestion for future work?  We believe that both ethnicity and the color people identify as are, in many ways, social constructs, and we are uneasy about using auto-generated labels for images as ground truth in a quick additional experiment.
Thanks for your response. I agree with the authors that their toolkit also supports cases when group annotations are unavailable. I realize that my evaluation might not have properly taken this into account. In that light, I have increased my assessment score about the contribution from poor to fair, and overall assessment from 3 to 4. 

I'm quite aware of the fact that NeurIPS CFP invites libraries as contributions, but that doesn't take away my concerns about the (lack of) novelty and originality of the underlying techniques implemented in the library.
Thank you for the response and plan to improve the paper.

I will keep my scores.