OxonFair: A Flexible Toolkit for Algorithmic Fairness
Authors: Eoin D. Delaney
Zihao Fu
Sandra Wachter
Brent Mittelstadt
Chris Russell
Venue: NeurIPS 2024 poster (2024)
Keywords: Fairness Toolkit
Algorithmic Fairness
Trustworthy AI
Primary area: infrastructure
TL;DR: We present a new toolkit for enforcing and measuring fairness with a focus on deep learning models.

Abstract: We present OxonFair, a new open source toolkit for enforcing fairness in binary classification. Compared to existing toolkits: (i) We support NLP and Computer Vision classification as well as standard tabular problems. (ii) We support enforcing fairness on validation data, making us robust to a wide range of overfitting challenges. (iii) Our approach can optimize any measure based on True Positives, False Positive, False Negatives, and True Negatives. This makes it easily extensible and much more expressive than existing toolkits. It supports all 9 and all 10 of the decision-based group metrics of two popular review articles. (iv) We jointly optimize a performance objective alongside fairness constraints. This minimizes degradation while enforcing fairness, and even improves the performance of inadequately tuned unfair baselines. OxonFair is compatible with standard ML toolkits, including sklearn, Autogluon, and PyTorch and is available  at https://github.com/oxfordinternetinstitute/oxonfair.

OxonFair: A Flexible Toolkit for Algorithmic Fairness
Eoin Delaney University of Oxford
Zihao Fu University of Oxford
Sandra Wachter University of Oxford
Brent Mittelstadt
Chris Russell University of Oxford
University of Oxford
firstname aasstammmooiiiiii ox. ac. uk
Abstract
We present OxonFair, a new open source toolkit for enforcing fairness in binary classification. Compared to existing toolkits: (i) We support NLP and Computer Vision classification as well as standard tabular problems. (ii) We support enforc- ing fairness on validation data, making robust wide range of overfitting challenges. (iii) Our approach can optimize any measure based on True Posi- tives, False Positive, False Negatives, and True Negatives. This makes it easily extensible and much more expressive than existing toolkits. It supports all 9 and all 10 of the decision-based group metrics of two popular review articles. (iv) We jointly optimize performance objective alongside fairness constraints. This minimizes degradation while enforcing fairness, and even improves the per- formance of inadequately tuned unfair baselines. OxonFair is compatible with standard ML toolkits, including sklearn, Autogluon, and Py Torch and available at tiggth.h...... hom.tttooooooooooeooooooottotttt http://ooooottteeeeeooeeeeeeeeee inteetiiiiitittt tute/ oxonfaii
Introduction
The deployment of machine learning systems that make decisions about people offers an opportunity to create systems that work for everyone. However, such systems can lock in existing prejudices. Limited data for underrepresented groups can result in ML systems that do not work for them, while the use of training labels based on historical data can result in ML systems copying previous biases. As such, it unsurprising that AI systems have repeatedly exhibited unwanted biases towards certain demographic groups in wide range of domains including medicine [1] finance [3 4. and policing [5] Such groups are typically identified with respect to legally protected attributes, such as ethnicity or gender 6
3
The field of algorithmic fairness has sprung up in response to these biases.
Contributions algorithmic fairness can broadly be split into methodological and policy-basee -baee approaches. While much methodological work focuses on measuring and enforcing (un)fairness, common criticism from the policy side is that this work can occur 'in isolation from policy and civil societal contexts and lacks serious engagement with philosophical, political, legal and economic theories of equality and distributive justice' g
In response to these criticisms, we have developed OxonFair, a more expressive toolkit for algorithmic faireess. We acknowledge that people designing algorithms not always the right people to decide on policy, and as such we have chosen to create as flexible toolkit as possible to allow policymakers and data scientists with domain knowledge identify relevant harms and directly alter the system behaviour to address them. Unlike existing Fairness toolkits such as AIF360 (9]. which take a method-driven approach, and provide access wide range of methods but with limited control over their behaviour, we take a measure-based approach and provide one fairness method that is extremely customizable, and can optimize usev-provided objectives and group fairness constraints.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Clooificr (dazno))
Patition
Farkem
OuusFair
Frontier Found
Aoc
(1)
DBU
60
Kandom
81%
09
1.19
1.5
(edull)
Teet
7.9%
86%
1.3%
XCiBooi
Yaun
107%
09
9024
0.89
Inyocardial iitrrcson
Test
899
11,85
875
2.99
0.70
Citaia
AIF360
Faideam
OwnFair
Number of medhods
0.63
Adjustblle Faimess Critria
Sugports Groupe
Upkdes prenete
Rondsting
Faimess detiations eaforeed per method
<4
Orgra conctior
Calk anikk kestoods IHe
Mechods needing groups at
Some
Gredeach Recuction
Supports Uniliy
0.5
Lag,n9
Disprstemp acceeror
Supports Tabular
Supports Computer Vision
Supports
0.5
0.09
Damoggahhic
The need for an objective when enforcing fairness. We evaluate range of methods with respect to balaneed accuracy and demographic parity (OxonFair generates a frontier of solutions). Only OxonFair and RejectOptimization optimize balanced accuracy. As we improve the balanced accuracy of fair methods by adjusting classification thresholds (gray lines) fairness deteriorates. To avoid this, we jointly optimize fairess measure and an objective. For more examples, see Figure 6 Right Top: Using validation data fairness. We compare against Fairlearn using standard algorithms with default parameters. These methods perfectly overfit and show no unfairness with respect to equal opportunity on the trainset, but substantial unfairness on test. OxonFair enforces fairness on held-out validation data and is less prone to overfitting. Right Bottom: A comparison of toolkits. AIF360 offers a large range of tabular methods, most of which do not allow fairness metric selection, Fairlearn offers fewer but more customizable tabular methods. OxonFair offers one method that can be applied to text, image, and tabular data, while supporting more notions of faimess and objectives.
Figure 1: Left:
To do this, we focus on one of the oldest and simplest approaches to group fairness: per- per-group thresholding TO 11 3], which is known to be optimal for certain metrics under range of assump- tions 12 B
Our contribution is to make this as expressive as possible while retaining speed, for the relatively low number of groups common in algorithmic fairness. Inherently, any approach that allows a sufficiently wide set of objectives, and sets per-group thresholds will be exponential with respect the number of groups, but we use a standard trick, widely used in the computation of measures such as mean absolute precision (mAP) to make this search as efficient as possible. Accepting the exponential complexity allows us to solve much wider range of objectives than other toolkits, including maximizing Fl or balanced accuracy (see Figure [[fft), minimizing difference in precision [H and guaranteeing that the recall is above k% for every group [8. Where groups are unavailable at test time, we simply use a secondary classifier to estimate group memberships [15] [6] and set different thresholds per inferred group to enforce fairness with respect to the true groups.
Thresholding can be applied to most pretrained ML algorithms, and optimal thresholds can be selected using held-out validation data unused in training. This is vital for tasks involving deep networks such as NLP and computer vision, where the training error often goes to zero, and fairness methods that balance error rates between groups cannot generalize from constraints enforced overfitting training data to previously unseen test data [7]. While overfitting is unavoidable in vision and NL.P tasks, is still concern on tabular data. Figure Top-Righ shows examples of decision trees, random forests (18) and XGBoost [19] trained with default parameters and obtaining 0 training error on standard datasets. This causes the Fairlearn reductions method 20 to fail enforce fairness.
NLP and vision are so challenging that two popular toolkits Fairlearn and AIF360 do not attempt to work in these domains. In contrast, target them, making of recent work (2I] that showed how fair classifiers based on inferred group thresholds can be compressed into a single network.
2 Related Work
Bias mitigation strategies for classification have been broadly categorized into three categories 6 22424: prepprocsssing in inoprccessing and post-processing.

---

Pre- Pre-processing algorithms improve fairness by altering the dataset in an attempt to remove biases such as disparate impact [TT before training a model. Popular preprocessing approaches include
simply reweighting samples in the training data to enhance fairness 25), optimizing this process by
learning probabilistic transformations [26], or by generating synthetic data (27 29].
In-processing/ / In-training methods mitigate bias by adjusting the training procedure. Augmenting the loss with fair regularizers [23 30 is common for logistic regression and neural networks. Agarwal et al. [3T iteratively alter the cost for different datapoints to enforce fairness on the train set. Approaches based on adversarial training typically learn an embedding that reduces an adversary's 's ability to recover protected groups whilst maximizing predictive performance [32 [35. Other popular approaches include Disentanglement [36] Domain Generalization [38] 40] [4]] and simple approaches such as up-sampling or reweighing minority groups during training. Notably, the case of high-capacity models in medical computer-vision tasks, a recent benchmark paper by Zong et al. 42 showed state-of-the-art in in-processing methods do not significantly improve outcomes over training without consideration of fairness. comprehensive benchmark study of in-processing methods in other domains is provided by Han et al. [43]
post-processing methods enforce fairness by using thresholds and randomization to adjust the predictions of a trained model based on the protected attributes [ 44]. Post-processing methods are typically model-agnostic and can be applied to any model that returns confidence scores.
Enforcing Fairness on Validation Data avoids the misestimation of error rates due to overfitting. It has shown particular promise in computer vision through Neural Architecture Search 455) adjusting decision boundaries 30 reweighting [46] and data augmentation [17]
2.1
Fairness Toolkits
Most toolkits such as Fairness Measures 47 TensorFlow Fairness Indicators (48. and FAT Forensics focus on measuring bias and do not support enforcing fairness through bias mitigation. FairML.
49
50
audits fairness by quantifying the importance of different attributes in prediction. This is best
suited for tabular data where features are well-defined. FairTest [51] investigates the associations between application outcomes (e.g., insurance premiums) and sensitive attributes such as age to highlight and debug bias in deployed systems. Acquitas (52) provides examples of when different measures are (in)appropriate with support for some bias mitigation methods in binary classification. Themis-ML [53] supports the deployment of several simple bias mitigation methods such as rela- belling 225, but focuses on linear models. Friedler et al. [22 introduce the more complete Fairness Comparison toolkit where four bias mitigation strategies are compared across five tabular datasets and multiple models (Decision trees, Gaussian Naïve Bayes, SVM, and Logistic Regression).
There are two fairness toolkits that support sklearn 18] like OxonFair. These are the most popular toolkits: Microsoft Fairleam [20] (1.9k GitHub Stargazers as of June 2024) and IBM AIF360 [9] (2.4k Stargazers). AIF360 offers diverse selection of bias measures and pre-processing, in- processing and post-processing bias mitigation strategies on binary classification tabular datasets. For mitigation, Fairlearn primarily offers implementations of [31 avoiding the use of the term bias, instead considering faimess through the lens of fairnss--relate harms [54] where the goal is to "help practitioners assess caimess--rlatted harms, review the impacts of different mitigation strategies and make trade-offs appropriate to their scenario". Lee & Singh (55) recognized Fairlearn as one of the most user-friendly fairness toolkits, and critiqued AIF360 as being the least user-friendly tooliit.
Both ATF360 and Fairlearn contain post-processing methods that select per-group thresholds. Unlike OxonFair, neither method uses the fast optimization we propose; both methods require group information at test time; AIF360 only supports two groups, but does use cross- cross-valddation to avoid overfitting; Fairlearn does not support the use of validation data, but does support more than two groups. According to their documentation, neither toolkit can be applied to NLP or computer vision.
Specialist solvers Fairret [56] is recent Py Torch library shown to enforce faimess on tabular data. As PyTorch is a focus of OxonFair (see Section 4.2), we compare with Fairret Appendix D.I] Cruz and Hardt [57] proposed an efficient LP-based formulation for post-processing Equalized Odds. It supports randomized thresholds, and dominates fairness methods such as OxonFair that use only one threshold per group. In Appendix D.2
we show how OxonFair can be extended to support randomized
3

---

And grne Gruu and sorted dasster responss
Head T: Logeloss Flandller
Head 2: Squared Group Precictor
Cammustre Sums
culputs
Aroup Group
Resigred Group a encrooee
THE
Posilire
Resulting For Clocsieer
Sample
uy + iing....
Negative Sampla
wi +AN
64 02 Output Values from the Squared Loss
Figure 2: Left: Summary of the fast path algorithm for inferred attributes (Section 4.1 Groups are noisily estimated using classifier. Within each estimated group, we cumulatively sum positive and negative samples that truly belong to each group. For each pair of thresholds, we select relevant sums from the inferred group and combine them. See Appendix A.1 Center: Combining two heads (original classifier and group predictor) create fair classífier. See Section 4.2) Right: The output of second head predicting the protected attribute in CelebA. The pronounced bimodal distribution
makes the weighted sum of the two heads close replacement for per-group thresholds.
thresholds, alongside determenistic variant and another using inferred group membership that are not supported by [57]
3 Toolkit interface
The interface of OxonFair decomposes into three parts: (i) evaluation of fairness and performance for generic classifier outputs. (ii) evaluating and enforcing fairness for particular classifiers. (iii) specialist code for evaluating and enforcing fairness for deep networks.
Code for the evaluation of classifier outputs takes target labels, classifier outputs, groups, and an optional conditioning factor input; while code for the evaluation and enforcement of fairness of particular classifier, are initialized using the classifier, and from then on take datasets (in the form of a pandas dataframe [58), a dictionary) input, and automatically extracts these factors from them.
The evaluation code provide three functions: evaluate which reports overall performance of the classifier; warlluate_perrgeee pewp-rrruupppppp which reports performance per group of the classifier; and evaluate evaluate_fairness fairness which reports standard fairness metrics. All methods allow the user to specify which metrics should be reported. We recommend data scientists focus on evaluate __raate_err____ _er__rrouppppppppppppppppppppppp -group which shows direct harms such as poor accuracy, precision, or low selection rate for particular groups.
OxonFair provides an interface FairPredictor ceator(classe (classifier, validation dala,,aaaa groups) that takes existing classifier as input, a validation dataset, and specification of the groups as an input and returns an object which we then enforce fairness on by calling .iit (objective constraint, value). Internally, the method explores wide range of possible thresholds for each group, membership of which is assumed to be either known or inferred by an auxiliary classifier.
The resulting FairPredictor has evaluation methods described above. When called without arguments, they report both the performance of the original and the updated fair classifier on the validation data. In addition, FairPredictor provides methods predict and predict_prote which make fair predictions and return scores corresponding to the left-hand side of Equation (I.
Calling fit optimizes the objective typically relevant performance criteria such as accuracy, subject to the requirement that the constraint is cither greater or less than the value. If the objective should be minimized maximized is inferred automatically, as is the requirement that the constraint less than or greater than the value, but this default behavior can be user overridden.
This is relatively minimal interface, but one that is surprisingly expressive. By explicitly optimizing an objective, we can not just minimize the degradation of the metric as we enforee fairness, but sometimes also improve performance over the unfair baseline that is not fully optimized with respect to this metric. Even when optimizing for accuracy, this can create situations where it looks like some improvements in fairness can be had for free, although generally this is artifact of the gap between optimizing log-loss and true accuracy training. By formulating the problem as generic constrained optimization, and not requiring the constraint to be typical faimess constraint, we leave it open

---

for enforcing a much broader space of possible objectives. This can be seen in Appendix C where we show how to enforce minimax faimess [59], maximize utility [60] combined with global recall constraints, and demonstrate levelling-up g by specifying minimum acceptable harm thresholds.
Under the hood, call to fit generates Pareto frontie@aand selects the solution that best optimizes the objective while satisfying the constraint. The frontier can be visualized with plot_frontier.
4 Inference
To make decisions, we assign thresholds to groups. We write f(x:) for the response of a classifier f. on datapoint v; z for the vector corresponding to the ordered set of thresholds, and G(x) for the
one-hot encoding of group membership. We make a positive decision if
f(x) t- G(z) 2 0
(I)
To optimize arbitrary measures we perform grid search over the choices of threshold, t.
Efficient grid sampling We make use of a common trick for efficiently computing measures such as precision and recall for a range of thresholds. This trick is widely used without discussion for efficient computation of the area under ROC curves, and we have had trouble tracking down an original reference for it. As one example, it is used by scikit-learn [18].
The trick is as follows: sort the datapoints by classifier response, then generate a cumulative sum of the number of positive datapoints and the number of negatives, going from greatest response to least. When picking threshold between points and + 1, TP is given by the cumulative sum of positives the decreasing direction up to and including i; FN is the sum of negatives in the same direction; FP is the total sum of positives minus TP, and TN is the total sum of negatives minus TN.
We perform this trick per group, and efficiently extract the TP, FN, FP and TN for different thresholds. These are combinatorially combined across the groups and the measures computed. This two stage decoupling offers substantial speed-up. If we write T for the number of thresholds, for the number of groups, and n for the total number of datapoints, our procedure is upper-hounded by O + n log n),, while the naïve approach is (nT*). No other fairness method makes use of this, and in particular, all the threshold- methods offered by AIF360 make use of a naïve grid search.
From the grid sampling, we extract Pareto frontier with respect to the two measure The thresholds that best optimize the objective while satisfying the constraint are returned as a solution. If no such threshold exists, we return the thresholds closest satisfying the constraint.
4.1 Inferred characteristics
When using inferred characteristics, we offer two pathways for handling estimated group membership. The first pathway we consider makes hard assignment of individuals to groups, based on a classifier response. The second pathway explicitly uses the classifier confidence as part of a per-datapoint threshold. In practice, we find little difference between the two approaches, but the hard assignment to groups is substantially more efficient and therefore allows for a finer grid search and generally better performance. However, the soft assignment remains useful for the integration of our method with neural networks, where we explicitly merge two heads (a classifier and a group predictor) of a neural network to arrive at a single fair model. For details of the two pathways see Appendix A
4.2 Fairness for Deep Networks
We use the method proposed in 21 (N.B., they used it only for demographic parity). Consider network with two heads f. and 9. comprised of single linear layers, and trained to optimize two tasks common backbone B. Let f be standard classifier trained maximize some notion of performance such as log-loss and g is a classifier trained to minimize the squared los
B with respect to
'A maximal set of solutions such that for every element in the set, any solution with a better score with respect to the objective would have a worse score with respect to the constraint, and vice versa.
In practice this done twice, once in coarse grid search to determine a good range, and then in second finer search within minimum and maximum range found by the first search
'The squared loss is used rather than log-loss so that the output of g(x) remains close to 0 and 1. With log-loss, the output pre-sigmoid is more likely to overwhelm confident decisions made by the original classifier.
5

---

Compas Logistic Regression (nnerred)
#Groups
Accuracy (1)
Dem. Par. (U)
Time
FairLeam
86.5%
3.8%
47.0s
Oxonl'air
86.9%
1,9%
0.6+ 42.9s
0.60
OxonFair(S) 5
0.35
FairLeam
86.7%
1.6%
28.4s
OxonFair
86.8%
1.2%
0.6+0.79
0.50
OxonFair(S))
0.6+41 Is
Die mne
Faim
86.5%
0.7%
25.0c
Diariii Fooder
ADverert declading
OxonFair 3
86.8%
2.1%
0.6+ 0.07s
OxonFair(S))
0.6+ 22.4s
Calormecki oadica
Log
FairLeam
86.9%
0.2%
20.0s
0.00
0.05
0.10
0.15
0.20
6.25
0.30
OxonFFair OxonFair(S) 2
86.9%
0.3%
0.6+0.04, 0.4s
Equal Opporturity Volatoo
0.6+1.22 1.24
Figure 3: Left: Results on Compas without using group annotations at test time. Right: Runtime Comparison for Fairlear Reductions and OxonFair Adult using Macbook M2. To alter the groups, we iteratively merge the smallest racial group with 'Other', reducing the search space. For both methods, we enforced demographic parity the exponential complexity of our approach, we remain significantly faster until we reach 5 groups. The 0.6+ indicates the seconds to train XGBoost. OxonFair(S) indicates the runtime of the naive slow pathway described in Appendix A.2 rather than our accelerated approach.
train set consisting of 70% of the data. Despite
a vector that is one hot-encoding of group membership. Any decision f(x) - t. g(x) 2 0 can now be optimized for given criteria by tuning weights w using the process outlined in the slow pathway.
As both f and g are linear layers on top of common nonlinear backbone B, we can write them as: g(z) = "a Bxx) + by
f(x) = wy Bxx) + by.
(2)
note that as fx) is a real number, and g(x) is a vector, yy is vector and by a real number, while Wy is matrix and by vector.
This means that the decision function f(x) - t g(x) 2 0 can be rewritten using the identity:
f(x) t g(x) = w ((x) + by - -t w B(x)= - t. by
(3)
(w. t wgy) B(x) + (b, tt. t by)
(4)
This gives a stage process for enforcing any of these decision/fairess criteria for deep networks.
1. Train a multitask neural network as described above.
2. Compute the optimal thresholds on held-out validation data as described in Appendix A
3. Replace the multitask head with neuron with weights (wy t. wgy and bias (by t by).
To maximize performance, the training set should be augmented following best practices, while, to ensure fairness, the validation set should nof] The resulting network f will have the same architecture as the original non-multitask network, while satisfying chosen criteria.
OxonFair has distinct interface for deep learning Training and evaluating NLP and vision frequently involves complex pipelines. To maximize applicability, we assume that the user has trained a two-headed network described above, and evaluated on a validation set. Our constructor DeepFai weeFFarrreiiccoco ctor takes as an input: the output of the two-headed network over the validation set; the ground-truth labels; and the groups. fit and the evaluation functionality can then be called in the same way. Once solution is selected, the method merge_ herge_heads htadd__pytorrreeeeeeeeepeeesesee pytorch generates the merged head, while extract co ieffiiient ients can be called to extract the thresholds /. from 4 when working with different framework.
4.3 Toolkit expressiveness
Out of the box, OxonFair supports all 9 of the Decision-basedd group faimeess measures defined by Verma and Rubin [6]] and all 10 of the fairness measures from Sagemaker Clarify 62 OxonFair
'In some situations, a credible case can made for including the mildest forms augmentation, such as righ- flipping in the validation set, to maximize the validation set size. We provide example notebooks for practitioners to get started with DeepPaixPredictor in our toolkit.

---

Table 1:
We report mean scores the 14 gender independent CelebA labels 28 Single task methods and FairMixup scores in the second and third blocks are from Zietlow et al. 17 ERM is the baseline architecture run without fairness. OxonFair (optimizing for accuracy and difference in equal opportunity (DEO),, has better accuracy (1) and DEO (L) scores than any other fair method.
ERM Uniconf.
Domain
Domain OxonFaiir
ERM
Debiasing Regularized
g-MOTTE
SSMOTT
ERM
FairMixup
multitask Adv. (24 Disc. [75
Ind. 41
DEO
single task GAN 76
Adaptive [77
[78]
[7]
Acc. DEO
93.07
92.71
92.96
92.63
92.75
92.47
92.12
91.05
92.56
92.64
92.74
88,46
16.47
19.63
14.61
7.78
3.21
12.54
9.11
3.77
14.28
15.11
7.97
3.58
supports any faireess measure (including conditional faireess measures) that can be expressed per group as a weighted sum of True Positives, False Positives, True Negatives and False Negatives. OxonFair does not support notions of individual fairness such as fairness through awareness (33 or counterfactual fairness [64] 65]
See Appendix B for discussion of how metrics are implemented and comparison with two review papers. Appendix C contains details of non-standard fairees metrics, including utility optimization
[60] minimax fairness [59 66 67]; minimum rate constraints [8] and Conditional Demographic
Parity 68 This also includes a variant of Bias Amplification 66 70].
5
Experimental Analysis
For tabular data, we compare with all group fairness methods offered by AIF360, and the reductions approach of Fairlearn. OxonFair is compatible with any learner with implementation of the method predict_prod consistent with scikit-learn [8] including AutoGluon (71) and XGBoost [9]. A comparison with Fairlearn and the group methods from AIF360 the adult dataset can be seen in Figures D [add O using random forests. This follows the setup of 19: we enforce fairness with respect race and binarize the attribute white vs everyone else (this required to compare with AIF360), 50% train data, 20% validation, and 30% test, and a minimum leaf size of 20. With this large leaf size, all errors on train, validation, and test are broadly comparable, but our approach of directly optimizing an objective and a fairness measure leads us to outperform others.
Figure D top right shows the importance of being able to use validation set to balance errors. Using sklearn's s default parameters we overfit to adult, and as the classifier is perfect on the training set, all fairness metrics that match error rates are trivially satisfied (72 17 The same behavior can be observed using XGBoost on the medical dataset
73 when enforcing equal opportunity with respect to sex In general, tabular methods need not overfit, and tuning parameters carefully can allow users to get relatively good performance while maintaining error rates between training and test.
Figure left shows Equal Opportunity on the COMPAS dataset. To show that OxonFair can also work in low-data regimes where we have insufficient data for validation, we enforce faimeess on the training set. As before, binarize race to allow the of AIF360. We drop race from the training data, and use inferred protected attributes to enforce faimesss. Here OxonFair generates a frontier that is comparable or better than results from existing toolkits, and OxonFair+ (see Section (A). further improves on these results. See Figure 3 right for a comparison with Fairlearn varying the groups.
5.1
Computer Vision and CelebA
CelebA 77 We use the standard aligned & cropped partitions frequently used in fairness evaluation Following Ramaswamy et al. (28), we consider the 26 gender-independe gender- dependent and inconsistently labelled attributes as the target attributes for our evaluations (see Table 12 for details). Male is treated as the protected attribute.
[17
21
28 4T
Implementation Details We follow Wang et al.'s setup 41. We use a Resnet-50 backbone 30
trained ImageNet [1). multitask classification model is trained, replacing the final fully- connected layer of the backbone with a separate fully-connected head that performs binary prediction for all attributes. Dropout 82 (p 0.5) is applied. All models are trained with batch size of
"This dataset is carefully curated and balanced. To induce unfairness we altered the sampling and dropped half the people recorded male and that did not have medical complications across the entire dataset.
7

---

Uniconf. Adv 74
Domain Disc.
Domain Ind. 41
OxonFair DEO
Table 2: A comparison against standard vision approaches on the more challenging CelebA at- tributes. OxonFair continues to work well here. All methods share common backbone and training process. An extended version of this table that con- siders minimax fairness can be found in Table 14
ERM
Gender- Gender-Dependent Attributes
Acc. (1)
86.7
86.1
86.6
85.6
85.8
DEO (1)
26.4
25.0
3.92
Inconsistently Labelled Attributes
Acc. (T)
83.0
82.5
83.1
82.3
82.1
DEO (1)
21.9
29.1
25.3
17.2
2.36
Pareto Frontier: Blond Hair
0.9
CelebA
8 8= 0.50
8= 0.75
8 = 0.90
Baseline
89.0
84.5
77.6
Accuracy
6.7-
Adversarial
87.8
82.4
75.2
Domain-Dep
82.3
76.8
68.6
Baseline (ERM)
Domain-Ind
89.2
86.2
79.8
6.5
OxonFair
Demain Independent
OxonFair
89.9
87.3
81.8
Domain Discriminative
Uniconf Adversarial
6.3
SA
25
Minimal Group Recall
Figure 4: Left: The Pareto frontier of min. group recall vs. accuracy Blond Hair demonstrates OxonFair's superior performance. Right: Comparing accuracy of fairness methods on 26 CelebA attributes while varying global decision thresholds to increase the minimum group recall level to 6.
32 is and using Adam { (learning rate le-4). We train for 20 epochs and select the model with highest validation accuracy. Images center-cropped and resized to 224 x 224. During training, we randomly crop and horizontally flip the images. See Appendix
Results:
Table D and demonstrates that using OxonFair as described in Section 4.2 generates fairer and more accurate solutions on unseen test data than other fair methods. Simple approaches such as Domain Independent training were more effective than adversarial training for enforcing fairness confirming 41 43) Occasionally, OxonFair finds solutions on the Pareto frontier that are both fairer and more accurate than the unconstrained classifier (See Figure 5).
Figure 4 shows a novel fairness evaluation motivated by medical use cases 42 where practitioners might want to correctly identify at least 8% of positive cases in each group. We evaluate how accuracy changes we guarantee that the minimum recall is above 8% for every group. For OxonFair, we call fit (accuracy, recall.min min, 5). For other methods, we vary a global offset to ensure that the minimum recall at least
5.2
NLP and Toxic Content
We conducted experiments on hate speech detection and toxicity classification using two datasets: the multilingual Twitter corpus (84) and Jigsaw (85). Experiments were performed across five languages (English, Polish. Spanish, Portuguese, and Italian) and five demographic factors (age, country, gender, race/cthnicity, and religion) were treated as the protected groups. For details, see Appendix F.I
We compare OxonFair with the following approaches. Base reports results of the standard BERT model [86]. CDA (Counterfactual Data Augmentation) [29 87 90) rebalances corpus by swapping bias attribute words (e.g., he/she) in dataset based on given dictionary. DP (Demographic Parity)
uses regularization 223 to enforce DP. EO (Equal Opportunity (3] uses the regularization of
23 43 to enforce EO. Dropout 88 90 is used as regularization technique [82] for bias mitigation
and improving small group generalization. Rebalance [TT, 9T method resamples the minor groups to the same sample size other groups to mitigate bias. We report scores Oxonfair optimized for

---

Test Set: Wearing Earrings
Test Set: Wearing Earrings
0.91
Min Group Min Label
Accuracy
Frontie
0.5
Origina predictor
0.89
Updated prediter
04
Domain Independent
Unicord Adversatial
0.3
0.88
0.87
Difference Equal Opportunity (DEO)
Accuracy
Figure 5: The Pareto frontier on test data when enforcing two faireess measures (DEO and Min Group Min Label Ace; Appendix C.I) for the Earrings attribute. Inspecting the Pareto frontier shows wide range of solutions, including some that improve fairness while retaining similar accuracy.
Table 3: Multilingual Twitter dataset:
Table 4: Jigsaw dataset: Religion (3
Gender (2 groups: Male, Female).
groups: Christian, Muslim, Other).
F1
Balanced
Acc.
DEO
F1
Balanced
Acc.
DEO
(1)
Acc. (1)
(1)
(1)
(1)
Acc. (1)
(1)
(L)
Base
40.8
63.2
89.8
21.4
Base
42.1
74.8
75.0
7.33
CDA 20
43.2
64.4
89.8
16.0
CDA
29
40.4
73.8
73.0
8.98
DP
23
37.2
61.7
89.5
17.9
DP
25
44.5
69.2
85.5
3.68
EO
32.0
59.6
EO
3
41.1
68.8
82,2
4.60
Dropout
88
32.2
59.8
88.9
13,8
Dropout
88
42.7
74.1
77.0
7.94
Rebalance 11
38.2
62.1
89.5
19.1
Rebalance 11
39.1
73.7
70.3
9.67
OxonFair Ac.
34.1
60.7
88.5
8.45
OxonFair Ac.
33.7
60.5
89.2
2.36
OxonFair F1
44.6
69.1
84.7
2.10
OxonFair F1
44.4
69.5
85.0
3.79
OxonFair B. Ac.
47.1
71.2
84.8
7.33
OxonFair B. Ac.
42.2
74.2
76.1
4.78
Oxon Fair Ac.
40.0
63.3
89.0
5.99
OxonFair Ac.
26.3
57.6
88.8
1.84
OxonFair F1
47.5
70.9
85.5
5.59
OxonFair F1
44.3
68.5
86.2
0.84
OxonFair B. Ac.
40.8
64.6
87.3
13.0
OxonFair B. Ac.
41.9
73.7
76.3
4.56
different metrics: Accuracy; F1 Score; and Balanced Accuracy, and always enforce that the DEO is under 5% on the validation set.
Results: Results are shown in Tables 3 and Our observations indicate that: 1) all debiasing methods improve the equal opportunity score and help mitigate bias Twitter, but not jigsaw. 2) our toolkit consistently reduces the difference equal opportunity more than any other approach; 3) for 4/6 experiments showing the value in targeting particular objective. For additional experiments on multilingual and multi-demographic data, and the Jigsaw race data, see Appendix F2 and Appendix F.3
actually improve the objective over the baseline while enforcing faimesss,
While improving fairness more than existing approaches, OxonFair performs substantially worse on these NL.P datasets than on CelbA. There are two challenges present in these datasets but not in CelbA: (i) likely much harder to infer gender or religion from short text data than it is to infer gender from photo of a face. (ii) The limited number of positively labelled datapoints examples that makes estimating DEO, Fl, and balanced accuracy unstable (see Tables 16 and [7 for details). To better understand the influence of the two factors we refit OxonFair using the true rather than inferred attributes at test time (bottom block) and see no reliable improvements, suggesting that we are most limited by data scarcity.
Herr OxonFair indicates the use of inferred group membership, while OxonFair" uses true group membership test time.

---

6 Conclusion
The key contributions of our toolkit lie in being more expressive than other approaches, and supporting NLP and computer vision. Despite this, most of the experiments focus on the standard definitions of Demographic Parity and Equal Opportunity. This is not because we agree that they are the right measures, but because we believe that the best way to show that OxonFair works is to compete with other methods what they do best. On low-dimensional tabular data, when optimizing accuracy and standard fairness measure, it is largely comparable with Fairlearn, but if overfitting or non-standard performance criteria and fairness metrics are a concern, then OxonFair has obvious advantages. For NLP, and computer vision, approach clearly improves part, this is due to the observation of [17], that methods for estimating or enforcing error-based fairness metries on high-capacity models that do not use held-out validation data can not work.
existing state-of ofte-offthh-art. In no small
We hope that OxonFair will free policy-makers and domain experts to directly specify fairness measures and objectives that are a better match for the harms that they face. In particular, we want to call out the measures in Figure BJas relevant to medical ML. The question of how much accuracy can we retain, while guaranteeing that classifier sensitivity (AKA recall) is above k% for every group, captures notions of fairness and clinical relevance a way that standard fairness notions do not (8).
We have chosen to optimize as broad a set of formulations as possible. As result, for certain metrics (particularly equalized odds (3]) the solutions found are known to be suboptima and for others [12] the exponential search is unneeded. Techniques targeting particular formulations may be needed to address this.
Limitations:
key challenge for most fairness approaches is in obtaining the group labels used to measure unfairnesss, and we no exception. In particular, the gender labels in CelebA and the race and religion labels in our NLP experiments consist of a small number of externally assigned labels that may not match how people self-identify. Improving and measuring fairness with respect these coarse labels can miss other forms of inequality. Moreover, major driver of unfairness is a lack of data regarding particular groups. However, this very absence of data makes hard for any toolkit to detect or rectify unfairness.
OxonFair is tool for altering the decisions made by ML systems that are frequently trained on biased data. Care must be taken that fair ML is used as a final step after correcting for bias and errors in data collation, and not as sticking plaster mask problems (9]. Indeed, inappropriate uses of fairness can lock in biases present in training (72). Under the hood, OxonFair performs a form of positive discrimination, where alter scores in response to (perceived) protected characteristics to rectify specific existing inequalitie As such, there are many scenarios where its use may be inappropriate for legal or ethical reasons.
Broader Impact:
7
Acknowledgements
This work has been supported through research funding provided by the Wellcome Trust (grant 223, Sloan Foundation (grant nr G-2021-16779), Department of Health and Social Care, EPSRC (grant nr EP/Y019393/1), and Luminate Group. Their funding supports the Trustworthiness Auditing for AI project and Governance of Emerging Technologies research programme at the Oxford Internet Institute, University of Oxford.
An early prototype version of the same toolkit, for tabular data, was developed while CR was working at AWS and is available online autogluon. fair (https://github com/ autogluon/ outogluon-faine CR is grateful to Nick Erickson and Weisu Yin for code reviews of the prototype. The authors thank Kaivalya Rawal for feedback the manuscript and testing the codebase.
References
[1] David Wen, Saad M Khan, Antonio Ji Xu, Hussein Ibrahim, Luke Smith, Jose Caballero, Luis Zepeda, Carlos de Blas Perez, Alastair K Denniston, Xiaoxuan Liu, et al. Characteristics of
$ $See Appendix D.2 ways to address this. See (2 for how other fairness methods may also be doing this.
10

---

publicly available skin cancer image datasets: systematic review. The Lancet Digital Health, 4(1):e64-e77) e74, 2022.
[2] Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan. Dissecting racial bias in an algorithm used manage the health populations. Science, 366(64644)444--533 2019.
[3] Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. Advances in neural information processing systems, 29, 2016.
[4] Emmanuel Martinez and Lauren Kirchner. The secret bias hidden in mortgage-approval algorithms. The Markup, 2021.
[5] Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. Machine bias. In Ethics of data and analytics, pages 254-264. Auerbach Publications, 2022.
[6] Solon Barocas, Moritz Hardt, and Arvind Narayanan. Fairness and machine learning: Limita- tions and opportunities. MIT Press, 2023.
[7] Richard Berk, Hoda Heidari, Shahin Jabbari, Michael Kearns, and Aaron Roth. Fairness in criminal justice risk assessments: The state of the art. Sociological Methods & Research, 50(1):3 2021.
[8] Brent Mittelstadt, Sandra Wachter, and Chris Russell. The unfairness of fair machine learning: Levelling down and strict egalitarianism by default. arXiv preprint 2xXiv:2302.02404, 2023.
[9] Rachel KE Bellamy. Kuntal Dey, Michael Hind, Samuel C Hoffman, Stephanie Houde, Kalapriya Kannan, Pranay Lohia, Jacquelyn Martino, Sameep Mehta, Aleksandra Mojsilovic, et al. Ai fairness 360: An extensible toolkit for detecting, understanding, and mitigating unwanted algorithmic bias. arXiv preprint arXiv: 181000194, 2018.
[10] Faisal Kamiran, Indrè Žliobaité, and Toon Calders. Quantifying explainable discrimination and removing illegal discrimination in automated decision making. Knowledge and information systems, 35:613 644, 2013.
[11] Michael Feldman, Sorelle A Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkata- subramanian. Certifying and removing disparate impact. In proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pages 259-20292 268. 2015.
[12] Sam Corbett-Davies Emma Pierson, Avi Feller, Sharad Goel, and Aziz Huq. Algorithmic decision making and the cost of fairness. In Proceedings of the 23rd aem sigkdd international conference knowledge discovery and data mining, pages 797-806, 2017.
[13] Zachary Lipton, Julian McAuley, and Alexandra Chouldechova. Does mitigating ml's impact disparity require treatment disparity? Advances in neural information processing systems, 31, 2018.
[14] Alexandra Chouldechova. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big data 5(2):153-163, 2017.
[15] Aditya Krishna Menon and Robert C Williamson. The cost of fairess in binary classification. Conference on Fairness, accountability and transparency, pages 107-118. PMLR, 2018.
[16] Luca Oneto, Michele Doninini, Amon Elders, and Massimiliano Pontil. Taking advantage of multitask learning for fair classification. Proceedings of the 2019 AAAI/ACM Conference AI, Ethics, and Society, pages 227-237, 2019.
[17] Dominik Zietlow, Michael Lohaus, Guha Balakrishnan, Matthäus Kleindessner, Francesco Locatello, Bernhard Schölkopf, and Chris Russell. Leveling down in computer vision: Pareto inefficiencies fair deep classifiers. Proceedings of the IEFF/CVE Conference on Computer Vision and Pattern Recognition, pages 10410-10421, 2022.
11

---

[18] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg. J. Vanderplas, A. Passos, Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
[19] Tianqi Chen and Carlos Guestrin. Xgboost: of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pages 785-794, 2016.
A scalable tree boosting system. In Proceedings
[20] Hilde Weerts, Miroslav Dudak, Richard Edgar, Adrin Jalali, Roman Lutz, and Michael Madaio. Fairlearn: Assessing and improving fairness of ai systems. Journal of Machine Learning Research, 24(257):1-8, 2023.
[21] Michael Lohaus, Matthäus Kleindessner, Krishnaram Kenthapadi, Francesco Locatello, and Chris Russell. Are two heads the same one? identifying disparate treatment in fair neural networks. Advances in Neural Information Processing Systems, 15:16548-16566 2022.
[22] Sorelle A Friedler, Carlos Scheidegger, Suresh Venkatasabbaamanian, Sonam Choudhary, Evan P Hamilton, and Derek Roth. A comparative study of cürmess--nnanncie interventions in machine learning. In Proceedings of the conference on fairness, accountability, and transparency, pages 329-338, 2019.
[23] Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rogriguez, and Krishna Gummadi. Fairness constraints: Mechanisms for fair classification. In Artificial intelligence and statistics, pages 962-970. PMLR, 2017.
[24] Agathe Balayn, Christoph Lofi, and Geert-Jan Houben. Managing bias and unfaimess in data for decision support: a survey of machine learning and data engineering approaches to identify and mitigate bias and unfairness within data management and analytics systems. The VLDB Journal, 3035)773 2021.
[25] Faisal Kamiran and Toon Calders. Data preprocessing techniques for classification without discrimination. Knowledge and information systems, 33(1):1-33, 2012.
[26] Flavio Calmon, Dennis Wei, Bhanukiran Vinzamuri, Karthikeyan Natesan Ramamurthy, and Kush Varshney. Optimized pre-processing for discrimination prevention. Advances in neural information processing systems, 30, 2017.
[27] Joymallya Chakraborty, Suvodeep Majumder, and Tim Menzies. Bias in machine learning software: Why? how? what to do? In Proceedings of the 29th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering, pages 429 440,4440045 2021.
[28] Vikram Ramaswamy, Sunnie SY Kim, and Olga Russakovsky. Fair attribute classification through latent space de-biasing. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 9301-9310, 2021.
[29] Ran Zmigrod, Sabrina J Mielke, Hanna Wallach, and Ryan Cotterell. Counterfactual data augmentation for mitigating gender stereotypes in languages with rich morphology. arXiv preprint
04571, 2019.
[30] Michael Lohaus, Michael Perrot, and Ulrike Von Luxburg. Too relaxed to be fair. In Interna- tional Conference on Machine Learning, pages 6360-6366 6369. PMLR, 2020.
[31] Alekh Agarwal, Alina Beygelzimer, Miroslav Dudík, John Langford, and Hanna Wallach. A reductions approach to fair classification. In International conference on machine learning, pages 60-69 PMLR, 2018
[32] Brian Hu Zhang. Blake Lemoine, and Margaret Mitchell. Mitigating unwanted biases with adversarial learning. Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pages 335-340, 2018
12

---

[33] David Madras, Elliot Creager, Toniann Pitassi, and Richard Zemel. Learning adversarially fair and transferable representations. In International Conference on Machine Learning, pages 3384-3393. PMLR, 2018.
[34] Han Zhao, Amanda Coston, Tameem Adel, and Geoffrey J Gordon. Conditional learning of fair representations. arXiv preprint vrXiv:1910.07162 2019.
[35] Byungju Kim, Hyunwoo Kim, Kyungsu Kim, Sungjin Kim, and Junmo Kim. Learning not to learn: Training deep neural networks with biased data. In Proceedings of the IEEE/CVF conference on computer vision and paltern recognition, pages 9012-9020, 2019.
[36] Enzo Tartaglione, Carlo Alberto Barbano, and Marco Grangetto. End: Entangling and disentangling deep representations for bias correction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13508 13517, 2021.
[37] Mhd Hasan Sarhan, Nassir Navab, Abouzar Eslami, and Shadi Albarqouni. Fairness by learning orthogonal disentangled representations. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIX 16, pages 746-761. Springer, 2020.
[38] Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv: 1911. 1911.0871,, 18773, 2019.
[39] Junbum Cha, Sanghyuk Chun, Kyungjae Lee, MlnnCheool Cho. Seunghyun Park, Yunsung Lee, and Sungrae Park. Swad: Domain generalization by seeking flat minima. Advances Neural Information Processing Systems, 34:22405-2244 2021.
[40] Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. arXiv preprint arXiv:2010.014: 2020.
[41] Zeyu Wang, Klint Qinami, loannis Christos Karakozis, Kyle Genova, Prem Nair, Kenji Hata, and Olga Russakovsky. Towards faimess in visual recognition: Effective strategies for bias mitigation. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 8919-8928, 2020.
[42] Yongshuo Zong, Yongxin Yang, and Timothy Hospedales. Medfair: Benchmarking fairness for medical imaging. In International Conference on Learning Representations (ICLR), 2023.
[43] Xiaotian Han, Jianfeng Chi, Yu Chen, Qifan Wang, Han Zhao, Na Zou, and Xia Hu. FFB: A fair fairness benchmark for in in-proces group fairness methods. In The Twelfth International Conference on Learning Representations, 2024.
[44] Geoff Pleiss, Manish Raghavan, Felix Wu, Jon Kleinberg, and Kilian Q Weinberger. On fairness and calibration. Advances in neural information processing systems, 30, 2017.
[45] Raman Dutt, Ondrej Bohdal, Sotirios Tsaftaris, and Timothy Hospedales. Fairtune: Opti- mizing parameter efficient fine tuning for fairness in medical image analysis. In The Twelfth International Conference on Learning Representations, 2024.
[46] Haonan Wang, Ziwei Wu, and Jingrui He. Fairif: Boosting faimnsss in deep learning via influence functions with validation set sensitive attributes. arXiv preprint 20X5::2001.05594 2024.
[47] M Zehlike, C Castillo, F Bonchi, S Hajian, and M Megahed. Fairness measures: Datasets and software for detecting algorithmic discrimination. URL http://ffimmsss-mmeee htw./faimeess-mmauueee... measures. org, 2017.
[48] Catherina Xu, Christina Greer, Manasi N Joshi, and Tulsee Doshi. Fairness indicators demo: Scalable infrastructure for fair ml systems, 2020.
[49] Kacper Sokol, Raul Rodriguez and Peter Flach. Fat forensics: A python toolbox for algorithmic fairness, accountability and transparency. Software Impacts, 14:100406, 2022.
[50] Julius A Adebayo et al. Fairml: Toolbox for diagnosing bias in predictive modeling, 2016.
13

---

[51] Florian Tramer, Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu, Jean- Jean-Pierre Hubaux, Mathias Humbert, Ani Juels, and Huang Lin. Fairtest: Discovering unwarranted associations in data-driven applications. In 2017 IEEF European Symposium on Security and Privacy (FuroS&P), pages 401-416 IEEE, 2017.
[52] Pedro Saleiro, Benedict Kuester, Loren Hinkson, Jesse London, Abby Stevens, Ari Anisfeld, Kit T Rodolfa, and Rayid Ghani. Aequitas: A bias and fairness audit toolkit. arXiv preprint 0rXiv:181..05577 2018.
[53] Niels Bantilan. Themis-ml: A fairness-aware machine learning interface for end-to-eut end discrimination discovery and mitigation. Journal of Technology in Human Services, 36(1):15- 30, 2018.
[54] Kate Crawford. The trouble with bias nips 2017 keynote. attps::/www.youtube 2017.
watch?v=fMym__BKU
[55]
Michelle Seng Ah Lee and Jat Singh. The landscape and gaps in open source fairness toolkits. In Proceedings of the 2021 CHI conference on human, humanffacorrs in computing systems, pages 1-13, 2021.
[56] Maarten Buyl. MaryBeth Defrance, and Tijl De Bie. fairret: framework for differentiable fairness regularization terms. In The Twelfth International Conference on Learning Represen- tations, 2024.
[57] André Cruz and Moritz Hardt. Unprocessing seven years of algorithmic fairness. In The Twelfth International Conference on Learning Representations, 2024.
[58] The pandas development team. pandas-dev/pandas: Pandas, February 2020.
[59] Natalia Martinez, Martin Bertran, and Guillermo Sapiro. Minimax pareto fairness: A multi objective perspective. In International Conference on Machine Learning, pages 6755-6764.
PMLR, 2020.
[60] Chloé Bakalar, Renata Barreto, Stevie Bergman, Miranda Bogen, Bobbie Chern, Sam Corbett- Davies, Melissa Hall, Isabel Kloumann, Michelle Lam, Joaquin Quiñonero Candela, et al. Fairness on the ground: Applying algorithmic fairness approaches to production systems. arXiv preprint arXiv:2103.0611 2021.
[61] Sahil Verma and Julia Rubin. Fairness definitions explained. In Proceedings of the international workshop on software fairness, pages 1-7, 2018
[62] Sanjiv Das, Michele Donini, Jason Gelman, Kevin Haas, Mila Hardt, Jared Katzman, Kr- ishnaram Kenthapadi, Pedro Larroy, Pinar Yilmaz, and Bilal Zafar. Fairness measures for machine learning in finance. The Journal of Financial Data Science, 2021.
[63] Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In Proceedings of the 3rd innovations in theoretical computer science conference, pages 214-226, 2012.
[64] Matt J Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. Counterfactual fairness. Advances in neural information processing systems, 30, 2017.
[65] Silvia Chiappa. Path-specific counterfactual fairness. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 7801-7808, 2019.
[66] Emily Diana, Wesley Gill, Michael Kearns, Krishnaram Kenthapadi, and Aaron Roth. Min- imax group fairness: Algorithms and experiments. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pages 66-76. 2021.
[67] Jacob D Abernethy, Pranjal Awasthi, Matthäus Kleindessner, Jamie Morgenstern, Chris Russell, and Jie Zhang. Active sampling for min-max fairness. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 53-65. PMLR, 17-23 Jul 2022.
14

---

[68] Sandra Wachter, Brent Mittelstadt, and Chris Russell. Why faimess cannot be automated: Bridging the gap between eu -dssccriminationn law and Computer Law & Security Review, 41:105567, 2021.
[69] Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. In EMNLP, 2017.
[70] Angelina Wang and Olga Russakovsky. Directional bias amplification. International Conference on Machine Learning, pages 10882- 10893, PMLR, 2021.
[71] Nick Erickson, Jonas Mueller, Alexander Shirkov, Hang Zhang, Pedro Larroy, Mu Li, and Alexander Smola. Autogluon-tabular Robust and accurate automl for structured data. arXiv preprint arXiv:2003.06505, 2020.
[72] Sandra Wachter, Brent Mittelstadt, and Chris Russell. Bias preservation in machine learning: the legality of fairness metrics under con-disriimination law. W. Va. L. Rev. 123:735, 2020.
[73] Sergey Golovenkin, Jonathan Bac, Alexander Chervov, Evgeny M Mirkes, Yuliya Orlova, Emmanuel Barillot, Alexander N Gorban, and Andrei Zinovyev, Trajectories, bifurcations, and pseudo-time large clinical datasets: applications to myocardial infarction and diabetes data. GigaScience, 9(11):giaa) bigial22,, 2020.
[74] Mohsan Alvi, Andrew Zisserman, and Christoffer Nellåker. Turning a blind eye: Explicit removal of biases and variation from deep neural network embeddings. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, pages 0-0, 2018.
[75] Amelie Royer and Christoph H Lampert. Classifier adaptation at prediction time. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1401-1409, 2015.
[76] Manisha Padala and Sujit Gujar. Frne: Achieving fairness through neural networks. In Proceed-
ings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, (1tbgeencee(/AA2200))) {ICCA-220) International Joint Conferences on Artificial Intelligence Organization, 2020.
[77] Michael Wick, -eann-Baptiste Tristan, et al. Unlocking fairness: trade-off revisited. Advances in neural information processing systems, 32, 2019.
[78] Ching-You Yao Chuang and Youssef Mrouch. Fair mixup: Fairness via interpolation. In Interna- tional Conference on Learning Representations, 2021.
[79] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pages 3730-3733 1730-3738,22 2015.
[80] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings the IEFE conference on computer vision and pattern recognition, pages 770-778, 2016
[81] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248-255. Icee, 2009.
[82] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-199) 2014.
[83] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 141.69980 2014.
[84] Xiaolei Huang, Linzi Xing, Franck Dernoncourt, and Michael Paul. Multilingual twitter corpus and baselines for evaluating demographic bias in hate speech recognition. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 1440-1448, 2020.
15

---

[85] Jigsaw unintended bias in toxicity classification, 2018.
[86] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 2018.
[87] Emily Dinan, Angela Fan, Adina Williams, Jack Urbanek, Douwe Kiela, and Jason Weston. Queens are powerful too: Mitigating gender bias in dialogue generation. arXiv preprint arXiv:1911.03844 2019.
[88] Kellie Webster, Xuezhi Wang. Ian Tenney, Alex Beutel, Emily Pitler, Ellie Pavlick, Jilin Chen, Ed Chi, and Slav Petrov. Measuring and reducing gendered correlations in pre-trained models. arXiv preprint arXiv:2010.06032, 2020.
[89] Soumya Barikeri, Anne Lauscher, Ivan Vulić, and Goran Glavaš. Redditbias: A real-world resource for bias evaluation and debiasing of conversational language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1941-1955, 2021.
[90] Nicholas Meade, Elinor Phole-Dayan and Siva Reddy. An empirical survey of the effectiveness debiasing techniques for pre-trained language models. arXiv preprint arXiv: arXiv:2110.0852) 2021.
[91] Yi Li and Nuno Vasconcelos. Repair: Removing representation bias by dataset resampling. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9572 9581, 2019.
[92] Agathe Balayn, Mireia Yurrita, Jie Yang, and Ujwal Gadiraju. "Z fairness toolkits, a checkbox culture?" on the factors that fragment developer practices in handling algorithmic harms. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society, pages 482-495. 2023.
[93] Michaela Hardt, Xiaoguang Chen, Xiaoyi Cheng, Michele Donini, Jason Gelman, Satish Gollaprolu, John He, Pedro Larroy, Xinyu Liu, Nick McCarthy, et al. Amazon sagemaker clarify: Machine learning bias detection and explainability in the cloud. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 2974 2983, 2021.
[94] Harvineet Singh, Matthäus Kleindessner, Volkan Cevher, Rumi Chunara, and Chris Russell. When do minimax-fair fair learning and empirical risk minimization coincide? In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 Pruceedings of Machine Learning Research, pages 31969-31989, PMLR, 23-29 Jul 2023.
[95] Sofie Goethals, Eoin Delaney, Brent Mittelstadt, and Chris Russell. Resource-constrained faimess. arXiv preprint 2rXiv:2406.0/290 2024.
[96] Matthew Groh, Caleb Harris, Luis Soenksen, Felix Lau, Rachel Han, Acrin Kim, Arash Koochek, and Omar Badri. Evaluating deep neural networks trained on clinical images in dermatology with the fitzpatrick 17k dataset. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1820-1828, 2021.
[97] Kweku Kwegyir-Aggrey, Jessica Dai, A Feder Cooper, John Dickerson, Keegan Hines, and Suresh Venkatasubraamanian. Repairing regressors for fair binary classification at any decision threshold. In NewrlPS 2023 Workshop Optimal Transport and Machine Learning, 2023.
[98] Michael Kearns, Seth Neel, Aaron Roth, and Zhiwei Steven Wu. Preventing fairness gerryman- dering: Auditing and learning for subgroup fairness. In International conference on machine learning, pages 2564-2572. PMLR, 2018
[99] David Freedman, Robert Pisani, and Roger Purves. Statistics. W. W. Norton & Company, New York, fourth edition, 2007. Hardcover, 700 pages.
16

---

[100] Peter J Bickel, Eugene A Hammel, and J William O' O'Connell. Sex bias in graduate admissions: Data from berkeley: Measuring bias is harder than is usually assumed, and the evidence is sometimes contrary to expectation. Science, 187(4175):398 404, 1975.
[101] Frances Ding, Moritz Hardt, John Miller, and Ludwig Schmidt. Retiring adult: New datasets for fair machine learning. Advances in Neural Information Processing Systems, 34, 2021.
[102] Pranjal Awasthi, Matthäus Kleindessner, and Jamie Morgenstern. Equalized odds postpro- cessing under imperfect group information. In Silvia Chiappa and Roberto Calandra, editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statis- tics, volume 108 of Proceedings Machine Learning Research, pages 1770-1780. PMLR, 26-28 Aug 2020.
[103] Vladimir N Vapnik. An overview of statistical learning theory. IEEE transactions neural networks, 10(5))988 9999999999 1999.
[104] Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. Learning fair represen- tations. In International conference machine learning, pages 325-333, PMLR, 2013.
[105] Drago Plečko and Nicolai Meinshausen. Fair data adaptation with quantile preservation. Journal of Machine Learning Research, 21(242):1-44, 2020.
[106] Toshihiro Kamishima, Shotaro Akaho, Hideki Asoh, and Jun Sakuma. Fairness-aware classifier with prejudice remover regularizer. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECMI. PKDD 2012, Bristol, UK, September 24-28, 2012. Proceedings, Part II 23, pages 35-50. Springer, 2012.
[107] Elizabeth Anne Watkins, Michael McKenna, and Jiahao Chen. The four-fifths rule is not disparate impact: woeful tale of epistemic trespassing in algorithmic fairness. arXiv preprint arXiv:2202 99955, 2022.
17

---

Frondier Found
Frontier Found
0.00
0.00
0.75
0.75
0.70
0.73
0.65
Unditd prediss
Roweighing
0.60
cridsarn sedact
0.55
Leg,eg
Leavects rmprrsortatinnn
0.00
0.05
0.10
5.25
0.20
0:23
0.20
3.25
0.20
Demsgraphic Partty
Demographi: Parity
Frontier Found
Frantier Found
0.700
0.53
0.55
0.50
0.00
0.05
9.15
0.53 9.55
6:15 0.20
Demographic Partt
Demogmphi:: Party
Figure 6: We show a full comparison of the methods provided by AIF360 and Fairlearn on the adult dataset with 4 different choices of metric (accuracy, balanced accuracy, F1, and Matthews correlation coefficient (MCC)), while enforcing demographic parity. We follow the design decisions of 9 and use random forest with 100 trees and minimum leaf size of 20. Only OxonFair allows the specification of an objective, and for all other methods we try to alter the decision threshold to better optimize the objective. However, we improve the objective, see fairness deteriorates. To avoid this, OxonFair jointly optimize both a faireess measure and an objective.
A Inferred characteristics
In many situations, protected attributes are not available at test time. In this case, simply use inferred characteristics to assign per-group thresholds and adjust these thresholds to guarantee fairness with respect to the true (i.e. uninferred) groups.
When using inferred characteristics, we offer two pathways for handling estimated group membership. The first pathway we consider makes hard assignment of individuals to groups, based on classifier response. The second pathway explicitly uses the classifier confidence as part of per-datapoint threshold. In practice, we find little difference between the two approaches, but the hard assignment to groups is substantially more efficient and therefore allows for finer grid search and generally better performance. However, the soft assignment remains useful for the integration of our method with neural networks, where we explicitly merge two heads of a neural network to arrive at a single fair model.
A.1
Fast pathway
The fast pathway closely follows the efficient grid search for known characteristics. We partition the dataset by inferred characteristics. and then repeat the trick. However, as the inferred characteristics do not need to perfectly align with the true characteristics, we also keep track of the true group datapoints belongs to, i.e., for all datapoints assigned to particular inferred group, we compute the cumulative sum of positives and negatives that truly belong to each group. This allows us vary the thresholds with respect to inferred groups while computing group measures with respect to the
18

---

true groups. This can be understood as replacing the decision function ① with f(x) - t G' 2 0 where G' is a binary vector valued function that sums 1, but need not correspond G exactly.
This explicit decoupling of inferred groups from the true group membership allows us to consider partitionings of the data that do not align with group membership. We found it particularly helpful to include an additional 'don't know' group. By default, any datapoint assigned scord 10 from the classifier below 2/3 is assigned this group, and receives different threshold those datapoints that the classifier is confident about. The improved frontiers are shown in the tabular experimental section as OxonFair+, where they offer clear advantage over our baseline OxonFair.
A.2 Slow pathway
The slow pathway tunes t to optimize the decision process f(x) - g(x) 2 O, where g is a real vector valued function. Given the lack of assumptions, no obvious speed-up was possible and perform a two stage naïve grid-search, first coarsely to extract an approximate Pareto frontier, and then finer search over the range of thresholds found in the first stage. This is then followed by final interpolation that checks for candidates around pairs of adjacent candidates currently in the frontier.
In situations where (ax) is the output of classifier and G' its binarization, it is reasonable to suspect that loss information from binarization might lead to a drop in performance when we compare the slow pathway with the fast. In practice, never found significant change, and in like-with-like comparison over a similar number of thresholds the fast pathway was as likely to be fractionally better as it was to be worse. Moreover, for more than 3 groups the slow pathway becomes pumitively slow, and to keep the runtime acceptable requires decreasing the grid size in a way that harms performance.
Despite this, we kept the slow pathway as it is directly applicable to deep networks as we describe in the next section. In practice, when working with deep networks make use of hybrid approach, and perform the fast and slow grid searches before fusing them into a single frontier and then performing interpolation. This allows us to benefit from the better solutions found by fine grid search when the output of the second head near binary (see Figure 2. and robustly carry over to the slower pathway where its binarization is bad approximation of the network output.
B Implementation of Performance and Fairness Measures
To make OxonFair readily extensible, we create a custom class to implement all performance and fairness measures. As such, should OxonFair not support particular measure, both the objectives and constraints can be readily extended by the end user.
Measures used by OxonFair defined as instances of a python class GroupMetrics. Each group measure is specified by a function that takes the number of True Positives, False Positives, False Negatives, and True Negatives and returns score; A string specifying the name of the measure; and optionally a bool indicating if greater values are better than smaller ones.
For example, accuracy is defined as:
accuracy gm gm. amm.GruupMetrcccmmmmm (1amdda TP, FP, FN, TN: (TP TN) / (TP + FP FN + TN), 'Accuracy')
For efficiency, approach relies broadcast semantics and all operations in the function must be applicable to numpy arrays. Having defined a GroupMetric it can be called in two ways. Either:
accuracy (tarraccyttrggeellaeeee.. predictions, groups)
Here sarget_labell and predictions are binary vectors corresponding to either the target ground- truth values, or the predictions made by classifier, with I representing the positive label and O otherwise. groups simply vector of values where each unique value is assumed to correspond to a distinct group.
The other way it can be called is by passing it a single 3D array of dimension 4 by number of groups by k, where k is the number candidate classifiers that the measure should be computed over.
User controllable threshold.
19

---

Table 5: The fairness measures in the review of [61] All 9 group metrics that concern the decisions made by a classifier are supported by OxonFair.
Vema and Rubin [61] Metrics
OxonFair name
Fairlearn
Group fairness statistical parity
demographic.parity
Yes
conditional- metrrcc
Conditional statistical parity
No
pos_pred_rate.diff pos
Predictive parity
predictive_parity
No
False positive error rate balance False negative error rate balance Equalized odds
false_pos__rae. false_neg_rate diff equalized_odds cond_use_accuraçe
Yes
Yes
Yes
Conditional use accuracy equality
No
Overall accuracy equality
accuracy.diff
No
Treatment equality
treatment. diff
No
Test-fairness or calibration
Not decision based
Well calibration
Not decision based
Balance for positive class
Not decision based
Balance for negative class Causal discrimination Fairness through unawareness Fairness through awareness No unresolved discrimination No proxy discrimination Fair inference
Not decision based
Individual fairness
Individual fairness
Individual fairness Individual fairness Individual fairness Individual faimesss
As convenience, GroupMetries automatically implements a range of functionality as sub-objects.
Having defined metric as above, we have a range of different objects:
metric. diff reports the average absolute difference of the method between groups.
metric. average metric.average reports the average of the method taken over all groups
metric. max_ maf_ reports the maximum difference of the method between any pair of groups.
metric. reports the maximum value for any group
metric. min reports the minimum value for any group.
metric. overall reports the overall value for groups combined, and is the same as calling metric directly
metric. ratio reports the average over distinct pairs of groups of the smallest value divided by the largest
metric.per
reports the value for every group.
All of these be passed directly fit, or the evaluation functions we provide.
The vast majority faireess metrics are implemented as a diff of a standard performance measure, and by placing min after any measure such as recall precision it is possible to add constraints that enforce that the precision or recall is above a particular value for every group.
Computing certain metrics, particular Conditional Demographic Disparity 668 and Bias Amplification [70], requires knowledge of the total number of TP etc. alongside the number of TP in each group.
Total Metrics
To implement these measures, we support lambda functions that take per-group values, followed by the global values, giving 8 arguments in total. These lambda functions should be passed to GroupMetric in the same way along ith the optional argument total_metric-Tin
20

---

Table 6: The post-training fairness measures in the review of 93 All measures supported by
OxonFair.
Post-training Metrics 93
OxonFair name
Fairlearn
Diff. in pos. proportions in predicted labels
demographic_pariss disparate_ _inttet__mactttt cond_accept... diff cond_reject.dii diff accuracy.de .iff diff decall.diff
Yes
Disparate Impact
No
Difference in Conditional Acceptance
No
Difference in Conditional Rejection
No
Accuracy Difference
No
Recall Difference
Yes
Difference In Acceptance Rates
acceptance_rate diff rejection_rate.dir treatment_equalite conditional.group.eee mettrics pos_ pos__pre__rate.de
No
Difference in Rejection Rates
No
Treatment Equality
No
Conditional Demographic Disparity
No
Table 7: Enforcing faimesss for all definitions in on COMPAS with inferred attributes. We enforce the fairness definitions with respect to three racial groups, African American, Caucasian, and Other consisting all other labelled ethnicities. There are total 350 individuals labelled 'Other' in the test set, making most metrics of fairness unstable and difficult to enforce. Nonetheless, we improve on all metrics. For all metries except disparate impact, we enforce that the score on train below 2.5% and for disparate impact we enforce that the score train is above 97.5%. XGBoost is used as the base classifier, and the dataset is split into 70% train and 30% test.
Measure (original)
Measure (updated)
Accuracy (original)
Accuracy (updated)
Demographic Parity
0.148706
0.097142
0.661 344
0.620588
Disparate Impact
0.668305
0.740940
0.661345
0.605042
Difference in Conditional Acceptance Rate
0.231862 862
0.151159
0.661345
0.642857
Difference Conditional Rejectance Rate
0.048625
0,025138
0.661345
0.655882
Difference in Accuracy
0.013172
0.006351
0.661345
0.665546
Difference in Recall
0.151210
0.105154
0.661345
0.612185
Difference in Acceptance Rate
0.070072
0.066591
0.661345
0.662605
Difference in Specificity
0.097490
0.064139
0.661345
0.660504
Difference Rejection Rate
0.050085
0.050215
0.661345
0.661345
Treatment Equality
0.201717
0. 0.105115
0.661345
0.660924
Conditional Demographic Parity
0.150927
0.073203
0.661345
0.626471
C Additional Metrics
To demonstrate OxonFair's versatility, (and the metrics of two review papers and how many can are implemented out of the box by our approach. An example showing how all clarify metrics can be enforced using inferred groups, and three group labels on compas can be seen in Table 7
C.1 Minimax Fairness
Minimax fairness [59 66 67 refers to the family of methods which minimize the loss of the group where the algorithm performs worst, i.c., they minimize the maximal loss. (44] observed that sufficiently expressive classifiers, such those considered by this paper, including boosting, random forests, or deep networks on image and NLP tended be per group optimal, when the groups do not correspond to the predicted label. As such they are already minimax optimal and the solutions found by minimax fairness methods are indistinguishable from those found by empiric risk mininization. This still leaves the case where groups include the label (for example, the groups may correspond to the product of gender and the variable we are trying to predict, such as sick or not sick). In this case, convincingly shown by (59), the solutions found do not correspond to ERM.
21

---

Here, we compare OxonFair against minimax fairness. To do this, we define a new performance measure corresponding the lowest accuracy over the positive or negative labelled datapoints.
TP
TN
min accuracy = min TP+FP
(5)
FN+TN
Martinez et al [59] argued that we should seek a Pareto optimal solution that has the highest possible overall accuracy, subject to the requirement it maximizes the lowest per group accuracy. We can do this in OxonFair by calling fpredictor fit (gm..n aain_acuracyy salaaacmiiii min, gm. ann.accurccy 0) Here min_accuracy.. min corresponds to the lowest min accuracy of any group. We use accuracy > 0 as the constraint, we do not want an active constraint from preventing us from finding the element of the Pareto frontier (see [59] for frontier details) with the highest minimum accuracy. Note that the groups used by OxonFair with this loss correspond to the true groups, such as ethnicity or gender, while the groups used by minimax fairmess are the product of these groups with the target labels. Existing methods for minimax fairness optimize the same loss and have indistinguishable accuracy. only differing in the speed of convergence 67 As such, in Table g we only report results for variant of 67
Similarly, in computer vision 17 optimized the same objective by iteratively generating synthetic data for the worst performing group. where groups were defined as the product of ground-trute labels,
and sex. We compare against them in Table 15
XGBoost: Adult (sex)
Min Accuracy
Overall Accuracy
ERM Training
70.3%
90,9%
Minimax Training 67
85.2%
88.9%
ERM Validation
58.8%
86,9%
Minimax Validation 667
76.2%
83.9%
OxonFair Validation
79.1%
84.4%
ERM Test
59.6% 77.9%
86.6%
Minimax Test [67].
84.1%
OxonFair Test
80.5%
84.6%
Table 8: Results for XGBoost: Adult (sex)
C.2 Utility Optimization
OxonFair supports the utility-based approach of Bakalar et al. [60], whereby different thresholds can be selected per group optimize utility-based objective. Utility functions can be defined in one line. In the following example, we consider a scenario where an ML system identifies issues that may require interventions. In this example, every intervention has cost of 1, regardless of if it was needed, but a missed intervention that was needed has a cost 5. Finally, not making an intervention when was not needed has a cost of O.
Figure shows code where fpredicton ctor minimizes the utility subject to the requirement that the overall recall cannot drop below 0.5.
Figure 7: Defining and optimizing custom utility function with OxonFair (60]
C.3 Levelling up
One criticism of many methods of algorithmic fairness is that enforcing equality of recall rates (as in equal opportunity) or selection rates (as in demographic parity) will decrease the recall/selection rate for some groups while increasing it for others. This behavior an artifact of trying to maximize accuracy [ and occurs despite faimess methods altering the overall selection rate [95]. As an alternative, OxonFair supports levelling up where harms are reduced to, at most, a given level per group 3 For example, if we believe that black patients are being disproportionately harmed by high number of false negatives cancer detection (i.e., low recall), instead of enforcing that these properties be equalized across groups, we can instead require that every group of patients has, at
22

---

least, minimum recall score. Depending on the use case, similar constraints can be imposed in with respect per-group minimal selection rates, or minimal precision. These constraints can be enforeed by single call, for example, enforcing that the precision is above 70% while otherwise maximizing accuracy can be enforced by calling: .fit (gm. (gm. accuracy,, gm. wm..rrccisnn. min, 0.7). See also Figure
0.6
Foaner
Drg nut prolicaa
docated crnditor
0.8
0.4
6.3
0.5
0.1
6.5
5.6
0.7 0.6
03
3.0
Minimal Group Recall
Figure 8: Levelling-up with OxonFair by imposing minimum group recall of 0.7 on the Fitzpatrick- 17k [96] validation set fpredictor fit (gm. accuracy, gm. www.lecllll... recall.min, 0.7).
To enforce levelling up
hard additional constraint, fit takes optional argument, force_levelling up. Setting this equal to >+ forces the selection rate to increase in the search procedure, while setting it equal to > means it can only decrease. As most performance metrics underlying fairness constraints (e.g. recall, selection rate, precision) are monotonic with respect to the selection rate (95), this can prevent levelling down.
The levelling-up constraint can be combined with standard forms of fairness egg. by call- ing .fit (gm. accuracy, gm. patnogrrphic_parittye. 0.01, force_levelling.uu they are most useful when using levelling up constraints in conjuncture with an inadequately opti- mized classifier. In some cireumstances calling fitt (gm. accuracy, gm recall min, 0.6) can result a decrease in recall rate for some groups providing this increases accuracy and does not drop below the recall below the specified minimal rate.
but
C.4 Fairness under constrained capacity
When deploying fairness in practice, we may be capacity limited. For example, as Figure g we may use the output of classifier for detecting cancer to schedule follow-up appointments. In such case, you might wish that the recall high for each demographic group. but be constrained by the number of available appointments. Calling fitt (git.gmrecalll www.ll..aii....................l min, gm. pos prspppreeeeeeeeppppppppppepeeeee pred_ rate, 0. greater_ is better_ betteer_constaaalll const=False) will maximize the recall on the worst-off group subject to
requirement that no more than 40% of cases are scheduled follow-up appointments.
In general, maximizing the group minimum of any measure that is monotone with respect to the selection rate, while enforcing a hard limit on the selection rate will enforce equality with respect to that measure (e.g. optimizing gm. recall min will result in equal recall a.k.a. equal opportunity, while maximizing gm. pos_pred____.pppppppppppppp rate. min will result in demographic parity), while also enforcing the selection rate constraints, See 995 for proof and discussion of the issues arising, and 97 for an alternate approach.
As such, calling (git min, gm.pos_ppa lase__red_raste k, greater_is _e___i_bbb_ee better False) will enforce equal opportunity at 1% selection rate, and fit (gm. gegp.pos_predie lerss_red_rrrr rate. rwt.rrateeiinn min, gm.pos pacppos_ppreee boos__pre__reee pate, 0.4, greater is wettteeett_sssseeeeeeee False) will enforce demographic parity at 5% selection rate.
C.5 Conditional Metrics
A key challenge of using fairness in practice is that often some sources of bias are known, and the practitioner is expected to determine if additional biases exist and to correct for them. For example, someone's salary affects which loans they eligible for, but salary has distinctly
23

---

Adal
Adul
fanDoa Gree Nat Woodeer Prot 9300
62-
Undal Coreter
Mie Prodict
Pestive Procktion Rusc
Figure 9: Solutions found when enforcing demographic parity with varying rate constraints. See Appendix C.4 Left: the change in precision as enforce demographic parity. Note that report precision as it is more informative than accuracy for low selection rates. Right: The ratio between selection rates (i.e. disparate impact) for different groups. We report the ratio rather than the difference, as the difference tends to zero as the selection rate also tends to zero. However, as the right figure shows, this ratio becomes unstable as the rate tends zero
different distribution for different ethnicities and genders. 665]. Identifying and correcting fairness here rapidly becomes challenging, when considering the intersection of attributes, many small groups arise and purely by chance some form of unfairness may be observed [98, 68 [8ggegee the use of technique from descriptive statistics that [99] had previously applied to the problem of schools admissions at Berkley [100]. In this famous example, every school in Berkley showed little gender bias, but due different genders applying at different rates to different schools, and the schools themselves having substantially different acceptance rate, a strong overall gender bias was apparent.
199] observed that you could correct for this bias by computing the per school selection-rate, and then taking a weighted average, where the weights are given by the total number of people applying to the school. The resulting selection rates are equivalent
weighted selection-rate over the whole population, where the weight wi for an individual in a particular group and applying to a #individuals in school
particular school is wi = #indiidduls in group and school To enforce this form of conditional demographic parity in OxonFair, we simply replace the sum of true positives etc. in Section 3] with the weighted sum. We support range of related fairness metrics, including conditional difference in accuracy; and conditional equal opportunity (note that for equal opportunity we replace the numbers used to compute w; with the same counts but only taking into account those that have positive ground-truth labels). As such metrics can level down (Appendix C.3) we also support conditional minimum selection rates, and conditional minimum recall.
In addition to this, we support the conditional fairness metrics based on EU/UK law of [68]. These measure the (weighted) proportion of people belonging to particular group in the set of advantaged or disadvantaged people. These metrics can be computed for both the input target variables a classifier trained predict and for the classifier outputs.
C.6 Bias Amplification Metrics
We also support variants of Bias Amplification, defined by Zhao et al. and Wang et al.
669 70
As with the other metrics, we focus on scenarios where ground-truth group assignments exist even if they are unavailable at test time, and as such we focus on the Attribute + Task Bias (see (70)).
Bias Amplification is implemented as per-group metric, gm. aiias_mplificattio measure is signed (a negative measure indicates the classifier reverses the bias present in the dataset), directly optimising it results in classifiers strongly biased in new direction. Instead, minimize the per group absolute bias amplification. The derivation is given below.
As this
Following the notation of Wang et al. 770, let A be the set of protected demographic groups: for example, = (male, female). An for a E is the binary random variable corresponding to the presence of the group thus P(Avouan = 1) can be empirically estimated as the fraction of images
24

---

in the dataset containing women. Let 7, with t 7 similarly correspond to binary target tasks. Let Ag and T, denote model predictions for the protected group a and the target task i, respectively.
1
BiasAmp-,
Σ
Yat yat + (1 Yue)( (- Dut)
REA.EE
Mot 1 [[((A, = 1,T: = 1) P(A. 1)P( = 1)]
P(T: 1|A, = P(T: = 1| =
measuring Allribule + Task Bias
A
(6)
P(Á. = -1 = - P(A. = 1 1)
if measuring Task + Attribute Bias
Of which, the Attribute - Task Bias is relevant here.
Each component can be written as function of the global True Positives, False Positives etc., and the per group True Positives, and as such it can be optimized by our framework, albeit, not by using standard group metrics. However, this metric is gamable, and consistently underestimating labels in groups where they overrepresented and vice versa would be optimal, but undesirable behavior that leads to negative score.
Instead, we consider the absolute BiasAmp:
[BiasAmpl
Σ
yar + (1 (ar)(- Aat)
T
BEANET
Σ
Datl
aEA,EE
We decompose | into the appropriate form for GroupMetric (see Appendix B follows:
A.t P(f 1|Ag 1) 1/Ag =)
(8)
TP + FN
Aut TP TN FP FN FN FFP
(9)
TP+TN FP+FN
A
(10)
TP+TN + FP+ FN
This will give per group estimate of the absolute bias amplification, and calling its average method will give the absolute bias amplification over all groups
D
Comparisons with specialist methods
D.1
Fairret
Fairret [56] is recently published toolkit for enforeing faimess in pytorch networks, using standard [23 This toolkit has only been shown to work for tabular data. There are mathematical reasons to think that it will not work for bias preserving fairness metrics [72] such as equalized opportunity, on computer vision or NL.P classifiers where classifiers obtain zero error on the training set [17], and therefore trivially satisfy all bias preserving fairness metries. However, as Fairret uses standard relaxations, it should have comparable accuracy/fainner trade-off's to OxonFair when weakly enforcing demographic parity on image data (see
fairness constraints
Fairret should also have comparable performance on NLP data to the DP and EO regularized approaches reported in the main body of the paper.
Here we focus on using Fairret to enforce Equal Opportunity on tabular data as shown their paper. Figure T0 shows comparison with Oxonfair. For a like-with-like comparison between Fairret and OxonFair we use 70% of the data for training for Fairret (which requires no validation data), and for OxonFair split this into 40% training data and 30% validation data.
25

---

Comparing Chanfaii
Comput bng Faire
0.54
3.01
0.00
XGleeet Frent prosecer upcaned prodictor
3.00
Mikooee hrme
Undated sredictr
Reural teor
Naural Net foott
0.76
Pormet
0.78
Rairnt
6.16
a:ko 6.15
Difirence Ezual Opparuntty (D20)
Derrggappic Party
(b)
Figure 10: Comparing OxonFair and Fairret [56] on adult using sex as the protected attribute. A simple neural network classifier with two hidden layers is used the base classifier. Fairness strengths are varied for different Fairret implementations. Difference in Equal Opportunity and Demographic Parity are considered. An OxonFair-baut frontier using XGBoost is also displayed.
In general, neural networks perform worse than boosting on tabular data, and should not be used where maximizing accuracy is concern. Nonetheless, we that Fairret shows worse accuracy/fairness trade-offs than OxonFair on neural networks with the same architecture.
Compared to OxonFair, there are three challenges faced by Fairret that might be contributing to its worse performance.
The enforced constraints are relaxation of the underlying integer fairness constraints, and even when completely satisfied, they need not imply fairness [30].
A mismatch between errors on the training and test set. Even when models do not obtain zero training error they can still overfit, and minimizing equal opportunity on the training set does not imply that it is optimized on the test set.
Failure to converge. To induce stability in the performance of Fairret we had to use much larger minibatch size of 1000, as the minibatch must be large enough to estimate the EO violation somewhat stably if want the method to converge. It is possible that either the batch size was still not large enough or that was so large that it caused issues with optimizing the log loss.
However, other issues might be the reason for the performance discrepancy.
D.2 Specialist Equalized Odds Solvers
In this section, we show how OxonFair can be adapted through the use of custom groups to mimic the
performance of Specialist Equalized Odds solvers. We compare against the recently published (57].
which was shown to outperform wide range of existing methods. Like us, [57] assigns thresholds on per-group basis to enforce fairness, and enforces fairness up to a prespecified threshold (e.g.. equalized odds is less than 5%) while maximizing accuracy. Unlike the default behaviour of Oxonfair, however, it randomly assigns members of each group one of two different thresholds. To mimic this behaviour, we make use of the same trick used in Section A. namely that the 'inferred groups' used to assign thresholds are not expected to align with the true groups, and that we can introduce additional groups to increase the expressiveness of the model.
Before showing how to enforce Equalized Odds, we strongly recommend that this not used in practice. [57] also notes that enforcing it is contentious. Thresholding methods such as OxonFair and [57] can be understood as methods that trade-off sensitivity or recall against specificity. When enforcing Equalized odds, we will move to new point on the sensitivity specificity curve, for the worst performing group, while degrading the performance for other groups. Viewed through the lens of levelling up [8] and which specific harms are incurred by each group, versus base classifier: enforcing equalized odds will increase one the sensitivity and specificity for the worst performing group and decrease the other; and may decrease sensitivity and specificity for all other
26

---

Per group Behaviour of Equalized Odes
compantsion of randomized and deterministic Eg, Odds Strategies
0.575
Deteministic threshald
Undsted predictor
Updatel p-0.5
0.570
APP 0
0.7-
0.865
Gooo Goo
The Croun Recal
Accume
0.860
0..0
group thresholiing
group thresholding
0.5
Eoads
Eoads
predictor
pradictor
predictor
Updated predlictor
0.50
0.85
5.00
2.93
1:50
2.000 0.005
2-030
0.015
0.020
0.025 0.000
Per Group Speciiicyy
Equalized Odts [L
Figure 11: Left: A close-up of possible per-group trade-offs when enforcing Equalized odds. This figure shows possible behaviour when enforcing Equalized odds with respect to sex on the adult dataset. Compared to the original predictor, we see substantial decrease in recall for the worst performing group accompanied by small increase in specificity. For the better performing group, both recall and specificity are decreased order to enforce fairness. Right: The accuracy/fairnes: trade-offs of OxonFair using single threshold, deterministic multi-thresholds, and randomized multi- thresholds. Single threshold performs substantially worse for strong fairness constraints, but the other two strategies are interchangeable. All results shown are validation data.
groups. (see Figure Figure I1 for an illustration the frontier found by OxonFair with respect to recall and specificity). While other methods are harder to analyze than thresholding, the fact that they have worse accuracy/fairnes trade-offs suggests that they deteriorate classifiers more than simple thresholding.
We consider four threshold sets for OxonFair.
1. Default OxonFair using one threshold per group.
2. A randomized version that mimics the behaviour of [57] by assigning members of each group two subgroups with 50/50 probability.
3. As randomized approaches can be criticized for their instability, we also consider a determin- istic version that flips the scores of positively labelled data points scored below threshold, and also flips the score of negatively labelled points scored above threshold.
4. An inferred variant that does the same 3, but doesn't require access groups time.
Additional code for the last three options is shown below. These functions be passed to FairPredictor using the inferred.groups option.
Our comparison with [57 can be summarized as the randomized and deterministic methods are broadly interchangeable both with [57] (see Figure [2) and with each other (see Figure I right), and they strongly outperform the default single-threshold OxonFair for strong fairness constraints.
As both [57] and multi-threshold OxonFair solve the same optimization problem via different routes (a specialist LP solver for [57], grid-search for OxonFair) we expect them to obtain similar solutions, and this we find in Figure 12 However, [57] makes use of formulation that more efficient for large numbers of thresholds, but less expressive and harder to adapt to infered group membership. or new fairness or performance constraints, As such, it is unsurprising that is substantially more efficient than grid-search over the 8 thresholds used in these experiments at least when finding solution with e.g. an Eodds violation
no more than 0.05. What was more surprising was that when using the code of [57] to compute an entire fairness/accuracy Pareto frontier, Oxonfair remained more efficient. See Table for details. This remains reminder that big-O notation uninformative with respect to runtime, when datasets extremely limited in their number of groups.
27

---

Method
57
(single fairness eval)
577
(Complete Frontier) OxonFair Complete Frontier
Runtime
3 sec
4m49.1 sec
1m18s
Table 9: Runtime comparison between 157 and OxonFair, on the Folktables dataset [[0]] using four racial groups. This represents the largest problem with the most groups reported in [57], and as such is the experiment where we would expect [57] outperform OxonFair the most with respect to runtime. While [57] is faster if enforcing fairess to known amount, e.g., maximizing accuracy subject EOdds<0.05%% OxonFair remains faster for computing the entire frontier.
Data
0.10
0.10
prol-yriceesingg of (57)
peei-proeweeeg el [57]
95% cort Intery
95% cont Imery.
0.08
Oxor Threshold
0.00
viladoon
0.00
0.06
0.04
0.04
soualaed
equallzed
0.22
0.02
0.00
0.50
0.60
0.70
0.75
0.80
0.500
0.625
0.600
0.676
0.700
0,725
0.700
0.775 0.800
aocuracy
accuracy
Figure 12: A comparison between single- single-threshold OxonFair and deterministic multi-threshold OxonFair with [57]. As expected, Multi-threshold directly comparable to (57]. with single threshold performing worse.
Original OxonFair Multi
OxonFair + FairLearn
Accuracy Equalized Odds
0.871016
0.866921
0.859960
0.866512
0.097950
0.026676
0.025532
0.045989
Table 10: A comparison Fairlearn, and OxonFair when enforeing Equalized Odds. We enforce fairness at 2% on validation to roughly match accuracy with Fairlearn. At this value, there is limited change between versions of Oxonfair and the multi-threshold approach obtains half the fairness violation of Fairlearn at similar accuracy.
Eodis with inferred groups (Velidation)
Eodds with inferred groups (Test)
0.36
0.84
0.82
Daptfure
DvarFarre
Orighal predictor
0.78
Drigins predictor
Upstated predictor
3.78
Updated coedicter
Daonter Null Thrmebole
Coonriii Mubliireshold
Updattdd predictor
0.76
Updated predictor
3.76
Falifea
0.00
0.01
0.02
5.03
0.24
0.05
8.05
5,77
0.00
0.22
0.04
0.05
5.09
0.10
Equa Oride [1_
Equalized Didss [I_im
Figure 13: Enforcing Equalized odds using inferred characteristics. While the tulti-threshold approach shows clear advantages on the validation set, this does not generalize to unseen test data. For unseen data, single threshold approaches show stronger degradation in accuracy, but their faimess constraints generalize better. This can be attributed to single threshold approaches selecting near constant classifiers when the constraints are strong, while the classifiers found by multi-threshold approaches are more vulnerable sampling differences between validation and test.
28

---

D.2.1 Equalized odds without using group membership at test-time
As [57 requires access groups at test time, cannot directly compare with [57 we instead compare against FairLearn on the adult dataset, using sex as the protected characteristic. Results can be seen in Table T0 and Figure 13
Our approach differs from [102], as they enforce fairness with respect to inferred groups (without access to true group labels), while we directly use the inferred group labels to enforce faimesss with respect to the underlying true groups.
E Computer Vision Experiments
Table 11: Hyperparameter details for the CelebA experiment.
Hyperparameter
Value/Range
Learning Rate
0.0001
Batch Size
32
Dropout Rate
0.5
Backbone
Resnet-50
Weight Decay
0
Optimizer
Adam
83
Epochs
20
Table 12: CelebA Attribute-lewe information from Ranaswamy et al. (88] The columns are target attribute name, percentage positive samples, skew. For example, Earrings has a skew of 0.97 towards g= 1, that is, 97% of positive Earrings samples have gender expression label g= (Female)
Attribute type
Attribute statisties
Inconsistently labeled
Positive
Skew
BigLips
24.1%
0.73
0=
BigNoose
23.6%
0.75
|=1
OvalFace
28.3%
0.68
g=
PaleSkin
4.3%
g
StraightHair
20.9%
0.52
g=
WavyHair
31.9%
0.81
g=
Gender-dependent
Positive
Skew
ArchedBrows
26.6%
0.92
Attractive
51.4%
0.77
BushyBrows
14.4%
0.71
9=1
PointyNose
27.6%
0.75
0=
RecedingHair
8.0%
g=1
Young
77.9%
0.66
Gender-independent
Positive
Skew
Bangs
15.2%
9
BlackHair
23.9%
0,52
g=1
BlondHair
14.9%
0.94
BrownHair
20.3%
0.69
0
Chubby
5.8%
0.88
0=1
EyeBags
20.4%
0.71
0=
Glasses
6.5%
0,80
9=1
GrayHair
4.2%
0.86
9-1
HighCheeks
45.2%
0.72
9
MouthOpen
48.2%
0.63
9
NarrowEyes
11.6%
0,56
9
Smiling
48.0%
0.65
Earrings
18.7%
0.97
0
WearingHat
4.9%
0.70
0=1
Average
24.1%
0.73
29

---

E.1 Methods
We extensively used the codebase of Wang et. 41 to conduct comparative experiment
Empirical Risk Minimization (ERM) 103
Acts as a baseline in our experiments where the goal is minimize the average error across the dataset without explicitly considering the sensitive attributes.
The goal is to learn an embedding that maximizes accuracy whilst minimizing any classifier's ability to recognize the protected class. The uniform confusion loss from Alvi et al. 74 is used following the implementation
Adversarial Training with Uniform Confusion [4]]
of [4].
Domain information is explicitly encoded and then the correlation between domains and class labels removed during inference.
Distrinniccri Training [41]
Trains a different classifier for each attribute where the classifiers do not see examples from other domains.
Domain-Independer Training
N 1 heads are trained to minimize the logistic loss over the target variables, where the total number attributes. A separate head minimizes the squared loss over the protected attribute Male. Fairness is enforced on validation data with two separate optimization criteria. OxonFair-DEO calls fpredictor.. fit (gm. accuracy, gm. gqquallopportuntty 0.01) to enforce Equal Opportunity. OxonFair-MGA calls fpredictor fit (gm. min cenn_accuracc.... min, gm cm..ccuracy, 0).
OxonFair + Multi-Head [21]: Described in Section
E.1.1
Compute Details
Computer vision experiments were conducted using a NVIDIA RTX 3500 Ada GPU with 12GB of RAM.
Table 13: Comparing accuracy of fairness methods while varying minimum recall level thresholds, 5.
CelebA 26 Auributes
8 0.75
0.85
8 = 0.90
8= 0.95
Baseline (ERM)
89.0
84.5
80.6
77.6
72.7
Adversarial
87.8
82.4
78.2
75.2
69.3
Donneen-eeeendent
76.8
72.4
68.6
Domain-Indeepenent
89.2
86.2
82.9
79.8
74.4
OxonFair
89.9
87.3
84.4
81.8
76.9
30

---

Table 14: Extended Version of Table Performance Comparison of Different Algorithmic Fairness Methods on the CelebA Test Set. Results monitor the mean Accuracy, Difference in Equal Opportunity (DEO) and the Minimum Group Minimum Label Accuracy across the attributes.
Uniconf. Adv 74
Domain
Domain
OxonFair DEO
OxonFair MGA
ERM
Disc. 41
Ind. 41
Gender-Independent Attributes
Acc.
93.1
92.7
93.0
92.6
92.8
90.9
Min grp. min acc.
64.1
72.3
76.5
71.2
72.3
85.8
DEO
16.5
19.6
14.6
7.78
3.21
3.52
Gender-Dependen Attributes
Acc.
86.7
86.1
86.6
85.6
85.8
82.3
Mín grp. min acc.
43,4
53.7
59.6
53.8
52.5
78.5
DEO
26.4
25.0
21.9
6,50
3,92
3.96
Inconsistently Labelled Attributes
Acc.
83.0
82.5
82.3
82.1
79.2
Min grp. min acc.
36.1
43.0
50.2
42.7
44.3
69.5
DEO
21.9
29.1
25.3
17.2
2.36
4.86
Table 15: Performance comparison of Baseline, Adaptive g-SMOTE, g g-SMOTE, OxonFair-DEO and OxonFair- MGA on the training set. Reported are the means over the 32 labels selected by [17 Methods marked are reported from Zietlow al. [17]
Protected Groups
ERM
Adaptive g-SMOTE
R-SMOTE
OxonFair-DEO
OxonFair-MGa
Full Training Set
Acc.
90.49
85.7T
87.27
89.21
86.15
Min. grp. acc.
61.74
68.06
61.81
54.20
78.48
DEO
24.70
12:27
21.91
3.93
5.58
Validation Set: Wearing Earrings
Test Set: Wearing Earrings
0.92
0.910
Accuracy
Accuracy
0.905
0.91
Frontiar
0.90
0.895
Original predictor
Upcated precictor
0.890
Domain Independent
0.89
Domain Discriminative
Uniconf Adversarial
0.88
0.3
0.6
0.880
0.0
Difference in Equal Opportunity (DEO)
Difference in Equal Opportunity (DEO)
Min Group Min Label Acc.
Min Group Min Label Acc.
0.5
0.4
0.3
0.2
0.805
0.090 0.895 0.900 0.905 0.910 0.915 0.920 0.925
0.87
0.88
0.89
0.90
0.91
Accuracy
Accuracy
Figure 14: comparison of the Pareto frontier validation and test data when enforcing faimess measures (DEO and Min Group Min Label Acc) for the Wearing Earrings attribute in CelebA whilst monitoring model accuracy.
31

---

Gender
Country
Ethnicity
Age
English
41200/7008/6927
44487/7744/7339
40731/6954/6845
39003/6628/6608
Polish
11782/1461/1441
2218/489/471
8567/1 199/1235
8610/1 199/1235
Spanish
2240/407/410
2299/436/439
2244/407/410
2249/407/410
Portuguese
1408/150/163
1105/198/197
1377/150/163
1389/150/163
Italian
273003550/369
3769/514/516
2706/348/368
2676/349/368
Table 16:
Multilingual Twitter corpus train/val/test statistics.
Coun- try=0
Coun- ty==1
Ethni- ity=0
Ethnic-
Gender=0
Gonder=1
Age=O
Age=1
it=1
English
5230/14461;
6856/18776;
4937/13279;
6550/18199;
4033/1764;
7855/13931;;
5901/14297;
6036/18614
1096/3010;
1447/3917;
1060/2966;
1357/38
851/2218:
1693/2905;
1236/2922;
1272/3883;
1074/3009
1459/3999
1056/2753
1341/3874
846/2226
1687/2996
1198/2990
1316/3663
371V3552;
18/3254;
215/2401;
1823248;
0/127:
5/29;
219/2401;
14/3248:
Polish
172/716;
13/730;
113/505:
14/730;
0/23;;
2/151;
117/505:
10/730;
194/787
29/674
103/266
31/673
0/253
7/153
113/526
21/673
3541997
505/639;
409/997:
Spanish
102/251;
71/159;
93/251;
80/159;
110/155:
68/100:
100/51;
73/159;
84/210
77/197
83/210
78/197
112/130
49/118
90/10
71/197
128/682;
18/134;
123/682;
23/134;
34/289
56/39;
166682;
30/1
Portugnese
24/60;
55/103;
29/60;
50/103;
24/40:
50168;
48/60;
31/103:
263/1127:
119/441:
209/112;
171/541:
10W748:
389367:
377/1221
3/541:
Itaiian
63/273;
19/96;
49/272:
33/96:
19/178:
70665;
81/272:
1/96:
63/244
23/106
42/243
44/106
19/169
83/71
86/242
0/106
Table 17: Multilingual Twitter corpus breakdown. We report the count of positive and total samples
across the train/val/test partitions for each ethnicity with specific values. We exclude samples where the label is marked as 'None' for particular ethnicity.
NLP Experiments
F1 Experimental Details
We employ a BERT-based model architecture [6 augmented with an additional head simul- taneously predict demographic factors (see Section cross-entropy loss for the primary prediction task and a mean squared error loss for the demographic predictions, aggregating these to compute the overall loss. We ensure data consistency by excluding entries with missing demographic information. To facilitate easy comparison with different models, we select the Polish language for the multilingual Twitter corpus, noted for its high DEO score, to demonstrate how various models can reduce this score. We also conducted our experiment on the Jigsaw data. Unlike the multilingual Twitter corpus, the Jigsaw religion dataset contains three groups: Christian, Muslim, and others. The entire model, including the BERT backbone, is fine-tuned for 10 epochs using an initial learning rate of 2 10 following the original BERT training setup. All experiments are conducted on an NVIDIA A100 SOGB GPU.
During training, we utilize the standard
F.2
Hate Speech Detection Task
We follow the methodology outlined in 84 to conduct the hate speech detection task using our tool. Variables such as age and country in the multilingual Twitter corpus are binarized using the same method described in [84]. The data splits for training, development, and testing are shown in Table 16
Multilingual Experiment. To demonstrate the capability of our proposed tool in handling multilin- gual scenarios, we conduct experiments across five languages: English, Polish, Spanish, Portuguese, and Italian and the results are shown in Table [8] Observations from the results indicate that: 1) Our model improves equal opportunity performance with minimal sacrifice to the main task performance. 2) The datasets in Polish and Portuguese show higher equal opportunity, indicating more severe bias compared other languages, yet our proposed method effectively enhances performance these conditions.
32

---

original DEO
updated DEO
original Accuracy
updated Accuracy
Gender
Country
0.85
English
5.13
3.19
84.0
84.2
0.80
Accuracy
0.80
Polish
21.4
10.1
89.6
85.8
0.75
0.75
Spanish
9.39
1.64
69.8
67.3
0.70
0.70
Portuguese
17.3
1.29
60.7
52.1
0.65
Italian
7.77
0.42
75.6
77.5
0.00
0.02
0.00
0.05
0.10
Equai Oppartunity
Equai Opportunity
Table 18: Multilingual Experiment.
Ethnicity
Age
0.85
0.85
original DEO
updated DEO
original Accuracy
updated Accuracy
Accurat
0.80
Accuracy
0.80
0.75
0.75
Gender
21.4
8.45
89.6
88.5
Rontler
Country
10.2
8,32
81.4
82.2
Updaced predkter
nal predicar
Ethnicity
8.56
4.92
83.1
82.7
0.65
0.65
Age
12.5
6.02
82.1
80.5
0.00
Equal Oppartunity
0.05
0.10
0.0
Equal Opportunity
0.1
0.2
Table 19: Demographic Experiments.
Figure 15: Demographics frontier plot.
Demographic Experiments. To demonstrate our tool's ability address various demographic factors in text, we conducted experiments focusing on age, country, gender, and ethnicity, with results detailed in Table 19 and Figure T3] The outcomes reveal that our tool effectively improves equal opportunities across all demographic factors, underscoring its capability to handle general debiasing scenarios.
Christian
Other
Muslim
Black
Asian
Train Valid
22845/1892
3783/554
9527/2390
Train
6718/2811
2187/246
5681/470
946/148
2425/578
Valid
1684/698
547/61
Test
2944/251
604/78
1119/319
Test
841/364
284/25
Table 20: Jigsaw religion data.
Table 21: Jigsaw race data.
F.3 Toxicity Classification Task
We also evaluate toxicity classification using the Jigsaw toxic comment dataset (85) which has been transformed into a Kaggle challenge. To demonstrate the ability of OxonFair to handle multiple protected groups, we consider religion the protected attribute and evaluate performance across three groups: Christian, Muslim, and Other. Owing to the limitted dataset size, all samples labelled as religion that was neither Christian nor Muslim were merged into Other and unlabeled samples were discarded. The statistics for this dataset are shown in Table 20 where each cell displays the count of negative and positive samples, respectively. The experimental results discussed in the main paper.
For the Jigsaw dataset, we follow the setup of 78]. selecting race as the protected attribute. We focus the subset of comments identified as Black Asian, as these two groups exhibit the largest gap in the probability of being associated with toxic comments. The data statistics are shown in Table 2] where each cell displays the count of negative and positive samples, respectively. The experimental results, presented in Table 23 demonstrate that our proposed tool outperforms all other models.
33

---

Meligion=Muslim
Celigion=Chrisian
Otligion=Other
Race=Black
Race=Asian
1917
1892/24737
554/4337
Irain
281 1/9599
246/2433
lest
319/1438
251/3195
78/682
lest 364/1205
25/309
valid
578/3003
4706115
148/1094
valid
698/2382
61/608
Table 22: Jigsaw corpus breakdown. We report the count of positive and total samples across the
train/val/test partitions for each religion and race with specific values. We exclude samples where the label is marked as 'None' for particular ethnicity.
Balanced Accuracy
Fl score
DEO
Accuracy
53.4
68.9
23.7
CDA
52.7
68.2
76.4
7.65
DP
23
47.4
64.6
72.6
4.35
EO
47.1
64.5
73.2
5.85
Dropout
88
52.4
68.0
72.0
12.7
Rebalance
51.7
67.5
74.4
5.57
OxonFair (Accuracy)
37.5
60.8
2.10
OxonFair (F1)
52.8
68.5
69.2
11.9
OxouFair (Balanced Accuracy)
52.7
68.5
68.5
0.41
OxonFair (Accuracy)
38.6
61.1
77.5
12.3
OxonFair (FI)
53.0
68.7
16.4
OxonFair (Balanced Accuracy)
53.2
68.9
67.8
20.5
Table 23: Jigsaw dataset: Race (w groups: Black, Asian).
34

---

G Comparison Table Information
In this section, we provide further details on the information from Figure While all approaches have many fairness definitions that can be computed, very few can be enforced via bias mitigation. As minimum, OxonFair supports enforcing the methods from tables 5 and [ eelimintinng duplicates give the number 14 in the table). In addition to this, it supports wide range of metrics that aren't ' used in the literature, for example minimizing the difference in balanced accuracy, Fl or Matthews correlation coefficient (MCC) between groups, e.g., by using balanced_acce _ccuucccccccccc difff as a constraint, It also supports the definitions set out Appendix [ including minimax notions; absolute bias amplification; and enforeing for minimum rates per group recall, precision, sensitivity actively promoting levelling-uy upp [8]
G.1 FairLearn Methods Support
Fairlearn provides an overview of the supported bias mitigation algorithms and supported fairness constraints their documentation [2] are dependent on the method.
The number performance and fairness objectives supported
Methods supported include ExpponeniateedGradient and GridSearch that provide wrapper around the reductions approach to fair classification of Agarwal et al. [3I]. Supported fairness definitions for classification are Demographic-Pariyy Equalized Odds, True Positive Rate Parity, False Positive Rate Parity and Error Rate Parity. For postprocessing the ThresholdOptimizer approach of Hardt et al. B supported. The adversarial approach of [32 is also supported and can enforce fairness based on Demographic Parity and Equalized Odds. The Correlationeemover method provides preprocessing functionality remove correlation between sensitive features and non-sensitive features through linear transformations. It should be emphasized that Fairlearn also provides an interface for defining custom Moments for fairness and objective optimization, however, as of the current version 0.10 no documentation or examples are provided for doing so.
G.2 AIF360 Methods Support
AIF360 provides support for a wide variety of methods 1 that enforce fairness, many of which overlap with Fairlearn. We consider group faireess approaches.
Preprocessing algorithms include DispiratelmpaacRemoveer
LFR [04], Optimized Pre-
processing [26]. Reweighting (25 and FairAdapt [105). Inprocessing algorithms include Adversari-
alDebiasing 322 PrejudiceRemover [T06), Exponentiated GradientReduction and GridSearchRedue-
tion 31. Postprocessing approaches include CalibraeddFaQOdssoootpppsessssse
44 EqOddsPostproo
cessing 3 RejectOpiionClassffccttton (25).
G.3 Societal Impacts
We reiterate the findings Balayn et al., who note that fairness toolkits can act as a double-edged sword 122 Open source toolkits can enable wider adoption of the assessment and mitigation of
bias and fairness related harms. However, if misused, these toolkits can create a flawed certification
of algorithmic fairness, endangering false confidence in flawed methodologies [55] 107). We join
growing calls in encouraging practitioners to be reflective in their use of fairness toolkits [60]. Specifically, urge practitioners adopt harms first approach to fairness and be reflective in their measurement and enforcement of fairness.
html
htt.:////eee.iioea..o.ooraroaaar html
35

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims in the abstract and introduction provide details on the paper's contribution and scope.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations are reported in the conclusions section.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide full descriptions of the information needed to reproduce the main claims and conclusions throughout the main body of the paper, the appendix and the substantial codebase for our toolkit.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We provide referenced details on how to access data and benchmarks used in our experimental demonstration. We provide code examples we simply train Py PyTochh implementations popular neural networks. We also provide all code for our toolkit including example notebooks.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Experimental settings are detailed in the main paper, appendix and also in the codebase. We use well known datasets that often have recommended splits.
7. Experiment Statistical Significance
Question:
Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The main contribution of this paper is a new library that offers increased func- tionality to data scientists and practitoners and not single new method. Our experimental demonstration highlights the ficxibility of OxonFair across domains and data modalaties (Computer Vision, NLP, Tabular Data)
8. Experiments Compute Resources
36

---

For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [No]
Justification: We include details about computer resources for each experimental section. We report time of execution when comparing against Fairlearn for multiple groups.
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: This paper has section in the conclusion that discusses the potential positive and negative societal impacts of this work. This is also expanded the appendix
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
12. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: A new library is presented which is supported by extensive documentation and example notebooks for practitioners. An Apache 2.0 licence is documented.
13. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
14. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
37


Reviews:
[{"review_id": "r0Qce3R7mX", "reviewer": "Reviewer_9LRe", "summary": "This paper presents AnonFair, a cutting-edge open-source toolkit designed to promote algorithmic fairness. Authors claim the following contributions:\n(1) Comprehensive support for NLP and Computer Vision classification, as well as standard tabular problems.\n(2) Enhanced robustness against overfitting challenges through the ability to enforce fairness on validation data.\n(3) Versatility in optimizing any measure that is a function of True Positives, False Positives, False Negatives, and True Negatives, making it easily adaptable and more expressive than other toolkits.\n(4) Seamless integration with popular ML toolkits such as sklearn, Autogluon, and pytorch.\n(5) AnonFair supports 9/9 and 10/10 of the group metrics of two prominent review papers and is accessible online at no cost.", "questions": "(1) Could you provide more high-level context for each of the experiments that you are running in order to make the paper more self-contained?\n(2) for NLP experiments, why do you think mitigation works for Twitter and not for Jigsaw?", "limitations": "Authors report some limitations, but further analysis on the experiments could raise more limitations that may be currently ignored.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This toolkit progresses in algorithmic fairness and enhances multidisciplinary collaborations, it is design to integrate the intervention of policy-makers.\n\nThe paper includes a complete section of experiments and comparison with existing toolkits. \n\nAnonFair key contributions include support to popular and relevant NLP and Computer vision areas.", "weaknesses": "* Lack of clarity in some reported experiments, e.g. results tables are not cited in the text, metrics are not well-contextualized (e.g. larger or lower scores are better?)\n\n* Lack of analysis, examples or human evaluation to better understand contributions and limitations of the method in each of the experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive comments and constructive feedback.\n\n---\n\n\n# Additional clarity in presentation    \n\nWe will use arrows in tables to indicate if larger **(↑)**, or lower **(↓)** scores are better. We will also discuss this when mentioning the different fairness metrics to improve accessibility for readers. For example, when using Difference in Equal Opportunity (a smaller score is better/fairer). To improve presentation, we will also add references to the results table and relevant appendices to the main body of the paper.  \n\n---\n\n# Questions and additional analysis of results                    \n\n\n\n\n>**Question (1)**: Could you provide more high-level context for each of the experiments that you are running in order to make the paper more self-contained? \n\nWe will add text to these experiments. \n\nIn brief, the computer vision and NLP experiments (tables 1 through 4), are standard fairness benchmarks where existing methods compete to minimize Equal Opportunity (i.e. difference in recall between groups) while maintaining high accuracy. This corresponds to a situation where high accuracy is important, but you don’t want the burden of low recall to disproportionately fall on particular groups. A scenario where this might be important is medical testing where you want to ensure that there is a similar recall rate for all groups.  \n\nResults in the appendices show the expressiveness of the toolkit. We simply want to show that these well-cited measures are optimizable using our approach. \n\n---\n\n>**Question (2)**: for NLP experiments, why do you think mitigation works for Twitter and not for Jigsaw?  \n\nGood question. While we do improve fairness on Jigsaw, it is not as reliable as on Twitter. One of the nice things about our approach is that we directly enforce constraints on validation data and given sufficient data these results should generalize to unseen test data.  This does not happen on Jigsaw with the same reliability we see on Twitter. Largely this can be attributed to data limitations. Equal Opportunity is inherently an unstable measure (it looks at the difference in recall between groups) where both the number of individuals in a particular group is small and the ratio of hate speech is low, we can have very limited data for measuring recall let alone differences in recall. \n\nWhile limited validation data could be worked around using techniques such as cross-fold validation (this is compatible with our approach, and integrating it is ongoing work), Jigsaw comes with a pre-existing test split, and this was frequently unrepresentative of the combined train and val set. This problem is exacerbated as unlike Twitter, Jigsaw contains scenarios with more than two protected groups. \n\nWe will add an appendix with data counts and discuss the limitations clearly there. \n\n---\n\n>Does this additional analysis raise new limitations?\n\nWe briefly touch on the issue of data in our limitations section, but we did not have sufficient space to discuss this in detail. We’ll add text based on the above answer to the appendix and link to it from the experimental section.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Fzy3CesDFd", "reviewer": "Reviewer_AKgy", "summary": "The paper describes details of a fairness toolkit (\"AnonFair\"), which confers fairness to any given machine learning classifier by exploring a wide range of prediction thresholds for different groups (which are either provided upfront or inferred through an auxiliary classifier). The toolkit is designed to be quite expressive, as it can optimize several different metrics, e.g., false positives/negatives, true positives, etc. The toolkit can work across all classifiers (which can output class probabilities), including ones trained on vision and NLP tasks.", "questions": "I feel that the current version is a good starting point (in terms of implementation) of existing fairness techniques and speeding them up and trying them out on vision and NLP tasks. To improve the paper, I would suggest clearly outlining the important problems that this toolkit now can enable researchers to answer (which was not possible before) and answer a few of those questions in the paper.", "limitations": "I believe the paper adequately communicates their shortcomings and cites past references when using them. However, I think it might help to also acknowledge that the underlying fairness techniques broadly apply to a wide range of classifiers, and naturally extend to classifiers in computer vision and NLP domains. Reading parts of the paper felt like that there are significant challenges in adoption of fairness techniques to NLP and CV, and this paper overcomes them through novel solutions—which is not the case.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper introduces and describes a toolkit that implements several fairness strategies and can support any fairness measure that can be expressed in terms of true positives, false positives, true negatives and false negatives. These techniques primarily rest upon adjusting the classification thresholds of different groups, and the paper also incorporates tricks to speed up their computations of precision and recall across different thresholds. The fairness techniques that this paper implements are (largely) classifier agnostic, and can be applied to a wide range of classifiers including NLP and vision classifiers (as this paper shows). Overall, I appreciate that expressivity and broad applicability of their toolkit.", "weaknesses": "While the toolkit might turn out to be useful for some practitioners, it is a relatively straightforward implementation of well-known (and simple) technique of adjusting prediction thresholds across groups. Exploring different thresholds can be computationally prohibitive, for which the authors use a standard trick to speed up their explorations (which I appreciate). The paper acknowledges and cites relevant papers/techniques that they implement.   Overall, the originality and novelty of their work is significantly limited, as the toolkit is an implementation of known and simple fairness techniques. Further, the underlying fairness techniques (not from the authors) are themselves applicable to most classifiers, so any implementation of the same could work for NLP and vision tasks—which is claimed to be one of the major contributions of this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their review, and we hope to address the issues raised. \n\n­In brief, there are two main issues we wish to discuss. 1. what this toolkit does 2. The limited novelty of any toolkit/library, and that such libraries are explicitly covered in the call for papers. \n\n1. We are concerned that the review substantially understates what the toolkit does. While our toolkit is a post-processing method, it does not simply ‘set per group thresholds’ and works when group annotations are unavailable at test time. To this end, we offer two modifications: \n     (a) A classifier agnostic approach that uses an auxiliary classifier to estimate groups, while enforcing fairness for the true groups – and not the estimated groups.   \n     (b) Model surgery for neural networks (section 4.2, based on work by  [4]). This involves training a multi-headed network that jointly estimates groups alongside the original task. These heads are then merged, resulting in a single fair network with one head and the same architecture as the original network. \n     Both approaches have only been previously shown to work for demographic parity [4,5] and not for any other fairness measure.\n\n2. The NeurIPS 2024 Call for Papers explicitly calls for “libraries, improved implementation and scalability” under the infrastructure theme. Software libraries have been published in the NeurIPS main track [1-3]. Like any good toolkit we prefer well-tested and understood components, over speculative new methods. Rather than rehashing an existing argument, we refer the reviewer to this publicly available debate on Openreview from an accepted Neurips toolkit paper last year [1].\n  \n­We still believe that novelty is important, but novelty should be evaluated with respect to other toolkits, and ask what can this work do that other toolkits cannot?  \n\nCompared to others, this is the only toolkit that works on computer vision and NLP, and is substantially more expressive than anything else out there. This generalization to NLP, and computer vision, works without group membership at test-time, and is not a natural extension of per group thresholding.  \n\nAmong many other measures, we are the only toolkit that simultaneously supports conditional fairness metrics, minimax fairness, and levelling up through minimum rate constraints. Unlike other toolkits, we jointly and efficiently optimize a performance objective. This not only minimizes degradation while enforcing fairness but can improve the performance of inadequately tuned unfair baselines. Our toolkit is compatible with more frameworks than existing approaches including scikit-learn, Autogluon, and PyTorch.  \n\n­\n\n> To improve the paper, I would suggest clearly outlining the important problems that this toolkit now can enable researchers to answer (which was not possible before) and answer a few of those questions in the paper. \n\nPlease see levelling up [6] examples in figure 4, and the adjacent table, figure 8 and appendix C.3. Minimax fairness [7] in appendix C.1, and table 8. Conditional metrics in appendix C.4.  Directional Bias Amplification [8] in appendix C.5. And a whole host of additional measures taken from the fairness review paper [9] in table 7. \n\n# References\n\n[1] Griffiths, Ryan-Rhys, et al. \"GAUCHE: a library for Gaussian processes in chemistry.\" Advances in Neural Information Processing Systems 36 (2023). \n\n[2] Jamasb, Arian, et al. “Graphein-a python library for geometric deep learning and network analysis on biomolecular structures and interaction networks.” Advances in Neural Information Processing Systems 35 (2022): 27153-27167. \n\n[3] Pineda, Luis, et al. “Theseus: A library for differentiable nonlinear optimization.” Advances in Neural Information Processing Systems 35 (2022): 3801-3818. \n\n[4] Lohaus, Michael, et al. \"Are two heads the same as one? Identifying disparate treatment in fair neural networks.\" Advances in Neural Information Processing Systems 35 (2022): 16548-16562. \n\n[5] Aditya Krishna Menon and Robert C Williamson. The cost of fairness in binary classification. In Conference on Fairness, accountability and transparency. PMLR, 2018.  \n\n[6] Mittelstadt, Brent, Sandra Wachter, and Chris Russell. \"The Unfairness of Fair Machine Learning: Levelling down and strict egalitarianism by default.\" arXiv preprint arXiv:2302.02404 (2023). \n\n[7] Martinez, Natalia, Martin Bertran, and Guillermo Sapiro. \"Minimax pareto fairness: A multi objective perspective.\" International conference on machine learning. PMLR, 2020. \n\n[8] Wang, Angelina, and Olga Russakovsky. \"Directional bias amplification.\" International Conference on Machine Learning. PMLR, 2021. \n\n[9] Hardt, Michaela, et al. \"Amazon sagemaker clarify: Machine learning bias detection and explainability in the cloud.\" Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining. 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wfWExSdRPF", "reviewer": "Reviewer_AiSk", "summary": "The paper introduces a new toolkit designed to enhance algorithmic fairness with greater expressiveness. Unlike existing toolkits, this one offers more customization options to optimize user-defined objectives and fairness constraints. Although the proposed toolkit currently includes only one method, it supports both computer vision and natural language processing (NLP) tasks. The authors compare the efficiency of this method, finding that the toolkit is relatively more efficient than Fairlearn. Comprehensive experiments were conducted on various datasets, and the results were compared with those from other popular toolkits.", "questions": "- In Figure 3, the proposed toolkit appears to encounter scaling issues when reaching 5 groups. Could you provide more details on why this occurs and elaborate on the underlying reasons for this limitation?\n- The paper presents results on multilingual datasets. Do you have any specific findings for each language, particularly regarding the effectiveness of the toolkit for individual languages?", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper introduces a versatile toolkit that supports both NLP and computer vision tasks, unlike existing toolkits which lack this capability.\n- The proposed toolkit employs efficient optimization techniques that accelerate the evaluation process.", "weaknesses": "- The formulation presented in Subsection 4.2 of the paper is limited to a single-layer model, which restricts its applicability across different machine learning models. To enhance the flexibility of the method, I recommend adopting a more generic notation, particularly if we aim to incorporate pretrained language models.\n- The abstract is quite unclear, especially the part that mentions \"9/9 and 10/10 of the group metrics of two popular review papers.\" I suggest rephrasing the abstract for better clarity and comprehension.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed feedback. We are happy to see that the reviewer appreciates the versatility of our toolkit beyond existing solutions and the efficiency of optimization in the toolkit.  \n\n**Clarification on notation:** The equations in section 4.2 contain a function, $B(x)$. This represents an arbitrary non-linear backbone shared between the two heads. While the heads are linear (as is common in multitask learning), arbitrary non-linear functions can be learned. This approach is used as written for the NLP and Computer Vision tasks which use complex non-linear backbones. We will emphasize this in the text. \n\n**Improvements to Abstract:** Thanks. We will rephrase the abstract to improve clarity.  \n\n# Questions \n\n> In Figure 3, the proposed toolkit appears to encounter scaling issues when reaching 5 groups. Could you provide more details on why this occurs and elaborate on the underlying reasons for this limitation?\n  \n\nThe slowdown as we increase the number of groups, is expected and supported by the analysis in Section 4 (lines 157-162) which shows that the run-time is exponential in the number of groups. This was a conscious decision to maximize expressibility at the expense of compute for a large number of groups. Having sufficient data to enforce fairness for a large number of protected groups is unfortunately rare in algorithmic fairness and in practice, these scaling issues are rarely encountered.  \n\n>The paper presents results on multilingual datasets. Do you have any specific findings for each language, particularly regarding the effectiveness of the toolkit for individual languages?  \n\nOur toolkit is language and data modality agnostic. While we see differences in performance, this seems to be predominantly driven by the quality of the data. In general, the most common languages have much more data available (particularly  English with three times more data than the next most available language), and this allowed curators to create datasets with a balanced subset of groups, and positive and negative labels. For example, in the case of well-resourced languages such as English (.499 of datapoints were labelled Female) and (.495 of English datapoints were labelled non-White). In contrast for Polish protected groups were very unbalanced (.324 Female) and (.105) non-White. \n\nThis means both that base classifiers for well-resourced languages tend to be fairer (at least when evaluated on the provided test set) but also that the estimation of statistics such as recall are more stable, and our fairness toolkit generalizes better to the test set. In Appendix E.2 Multilingual Experiment (from line 813), we find that the base classifiers in Polish and Portuguese show higher difference in equal opportunity, indicating more severe bias compared to other languages (e.g., English). Where there is such a large initial bias, our toolkit can make larger improvements in fairness, even  if the underlying statistics cannot be reliably estimated.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "suISL3caiH", "reviewer": "Reviewer_JNWw", "summary": "This paper describes a new toolkit for algorithmic fairness, enabling the optimization of any fairness measure that is a function of the confusion matrix. Experiments on vision and NLP demonstrated the effectiveness of the proposed toolkit.", "questions": "- L5: \"True positives, false positives, ...\" => \"the confusion matrix\"\n - L6: \"extendable\" => \"extensible\"", "limitations": "The authors adequately discussed the limitations of their toolkit.", "rating": 7, "confidence": 1, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "An easy-to-use toolkit for enforcing algorithmic fairness.", "weaknesses": "Presentation could be made more self-contained, e.g. a table listing the supported fairness metrics, as functions of the confusion matrix. This would help readers not familiar with the field.\n\nIt seems that only binary classification is supported. How can such metrics be extended to other tasks?\n\nSome minimal code snippets for the interface could be shown as examples.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the feedback and helpful suggestions that will be integrated to improve the paper.  \n\n**Improvements in presentation:** We will add the definition of equal opportunity (the most common fairness definition, corresponding to difference in recall between groups) to the work. Honestly, we have substantial problems with space constraints in this work. The two review papers we cite [1,2] only list and discuss metrics and come in at a combined 15+ double-columned pages. We will try to include the most common or most relevant metrics in our work, but we are not a review paper, and providing references for most of the metrics is probably the most sensible option. \n\n**Beyond binary classification:** Tasks beyond binary classification are on our roadmap for future work. Fair ranking is of particular interest, as failures in ranking have clear well-defined harms and may be compatible with our accelerated approach. For multiclass classification, from a policy perspective, there are a couple of metrics that are generalizations of demographic parity (every labelling rate should be the same for all groups), and equalized odds (confusion matrix should be the same for all groups), but they seem to have been chosen for mathematical convenience, rather than because they correspond to clear harms.  \n\n**Minimal Code Snippets:** We agree that minimal code snippets would improve the paper. We have code snippets in the appendix (Appendix C.2 and figure 8). and we will add further examples. In addition to code snippets, we also have Jupyter Notebook tutorials and example studies for practitioners to get up and running with our toolkit. We will highlight the vast availability of resources and examples that our toolkit provides, and our toolkit empowers the community to build on these.  \n\n\n**Typos:** Thanks for pointing out typos, these will be promptly corrected.  \n\n[1] Verma, Sahil, and Julia Rubin. \"Fairness definitions explained.\" Proceedings of the international workshop on software fairness. 2018. \n\n[2] Hardt, Michaela, et al. \"Amazon sagemaker clarify: Machine learning bias detection and explainability in the cloud.\" Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining. 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eHIhFf9cWw", "reviewer": "Reviewer_aniZ", "summary": "The paper introduces \"AnonFair,\" a toolkit designed to enforce algorithmic fairness across various domains, including NLP, computer vision, and traditional tabular data. It is compatible with popular machine learning frameworks like sklearn, AutoGluon, and PyTorch. Unlike well-established fairness tools like FairLearn and AIF360, AnonFair extends to different types of data, including NLP and vision.\n\nOther tools offer many methods but limited control over them, while AnonFair uses a single, highly customizable method that allows for per-group thresholding.\n\nIt specifically addresses the issue of overfitting by utilizing validation data, making it more reliable when traditional methods might fail.\n\nEmpirical evidence presented shows that AnonFair performs well, often matching or surpassing other methods in fairness benchmarks without being specifically optimized for complex or high-dimensional scenarios.\n\nAnonFair seems to provide a robust and adaptable solution for implementing fairness in machine learning, in ways that other tools do not currently offer.", "questions": "1. The paper mentions that hard assignment is more efficient than soft assignment, while appendix A adds some operational details, it remains unclear how these methods specifically compare in terms of quantitative metrics. Could the authors provide specific metrics or comparisons that demonstrate the efficiency and performance benefits of hard assignment?\n2. The discussed limitations reads a bit out of context given provided evidence in the paper. What makes the mentioned  solutions suboptimal, and how significant are these shortcomings?  Also it was not clear to me, after finishing reading,  when it is adequate to use this tool and what could be use cases when it fails. Including this into the conclusion could make the reader grasping the full picture. \n3. Is Figure 6 part of the Appendix or misplaced?", "limitations": "Some of the limitations are acknowledged, but could be expanded with more actionable insights. \nA call to action for community engagement, such as through open-source collaboration would also encourage broader impact and adoption of AnonFair against its competitors. \nIt would be beneficial if the authors suggested potential improvements or future research directions for the suboptimal fairness metrics and data scarcity issues mentioned.\nThe broader impact section identifies ethical concerns well. However, detailing the intended applications and scenarios where AnonFair might be most effective, or where it could fail, would provide readers and users with clearer guidance on its practical use and limitations.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper does well in positioning AnonFair against competing tools by demonstrating its performance on standard fairness metrics and its versatility across a variety of use cases.\n- AnonFair supports NLP and computer vision classification tasks, allowing broader applicability.\n- The toolkit uses validation data to combat overfitting, ensuring that fairness measures remain robust across both training and unseen data.\n\n- The toolkit not only competes well in terms of accuracy and fairness metrics but also offers significant advantages in computational efficiency.", "weaknesses": "- Some sections are overly detailed, such as the introduction, while others are missing necessary depth:\n    - Section 3 could use a clearer structure, possibly with a diagram, to help readers understand how to interact with the toolkit.\n    - The section on toolkit expressiveness needs more detailed examples and explanations of how the supported fairness measures are implemented. \n    - Results discussion is kept very brief and could benefit from specific numerical examples, like percentage improvements compared to other methods.m actual numbers, such as how much % improvement in comparison to method XY and such.\n\n- The paper assumes readers are familiar with fairness terminology and metrics without adequate explanations or definitions for some acronyms (e.g., DEO in Table 3 and 4).\n    - Subsection 4.3 lists supported fairness measures but fails to provide examples or brief explanations, making it less informative for those not familiar with these terms.\n\n- Lack of consistency in terminology usage; for example, \"EOp\" in Figure 1 (top right) vs. \"EO\" in Section 5.2, “AnonFair” missing before \"Frontier\" in Figure 1 (left), and inconsistent references like \"See Figure\" vs. \"See fig..\"\n\n- A stronger call to action for community engagement, such as through open-source collaboration or empirical validation studies, could significantly enhance the broader impact and encourage more widespread adoption and refinement of AnonFair.\n\n- The paper would benefit from a summary of explicit cases and recommendations advising users on the best scenarios for using the tool.\n\n- Figure 2 is not referred to in the paper, or did I miss this part.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our manuscript and for providing detailed, helpful, and constructive feedback. We hope to address outstanding weaknesses and concerns below.  \n\n**Improving presentation:** The idea of a figure/flow chart is a good one, but there is insufficient space in the paper. We will add this directly to the toolkit documentation instead, where it is more likely to be seen by someone interested in using the toolkit. \n\n**Details on implementing fairness measures:** Please see Appendix B for details of how measures are implemented. In brief, you can just write down a function of the confusion matrix. This is computed per group, and standard measures such as minimum per group, or average difference or ratio of measures between groups are automatically computed. These correspond to different notions of fairness. We will add an additional example of this for Equal Opportunity, as this is one of the most widely used group fairness measures.  \n\n**Results discussion and numerical examples:** We will discuss specific examples from the tables and figures where our toolkit shows clear improvements over other approaches. \n\n**Improving the consistency of fairness terminology and abbreviations:** Thanks for flagging the inconsistency. We will correct this. \n\n**Call to action for community engagement:** Thank you. We will add a call for community engagement in the abstract. We strongly agree with the reviewer’s focus on making the toolkit more accessible through documentation. Unfortunately, space constraints mean that much of the information is going to lie outside the paper. We have spent much of the past months since submission extending the documentation and adding additional examples. This will continue to be a focus going forward. \n\nWe invite community-driven contributions principally as pull requests reflecting the needs and considerations that researchers come across in practice. In this manner, we can support more advanced features without overwhelming the codebase and maintenance requirements. Our toolkit is now a pip installable library, and the community can raise issues, concerns, and requests on a GitHub repository. In this way, practitioners can collaboratively build on each other's work.  \n\n**Explicit cases and recommendations advising users on best practices. What are the adequate uses of this tool and what could be the use cases when it fails?** \n\nThese are very important questions. The short answer is that this is too important to be compressed into the conclusion of the paper. \n\nThis toolkit arose from a promise to a healthcare provider that we would go through existing toolkits and return a list of best practices. Essentially, our conclusion was, it didn’t matter what we suggested as no existing toolkit worked for NLP or computer vision, and those that existed for tabular data couldn’t enforce measures like bias amplification, conditional measures, or levelling up. The purpose of this paper is to lay out a technical toolkit which is expressive enough to support both our recommended best practices, and hopefully the best practices of other researchers who disagree with us. Follow-up white papers will set out how we think it should be used. Use cases, best practices and documentation are our core focus going forward. \n\n \n**Figure 2 not referred to in the main paper:** We will add references to this figure in Section 4 (specifically lines 156 and 189 and footnote 5).  \n\n# Questions\n\n**Question 1:** For a performance comparison, we reran the unconditional fairness metrics from table 7 with inferred attributes and the slow vs fast pathway. Using default parameters, the fast pathway has an average of 1.2% better fairness across the metrics and wins in 7/10 of the cases, while maintaining similar accuracy (win rate 5/10, an average decrease in 0.28% of accuracy).  \n\nFor an efficiency comparison, we reran the comparison with fairlearn, on adult, varying the number of groups. \n\n| # Groups  |5|4|3|2|\n|-|-|-|-|-|\n| FairLearn |33.2s|33.3s|25.4s|19.4s|\n|Fast| 0.7+55.7s|0.7+0.93s|0.7+0.078s| 0.7+0.05s|\n|Slow| - | 0.7+454s| 0.7+22s|0.7+1.5s|\n\n\nWe will add these numbers to the paper. \n\n**Question 2:** Apologies. The fact that thresholding is suboptimal for Equalized Odds is widely known in the literature [1]. The suboptimality comes from the fact that Equalized Odds is a combination of two fairness measures (difference in true positive rates and difference in false positive rates) and that ideal thresholds for one measure need not minimize the other. Instead, a combination of thresholding and randomization is often used. Our approach is only known to be sub-optimal when the true attributes are used, and not with data that requires the use of inferred attributes. The rest of the time, Anonfair has no guarantees regarding optimality.  This is the same as most widely used fairness methods including FairLearn. Unfortunately, the only answer to if our approach should be used is the same as for the rest of machine learning – the experiments might look good, but where alternatives exist, it is best practice to compare them on your use case. \n\nThe only case where our method can be said to fail is where insufficient data is provided to generalize from validation to unseen test data. Otherwise, the method may be suboptimal (better fairness accuracy trade-offs might exist), but it will work, and similar fairness/accuracy trade-offs will be seen on unseen test data. \n\n**Question 3:** Figure 6 is part of the appendix and is an expanded Figure 1 from the main body. While Figure 1 compares fairness toolkits with a focus on optimizing for balanced accuracy, Figure 6 broadens this comparison by including additional metrics.\n\n[1] Hardt, Moritz, Eric Price, and Nati Srebro. \"Equality of opportunity in supervised learning.\" Advances in Neural Information Processing Systems 29 (2016).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their helpful and largely positive comments (**overall scores 7,6,6,6,3**). The suggestions are informative, and we will adjust presentation in the paper wherever an issue has been raised.
\
\
Our toolkit provides a “robust and adaptable solution for implementing fairness in machine learning, in ways that other tools do not currently offer” (**reviewer aniZ**). We “include support to popular and relevant NLP and Computer vision areas” (**reviewer 9LRe**) and this is “unlike existing toolkits which lack this capability” (**reviewer AiSk**). We provide “an easy-to-use toolkit for enforcing algorithmic fairness” (**reviewer JNWw**) and reviewers “appreciate the expressivity and broad applicability” of the toolkit (**reviewer AKgy**). We contribute “a complete section of experiments and comparison with existing toolkits” and our toolkit represents “progress in algorithmic fairness and enhances multidisciplinary collaborations” (**reviewer 9LRe**).   
\
The score 3 (**reviewer AKgy**) arises from a concern regarding lack of novelty. This is entirely understandable. This is a library submission and as such, most of what we are doing is putting together existing pieces in a useful way and performing good engineering to substantially increase expressiveness and performance (all of which the review acknowledges and appreciates). However, the NeurIPS 2024 Call for Papers explicitly includes “libraries, improved implementation and scalability” under the infrastructure theme. These software libraries have been published in the NeurIPS main track [1-3]. 


Beyond this, the review misses some of our more substantial contributions. We are not simply adjusting thresholds. All NLP and computer vision experiments are the result of network surgery (see section 4.2), where a new fair network is generated with the same underlying architecture as the base network. These experiments do not use protected attributes at test-time.  

Moreover, the review requested additional experiments showing fairness definitions that this toolkit, and no other toolkit can solve.  These can be seen in tables 6,7 and 8, figure 4, and appendices C.1, C.3, C.4, and C.5.  

Otherwise, all reviewers are in agreement about our contribution. This is the only fairness toolkit to work for computer vision or NLP and represents a step forward not just in terms of the domains where it has been applied but also in the wide range of fairness constraints that it can solve.


# References  

[1] Griffiths, Ryan-Rhys, et al. "GAUCHE: a library for Gaussian processes in chemistry." Advances in Neural Information Processing Systems 36 (2023). 

[2] Jamasb, Arian, et al. “Graphein-a python library for geometric deep learning and network analysis on biomolecular structures and interaction networks.” Advances in Neural Information Processing Systems 35 (2022): 27153-27167. 

[3] Pineda, Luis, et al. “Theseus: A library for differentiable nonlinear optimization.” Advances in Neural Information Processing Systems 35 (2022): 3801-3818.

Meta-review: The paper presents AnonFair, a new open-source toolkit for enforcing algorithmic fairness that supports NLP and computer vision classification tasks in addition to tabular problems. Such broader applicability to different data types is a key advantage over existing fairness toolkits.

AnonFair can enforce fairness on validation data, making it more robust to overfitting challenges compared to other approaches. It supports 9/9 and 10/10 of the group metrics from two prominent fairness review papers.

AnonFair is compatible with popular ML frameworks like scikit-learn, AutoGluon, and PyTorch.

**Strengths** Broader applicability to NLP and computer vision tasks compared to existing fairness toolkits, with more expressive and customizable fairness measures. It features improved robustness to overfitting by using validation data and competitive performance on standard fairness benchmarks

**Weaknesses**

Limited novelty in the underlying fairness techniques (mainly implements known methods). It currently only supports binary classification tasks
Scaling issues with larger numbers of protected groups



Overall,
this is a solid toolkit paper with useful contributions, particularly in extending fairness approaches to new domains like NLP and computer vision. The main concern is the limited technical novelty but the practical utility and engineering contributions outweigh this limitation for a toolkit paper.
Decision: Accept (poster)