Title: Fairness without Harm: An Influence-Guided Active Sampling Approach
Venue: NeurIPS 2024
Accepted: yes
Keywords: Fairness; Sampling
Abstract: The pursuit of fairness in machine learning (ML), ensuring that the models do not exhibit biases toward protected demographic groups, typically results in a compromise scenario. This compromise can be explained by a Pareto frontier where given certain resources (e.g., data), reducing the fairness violations often comes at the cost of lowering the model accuracy. 
In this work, we aim to train models that mitigate group fairness disparity without causing harm to model accuracy.
Intuitively, acquiring more data is a natural and promising approach to achieve this goal by reaching a better Pareto frontier of the fairness-accuracy tradeoff. The current data acquisition methods, such as fair active learning approaches, typically require annotating sensitive attributes. However, these sensitive attribute annotations should be protected due to privacy and safety concerns. In this paper, we propose a tractable active data sampling algorithm that does not rely on training group annotations, instead only requiring group annotations on a small validation set. Specifically, the algorithm first scores each new example by its influence on fairness and accuracy evaluated on the validation dataset, and then selects a certain number of examples for training. 
We theoretically analyze how acquiring more data can improve fairness without causing harm, and validate the possibility of our sampling approach in the context of risk disparity. We also provide the upper bound of generalization error and risk disparity as well as the corresponding connections.
Extensive experiments on real-world data demonstrate the effectiveness of our proposed algorithm. Our code is available at [github.com/UCSC-REAL/FairnessWithoutHarm](https://github.com/UCSC-REAL/FairnessWithoutHarm).

Fairness Without Harm:

Abstract

The pursuit of fairness in machine learning (ML), ensuring that the models do not exhibit biases toward protected demographic groups, typically results in a compromise scenario. This compromise can be explained by a Pareto frontier where given certain resources (e.g., data), reducing the fairness violations often comes at the cost of lowering the model accuracy. In this work, we aim to train models that mitigate group fairness disparity without causing harm to model accuracy. Intuitively, acquiring more data is a natural and promising approach to achieve this goal by reaching a better Pareto frontier of the fairness-accuracy tradeoff. The current data acquisition methods, such as fair active learning approaches, typically require annotating sensitive attributes. However, these sensitive attribute annotations should be protected due to privacy and safety concerns. In this paper, we propose a tractable active data sampling algorithm that does not rely on training group annotations, instead only requiring group annotations on a small validation set. Specifically, the algorithm first scores each new example by its influence on fairness and accuracy evaluated on the validation dataset, and then selects a certain number of examples for training. We theoretically analyze how acquiring more data can improve fairness without causing harm, and validate the possibility of our sampling approach in the context of risk disparity. We also provide the upper bound of generalization error and risk disparity as well as the corresponding connections. Extensive experiments on real-world data demonstrate the effectiveness of our proposed algorithm. Our code is available at github.com/UCSC-REAL/FairnessWithoutHarm .

1 Introduction

Machine Learning (ML) has dramatically impacted numerous optimization and decision-making processes across various domains, such as credit scoring [62] and demand forecasting [14]. Algorithmic fairness embraces the principle, often enforced by law and regulations, that the decision-maker should not exhibit biases toward protected group membership [84], identified by characteristics such as race, gender, or disability. However, the pursuit of fairness unavoidably results in a compromise scenario where reducing the fairness violations usually leads to a degradation in accuracy, which has been observed and verified by numerous literature [52, 25, 81, 69, 83]. Theoretically, the phenomenon can be understood through a Pareto frontier on the tradeoff between group fairness and accuracy

[72, 51, 10, 74]. That is, as illustrated in Figure 1, given certain resources such as training data, when a model has reached a point on the Pareto frontier, without more data resources, it is impossible that one can improve fairness without worsening model accuracy.

One major source of unfairness and a major cause of the fairness-accuracy tradeoff is biased training data. If an unbiased and 'fairer' dataset is available, we will be hopeful that unfairness can be alleviated without compromising accuracy. Furthermore, such a 'fairer' dataset would allow for obtaining a fair and accurate model through the standard empirical risk minimization (ERM) with cross-entropy (CE) loss. The above observation points to a promising way to improve fairness via actively acquiring more informative data, aiming to shift towards a better Pareto frontier of the fairness-accuracy trade-off [1, 46]. However, existing approaches that seek more data, such as fair active learning [4], typically require annotating sensitive attributes for training data. In practice, these sensitive attribute information such as race and gender, should be protected due to privacy regulations [35, 5, 66]. In the normal active learning scenario, collecting more data with sensitive attributes heightens privacy and safety risks due to the increased probability of leaking sensitive information.

Therefore, we ask the following question: When not disclosing more annotations of training sensitive attributes, how can we acquire more data to improve model fairness without sacrificing accuracy?

In this paper, we propose a tractable active data sampling algorithm in a training sensitive attributes-free way, which solely requires sensitive attributes on a small validation set . Particularly, the algorithm evaluates each example's influence on fairness and accuracy using the validation dataset for ranking and then selects a certain number of examples to supplement the training set for training. We name our solution Fair Influential Sampling (FIS). The core challenge is approximating the corresponding influences of each new example without accessing its sensitive attributes. Technically, we evaluate the importance (influences) of each new example by comparing its gradient to that derived from the entire validation set. This comparison helps quantify the hypothesized change of group fairness disparity metric when adding this example to the training set. As a result, the requirement

0.20

0.21

0.22

0.23

0.24

error

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

fairness violation

rich data

scarce data

original point

new point

shift

of training sensitive attributes can be relaxed, as gradient derivation serves as a role of fairness constraints to measure the group fairness disparity. The main contributions of our work are summarized as follows.

We develop a tractable active data sampling algorithm (Algorithm 1) that does not rely on training sensitive attributes. The algorithm scores each new example based on the combined influences of prediction and fairness and then opts for a certain number of examples for training. [Section 4]

We theoretically analyze how acquiring more data can improve fairness without harm from a distribution shift perspective view, and validate the possibility of our sampling approach in the context of risk disparity. We also provide the upper bound of generalization error and risk disparity as well as the corresponding connections (Theorem 5.1 and Theorem 5.2). [Section 5]

Empirical experiments on real-world datasets (CelebA, Adult, and COMPAS) substantiate our claims, indicating the effectiveness and potential of our proposed algorithm in achieving fairness for ML classifiers. [Section 6]

2 Related work

Fairness-accuracy tradeoff There are numerous works that have been successful at mitigating fairness disparities [24, 32, 2, 78, 69]. However, these works typically rely on protected sensitive

attributes of training examples to measure the fairness disparities across groups. Moreover, a fairnessaccuracy tradeoff has been shown, meaning that enforcing fair constraints heavily degrades the model performance [52, 25, 81]. Notably, Chen et al. [20] characterized the change of the fairness violation when the data distribution is shifted. Except for training sensitive attributes, this paper does not work in the classical regime of the fairness-accuracy tradeoff. By properly collecting new data, we can improve both accuracy and fairness, which cannot be achieved by working on a static training dataset that naturally incurs such a tradeoff. Besides, compared to prior works [52, 56], our method does not require additional assumptions about the classifier and the characteristics of the training/testing datasets (e.g., distribution shifts). Relevant work [42] utilizes the influence function to reweight the data examples but requires re-training. Our method focuses on soliciting additional samples from an external dataset while [42] reweights the existing and fixed training dataset.

Active learning The core idea of active learning is to rank unlabeled instances by developing specific measures, including uncertainty [41, 45], representativeness [22], inconsistency [70], variance [34], and error [59]. A related line of work [48, 71, 31, 76] concentrates on ranking unlabeled instances based on the influence function. Compared to these studies with a focus on prediction performance, our work poses a distinct challenge taking into account fairness violations. Our approach is more closely with the fair active learning approach [4]. However, this framework still relies on training sensitive attributes and then unavoidably encounters the tradeoff between fairness and accuracy.

Fair classifiers without demographics There are various studies to achieve fairness without demographics. For example, Zhao et al. [82] explores the correlations between sensitive attributes and non-sensitive attributes to learn fair and accurate classifiers. Yan et al. [77] investigates the class imbalance problem with a KNN-based pre-processing method. Chai et al. [16] utilizes the soft labels from an overfitting teacher model to train a student model to avoid using demographics. A line of research establishes theoretical connections between features and attributes to avoid using demographic information, employing methods like causal graphs [63], correlation shifts [58], and demographic shifts [30]. In contrast, our approach refrains from making assumptions. Another line of work utilizes distributionally robust optimization (DRO) to reduce fairness disparity without relying on training sensitive attributes [33, 38, 47, 40, 67, 64]. Although these works evaluate the worst-case group performance in the context of fairness, their approaches differ as they do not strive to equalize the loss across groups. Besides, in these studies, accuracy and worst-case accuracy are used as fairness metrics to showcase the efficacy of the proposed algorithms. However, these fairness metrics are restrictive and inconsistent with common definitions such as demographic parity (DP).

Fair classification The fairness-aware learning algorithms, in general, can be categorized into preprocessing, in-processing, and post-processing methods. Pre-processing methods typically reweigh or distort the data examples to mitigate the identified biases [7, 9, 65, 60, 15, 17, 80, 18, 68]. More relevant to this work is the importance reweighting , which assigns weights to training examples [37, 36, 23, 21, 57, 44]. Our algorithm bears similarity to a specific case of importance reweighting, particularly the 0-1 reweighting applied to newly added data. Other parallel studies utilize importance weighting to learn a complex generative model in a weakly supervised setting [23, 21], or to mitigate representation bias in training datasets [44]. Post-processing methods typically enforce fairness on a learned model through calibration [26, 27, 32], but these work might not achieve the best fairness-accuracy tradeoff [75, 55]. In contrast, these post-processing works still require sensitive attributes during the inference phase. Recent work [19] develops a bias score classifier that operates independently of sensitive attributes; however, it is constrained to binary classifications.

3 Preliminaries

Problem setup We consider a standard K -class classification task whose training (test) data distribution is P ( Q ). Let P := { z n } | P | n =1 represent the training dataset following distribution P , where | P | denotes the corresponding sample size. Each example, denoted as z n := ( x n , y n ) , comprises two random variables: the feature vector x and the label y . The model trained on P is evaluated by the test dataset Q := { z ◦ } | Q | n =1 , where ( · ) ◦ denotes that the data follows distribution Q , each example z ◦ n := ( x ◦ , y ◦ , s ◦ ) , and the sensitive group s ◦ n often refers to characteristics such as race, gender, etc. To align the fairness requirements on the test set Q with the model trained on P , a

popular way is to exploit the sensitive attributes s [24, 72] or their proxies [84] in P and use them to formulate Lagrangians during training. However, extending these approaches to the active learning setting would require disclosing more sensitive attributes during sampling and training [4], which contradicts our goal. To avoid disclosing more sensitive attributes, we align the fairness requirements by a small hold-out validation dataset Q v := { z ◦ n } | Q v | n =1 that is independent and identically distributed (IID) with the test set Q . We defer more technical details to Section 4.

Following the general active learning setting [4], we would acquire new examples from a large unlabeled dataset U := { x ′ n } | U | n =1 within a limited labeling budget B ( ≪| U | ) [48, 71, 31]. Denote the solicited example by z ′ n := ( x ′ n , y ′ n ) , where y ′ n is the ground-truth label. Note that the protected sensitive attributes from datasets P and U remain undisclosed during sampling and training. In this paper, we aim to incrementally update a model that was initially trained on P using standard ERM, by incorporating newly solicited data z ′ n , such that the model can improve fairness without worsening model accuracy. Thus, the core challenge is efficiently determining new examples that induce a significantly better Pareto frontier. In the proceeding section 4, we shall delve into how to acquire new data from unlabeled set.

Fairness definition Note that this work focuses solely on active sampling to build a fairer dataset, which is then used to train the model through standard ERM with Cross-Entropy (CE) loss. Without relying on additional assumptions about the model or training/testing dataset, an intuitive and natural approach is to analyze the expected risk. Therefore, we introduce the concept of risk disparity as an intermediate-term for theoretical analysis of fairness.

r

Definition 3.1 (Risk disparity [33, 79, 3]) . Define Q k as the sub-distribution of Q corresponding to group k . Given the optimized model parameters w P trained on set P , risk disparity is defined as: R Q k ( w P ) -R Q ( w P ) , where R Q ( w ) := E z ∼Q [ ℓ ( w , z )] denotes expected risk induced on target distribution Q .

Definition 3.1 naturally quantifies the discrepancy in a trained model's performance between a specific group set Q k and the entire test set Q . That is, a model can be deemed fair if it exhibits consistent performance for a group set Q k as compared to the test dataset Q . In settings such as face or speech recognition, this fairness definition implies the necessity for all demographic groups to receive the same quality service [13]. For completeness, we also include two well-known definitions of fairness:

Definition 3.2 (Demographic Parity ( DP )) . A classifier f adheres to demographic parity concerning the sensitive attribute s if: E [ f ( w , x )] = E [ f ( w , x ) | s ] .

Definition 3.3 (Equalized Odds ( EOd [32]) . A classifier f meets the equalized odds with respect to the sensitive attribute s if: E [ f ( w , x ) | y ] = E [ f ( w , x ) | y, s ] .

Even though there may be a general incompatibility between risk disparity and popular group fairness metrics like DP and EOd, under the criteria of the proposed fairness notion, these definitions could be encouraged [61, 33]. More details and proof can be found in the Appendix B.

Proposition 3.1. (Informal) Under appropriate conditions, the risk disparity can serve as a lower bound for fairness disparities based on common fairness definitions, such as DP and EOd.

Remark 3.1 (Connections to other fairness definitions) . Definition 3.1 targets group-level risk fairness, which has similar granularity to other fairness notions such as accuracy parity [78], device-level parity [43], small accuracy loss for groups [79, 10, 50, 33], and bounded group loss [3].

4 Improving fairness without harm via data influential sampling

In this section, we first introduce how to measure the importance (influence) of each example on accuracy and fairness without using the corresponding sensitive attributes, respectively. Then, we propose an influence-guided sampling algorithm that actively acquires new data based on the influences for further training.

4.1 Finding influential examples

To avoid using training sensitive attributes, our primary idea is to find newly acquired data that assists in creating a 'fairer' dataset, which allows for training a fair and accurate model via standard ERM.

Initially, we explore whether newly acquired data enhances fairness by examining the training process, where the model is typically updated using gradient descent. The change of model parameters by performing one step gradient descent on newly acquired data z ′ is

where η refers to the learning rate and ℓ ( · ) is the training loss function. It should be noted that before we solicit the true labels of samples z ′ , we first use proxy labels. In the following subsection 4.2, we will present a strategy for proxy labels. Training on z ′ affects the model's prediction on validation data z ◦ n regarding both accuracy and fairness. If the updated model w t +1 outperforms the previous one w t evaluated on the validation dataset in terms of fairness and accuracy, this acquired data z ′ helps to reduce the fairness disparity without worsening accuracy.

To separately measure the accuracy and fairness performance of the updated model on the validation set, we introduce two types of loss functions: fairness loss ϕ ( w , z ◦ n ) and accuracy loss ℓ ( w , z ◦ n ) , where validation data z ◦ n = ( x ◦ n , y ◦ n , s ◦ n ) . Note that these loss functions are developed for sampling, not for training. Besides, training loss function ℓ ( · ) can be reused as the accuracy loss function due to the same update target. One can identify training loss ℓ ( · , z ′ ) and accuracy loss ℓ ( · , z ◦ n ) based on the input data used. Without loss of generality, we assume that ℓ ( · ) and ϕ ( · ) are differentiable w.r.t. w . Here, we do not restrict the generality of the fairness loss function; it can be any smoothed version of fairness metrics such as DP or EOd. Following this, we develop the influence of the accuracy and fairness components for finding the samples, respectively.

Influence of accuracy component When model parameters are updated from w t to w t +1 by adding a new example z ′ , the influence of model's accuracy on one validation example z ◦ n is:

For ease of notation, we use Infl acc ( z ′ , z ◦ n ) to represent Infl acc ( z ′ , z ◦ n ; w t , w t +1 ) . By applying firstorder Taylor expansion, we obtain the following closed-form statement:

Lemma 4.1. The accuracy influence of new example z ′ on the validation dataset Q v is:

Intuitively, the more negative Infl acc ( z ′ ) is, the more positive the model accuracy (performance) that example z ′ can provide.

Influence of fairness component When model parameters are updated from w t to w t +1 by adding a new example z ′ , the influence of model's fainess on one validation example z ◦ n is:

For simplicity, we write Infl fair ( z ′ , z ◦ n ; w t , w t +1 ) as Infl fair ( z ′ , z ◦ n ) . Then, similarly, we have:

Lemma 4.2. The fairness influence of new example z ′ on the validation dataset Q v is:

Similar to the accuracy component, the greater the negativity of Infl fair ( z ′ ) is, the greater the positive impact that the example z ′ has on fairness.

Intuitions These two components evaluate the accuracy and fairness impact of each example by comparing the gradient originating from a single data sample with the gradient derived from the entire validation set, respectively. This comparison helps quantify the potential advantage of including this specific example in training. For instance, if the gradient obtained from one example has a similar direction to the gradient from the validation set, it indicates that incorporating this example contributes to enhancing the model's fairness or accuracy.

Algorithm 1 Fair influential sampling (FIS)

1: Input: training set P , unlabeled set U , validation set Q v , new acquired set S t = {} , ∀ t ∈ [ T ] rounds, number of new selected examples in each round r , tolerance ϵ .

2: Warmup: Train a classifier f solely on P by minimizing the empirical risk R ℓ . Obtain model parameters w 1 and validation accuracy (on Q v ) VAL 0 .

3: for t in { 1 , 2 , · · · , T } do

4: Guess proxy label ˆ y ′ for new examples ˆ z ′ using Eq. (5).

5: Compute the influence of accuracy and fairness component using Eq. (2) and Eq. (4):

6: S original = { Infl fair (ˆ z ′ ) | Infl acc (ˆ z ′ ) ≤ 0 , Infl fair (ˆ z ′ ) ≤ 0 , ˆ z ′ ∈ U }

7: while | S t | < r do

8: Find top-( r -| S t | ) annotated examples ˆ z ′ n based on the lowest fairness influence and then inquire about true labels y ′ :

11: U ← U ∩ S t ; S original ← S original ∩ S t

12: end while

13: Continue to train the model f on the set S t via standard ERM. Obtain the updated model parameters w t +1 . If the model's validation accuracy (on Q v ) VAL t does not meet the desired threshold VAL 0 , reject the updated model.

14: end for

15: Output: models { w t | VAL t > VAL 0 -ϵ }

Training sensitive attributes are not disclosed One can easily check that neither the influence of accuracy nor fairness components require the sensitive attributes of any example z ′ , as the example z ′ only appears in the first-order gradient of the accuracy loss ∂ w t ℓ ( w t , z ′ ) . In the fairness component, calculating the ∂ w t ϕ ( w t , z ◦ n ) only relies on validation example z ◦ n 's sensitive attributes. Here, we also validate how accurate the first-order estimation of the influence is in comparison to the real influence [39], and find that the estimated influences for most of the examples are very close to their actual influence values. We refer the readers to Appendix C.1 for more details.

Even without disclosing training sensitive attributes, the correlations between non-demographic features and demographic information may still lead to privacy leakage issues [82]. To address this potential privacy concern, we provide further discussions and theoretical analysis using differential privacy in Appendix F.

4.2 Algorithm: fair influential sampling (FIS)

Following Lemma 4.1 and Lemma 4.2, we can efficiently select those examples with the most negative fairness influence and negative accuracy influence. This sampling method aids in reducing fairness disparities without worsening model accuracy.

Labeling Before presenting our sampling algorithm, it is necessary to address the problem of not accessing the true labels of new solicited examples. Lacking the label information for new examples poses a challenge in determining the corresponding influence on accuracy and fairness, a fact that is substantiated by Lemma 4.1 and Lemma 4.2. Intuitively, one can always recruit human annotators to get the ground-truth labels for those unlabeled examples. However, it is impractical due to the limited labeling budgets. To tackle this problem, another common approach is utilizing a model that has been effectively trained on dataset P to produce proxy labels, which approximate the calculation of influences for examples from a substantial unlabeled dataset U . It's important to note that these proxy labels are exclusively used during the sampling phase. To maintain good model performance, we still need to inquire about the true labels of the selected data examples for subsequent training. Here, we propose to annotate the proxy labels with the model trained on the labeled set P . In particular, we introduce a strategy that employs lowest-influence labels for annotating label ˆ y ′ given x ′ :

Here, we denote ˆ z ′ := ( x ′ , ˆ y ′ ) for the proxy labels.

Proposed algorithm The full procedure is outlined in Algorithm 1. Note that the tolerance ϵ is applied to monitor the performance drop in validation accuracy. In Line 2, we initiate the process by training a classifier f solely on dataset P , that is, performing a warm start. Subsequently, T -round sampling iterations are applied to acquire more examples to dataset P . Following the iterative fashion, FIS guesses labels using Eq. (5) in Line 4. Then, we calculate the scores for proxy examples based on the accuracy and fairness influence using Eq. (2) and Eq. (4) respectively. In Lines 6-12, we would opt for r samples based on the influence scores, and inquire about the true labels of these examples. However, due to the gap between proxy labels ˆ y ′ and true label y , the accuracy and fairness influences of the topr samples based on inquired true labels z ′ may not necessarily satisfy the same conditions ( Infl acc (ˆ z ′ ) ≤ 0 , Infl fair (ˆ z ′ ) ≤ 0 ). Therefore, we use a while loop to iteratively select the top-( r -| S t | ) examples for labeling until we obtain r samples whose fairness influences based on true labels meet the conditions. Subsequently, in Line 11, we update sets U and S original to prevent duplicate sampling. In Line 13, we would continue training using new examples with true inquired labels from set S t . We save the model parameters at each round as checkpoints w t . To avoid potential accuracy drops incurred by excessively large random perturbations, we exclusively choose and offer models for output whose validation accuracy exceeds the initial validation accuracy V AL 0 . Although we propose a specific strategy for guessing labels, our algorithm is flexible and compatible with other labeling methods. A comparative analysis of computational costs is detailed in Appendix C.2.

5 How more data improves fairness without harm?

In general, acquiring new data to supplement the original training dataset would potentially raise the distribution shift problem, affecting both accuracy and fairness. In this section, from a distribution shift perspective view, we first present a generalization error bound (accuracy side, Theorem 5.1) and risk disparity bound (fairness side, Theorem 5.2). The theoretical results jointly provide a high-level key insight that controlling the negative impact of distribution shift on generalization error, which refers to the model accuracy, could allow for improving fairness without harm. This theoretical insight validates the possibility of our sampling approach.

Without loss of generality, we discretize the whole distribution space and suppose that the train/test distributions are both drawn from a series of component distributions { π 1 , · · · , π I } [28]. Then, the empirical risk R P ( w ) calculated over a training set P can be reformulated by splitting samples based on the component distributions:

where p ( P ) ( π = i ) represents the frequencies of examples in P drawn from component distribution π i . Then, we can define the measure of probability distance between two sets or distributions as dist ( A , B ) := ∑ I i =1 | p ( A ) ( π = i ) -p ( B ) ( π = i ) | . To reflect the implicit unfairness in the models, we introduce two basic assumptions in convergence analysis [43].

Assumption 5.1 ( L -Lipschitz Continuous) . There exists a constant L > 0 , for any v , w ∈ R d , R P ( v ) ≤ R P ( w ) + ⟨∇R P ( w ) , v -w ⟩ + L 2 ∥ v -w ∥ 2 2 .

Assumption 5.2 (Bounded Gradient on Random Sample) . The stochastic gradients on any sample z are uniformly bounded, i.e. , E [ ||∇R P ( w t , z ) || 2 ] ≤ G 2 , and training epoch t ∈ [1 , · · · , T ] .

Analogous to Assumption 5.2, we further make a mild assumption to bound the loss over the component distributions π i according to the corresponding model, that is, E z ∼ π i [ ℓ ( w P , z )] ≤ G P , ∀ i ∈ I , where G P is a bounding constant. For completeness, we first analyze the upper bound of generalization error, specifically from the standpoint of distribution shifts. Omitted proof can be found in Appendix D.

Theorem 5.1 (Generalization error bound) . Let dist ( P , Q ) , G P be defined therein. With probability at least 1 -δ with δ ∈ (0 , 1) , the generalization error bound of the model trained on dataset P is

Note that the generalization error bound is predominantly impacted by the shift in distribution, especially when we consider an overfitting model, i.e., the empirical risk R P ( w P ) → 0 .

Theorem 5.2 (Upper bound of risk disparity) . Suppose R Q ( · ) follows Assumption 5.1. Let dist ( P , Q ) , G P , dist ( P k , Q k ) and dist ( P k , P ) be defined therein. The initial learning rate η 0 satisfies η 2 0 < 1 √ 2 TL , where T is the number of training epochs. Given model w P and w k trained exclusively on group k 's data P k , with probability at least 1 -δ with δ ∈ (0 , 1) , then the upper bound of risk disparity is

where Υ = √ log(4 /δ ) 2 | P | + √ log(4 /δ ) 2 | P k | + ϖ + ϖ k . Note that E z ∼ π i [ ℓ ( w k , z )] ≤ G k , ϖ = R P ( w P ) -R ∗ Q ( w Q ) and ϖ k = R P k ( w k ) - R ∗ Q k ( w Q k ) . ϖ and ϖ k can be regarded as constants because R P ( w P ) and R P k ( w k ) correspond to the empirical risks, R ∗ Q ( w Q ) and R ∗ Q k ( w Q k ) represent the ideal minimal empirical risk of model w Q trained on distribution Q and Q k , respectively.

Interpretations of Theorem 5.2 Eq. (6) illustrates several aspects that induce unfairness. (1) Group biased data . For group-level fairness, the more balanced the data is, the smaller the risk disparity would be; (2) distribution shift . For source/target distribution, the closer the distributions are, the smaller the performance gap would be; (3) Data size . For training data size, a larger data size (potentially eliminating data bias across groups) would lead to a smaller performance gap.

Main observation Theorem 5.1 underscores how the generalization error is impacted by distribution shifts. Theorem 5.2 implies that the risk disparity is essentially influenced by the distribution shift and the inherent group gap term. In practice, approaches that mitigate the group gap, such as imposing fairness regularizers, acquiring new data, or reweight the training data samples [37], will inevitably incur additional distribution shifts between the training and test data. The incurred distribution shift further leads to a performance drop due to the generalization error in Theorem 5.1. Nonetheless, one theoretical insight is that if one can control the negative impacts of potential distribution shifts through generalization error while implementing fairness-enhancing strategies, it is possible to achieve the goal of improving fairness without causing harm. This high-level insight supports the effectiveness of our proposed sampling approach, in which we acquire new data to reduce the group gap through fairness components while preventing the potential adverse impacts of distribution shifts using the accuracy influence component.

6 Empirical results

In this section, we empirically demonstrate the disparate impact across groups and present the effectiveness of the proposed Fair Influential Sampling method to mitigate the disparity.

6.1 Experimental setup

We evaluate the performance of our algorithm on three real-world datasets across three different modalities: CelebA [49], UCI Adult [8] and Compas [6]. We implement the fairness loss ϕ ( · ) based on three common group fairness metrics: difference of demographic parity ( DP ), difference of equality of opportunity ( EOp ), and difference of equal odds ( EOd ). We compare our method with five baselines: 1) Base (ERM): directly train the model on the training dataset P ; 2) Random: train the model on dataset P and randomly sampled data from U with inquired true labels; 3) BALD [12]: active sampling according to the mutual information; 4) ISAL [48]: selects unlabeled examples based on the calculated influence in an active learning setting. We apply model predictions as pseudo-labels; 5) Just Train Twice (JTT) [47]: reweighting those misclassified examples for re-training. Here, we examine a weight of 20 for misclassified examples, marked as JTT-20. Recall that we present the average result of the classifier w t outputs from Algorithm 1. The general term ' fairness violation ' is utilized to quantify the absolute differences based on fairness metrics, such as DP and EOd. More details on datasets and hyper-parameters are provided in Appendix E.1.

6.2 Main results

Note that all the experimental results presented subsequently are from three independent trials, each conducted with distinct random seeds. We present the primary results as tuples in the form

Base(ERM)

Random

BALD

ISAL

JTT-20

■

FIS

CelebA-Smiling (DP)

CelebA-Smiling (EOp)

CelebA-Attractive(DP)

CelebA-Attractive(EOp)

fairness violation

0.20

0.08

0.4

0.30 -

0.15

0.25

0.06 -

0.3

0.20

0.10

0.04

0.2

0.15

0.05

0.1

0.10

0.02

0.05

0.70

0.75

0.80

0.85

0.7

0.8

0.9

0.60

0.65

0.70

0.60

0.65

0.70

CelebA-Young(DP)

CelebA-Young(EOp)

CelebA-Big_Nose(DP)

CelebA-Big_Nose(EOp)

fairness violation

0.25

1

0.125

0.25

FE0

0.20

0.100

0.20

0.2

0.15

0.075

0.15

0.10

0.050

0.10

0.1

0.05

0.025 -

0.05

0.0

0.72

0.74

0.76

0.78

0.80

0.6

0.7

0.8

0.72

0.74

0.76

0.78

0.80

0.72

0.74

0.76

0.78

0.80

Adult-Income(DP)

Adult-Income (EOp)

Compas-Recidivism (DP)

Compas-Recidivism (EOp)

0.200

fairness violation

0.30

0.175

0.35

0E'0

0.25

0.25

0.150

0.30

0.125

0.20

0.20

0.100

0.25

0.15

0.075

0.10-

0.65

0.70

0.75

0.80

0.6

0.7

0.8

0.64

0.66

0.68

0.70

0.60

0.65

0.70

testaccuracy

test accuracy

testaccuracy

test accuracy

( test_accuracy , fairness_violation ) to facilitate comparison of the fairness-accuracy tradeoff. Due to space limits, we provide a full version of the experimental results (tables) in Appendix E.2.

Results on image datasets Initially, we train a vision transformer using a patch size of (8 , 8) on the CelebA face attribute dataset [49]. We select four binary classification targets, including Smiling , Attractive , Young , and Big Nose . The sensitive attribute is gender . 2% of the labeled data is allocated for training, while the remaining 98% is reserved for sampling purposes. Then, the test dataset is split into two independent portions: a new test set and a validation set, with 10% of the test data randomly designated as the hold-out validation set. For ease of computation, only the last two layers of the model are used to calculate the influence of accuracy and fairness components. For Figure 2, one main observation is that FIS outperforms baselines with a significant margin on three fairness metrics while maintaining the same accuracy level. This improvement, as indicated in Theorem 5.2, can be attributed to FIS assigning priority to new examples based on the fairness influence, then avoiding accuracy reduction via their accuracy influence.

Results on tabular datasets Next, we work with multi-layer perceptron (MLP) with two layers trained on the Adult [8] and Compas dataset [6], respectively. We select age as the sensitive attribute for the Adult dataset and race for the Compas dataset. For two datasets, we resample the data to balance the class and group membership [17]. The whole dataset is split into training and test sets at a 4:1 ratio. Then, we randomly re-select 20% of the training set for initial training and the remaining 80% for sampling. Also, 20% examples of the test set are selected to form a validation set. The MLP model is a two-layer ReLU network with a hidden size of 64. We utilize the whole model parameters to compute the influence of accuracy and fairness for examples. Figure 2 summarizes the main results of the Adult and Compas datasets. On the Adult dataset, we observe that our sampling method achieves the lowest violation for equality of opportunity and has a comparable performance for the DP metric. Besides, our algorithm achieves a much better accuracy-fairness trade-off than

other baselines on the Compas dataset. JTT-20 achieves a lower fairness violation with the price of a significant accuracy drop compared to other baselines.

6.3 Ablation study

What is the impact of label budgets? Here, we examine how the varying label budgets r affect the trade-off between accuracy and fairness. For ease of comparison, we adhere to a consistent label budget per round to illustrate their respective impacts. As shown in Figure 3, our method consistently preserves a lower fairness violation than the BALD and ISAL baselines with a similar test accuracy. While we observe that the JTT-20 algorithm can achieve a near-zero fairness violation under a limited budget on the CelebA dataset, we argue that the model accuracy is rather uninformative (about 50%). More empirical results can be found in Appendix E.3.

How does the validation set size affect the performance? We further explore the impact of adjusting the validation set size on our algorithm's performance. We present the test accuracy and fairness violations across different validation set sizes on the CelebA dataset. Note that the default validation set size is set to 1% of the whole dataset size. In particular, the minimum scale of the validation set size is set to 1 / 5 × (nearly 400 CelebA images). The results in Figure 3 indicate that our algorithm still retains the test accuracy and fairness violation when we vary the validation set size. Additional results conducted on Adult and Compas datasets are provided in Appendix E.4.

0

10

20

30

40

50

60

70

label budget

0.0

0.2

0.4

0.6

0.8

1.0

test accuracy

CelebA

BALD

JTT-20

ISAL

FIS

0

10

20

30

40

50

60

70

label budget

0.0

0.1

0.2

0.3

DP gap

CelebA

7 Conclusions and limitations

In this work, we are interested in facilitating ML models that mitigate group fairness disparity without harming model accuracy. To achieve this, different from current active sampling methods, we propose a tractable fair influential sampling method FIS, which avoids the need for training group annotations during the sampling or training phase, thereby preventing the potential exposure of sensitive information. In particular, this algorithm acquires data samples from a large dataset for training based on the influence of fairness and accuracy evaluated on the auxiliary validation dataset. Empirical experiments on real-world data validate the efficacy of our proposed method.

Nonetheless, we recognize that our method has limitations. Although the proposed sampling algorithm does not require sensitive attribute information from the massive data, it relies on a clean and informative validation set that contains the sensitive attributes of data examples. We consider this as a reasonable requirement in practice, given the relatively modest size of the validation set. Besides, one potential concern is that the sampling strategy may become inefficient when the collected validation set is noisy. However, this practical issue can be heavily alleviated by using loss correction methods [54, 84] or noise-tolerant fairness loss functions to rectify the error terms [53, 73, 29]. In future work, we aim to address this limitation by developing more robust sampling strategies that can perform effectively even with noisy validation sets.

Acknowledgment

This work is partially supported by the National Science Foundation (NSF) under grants IIS-2143895, IIS-2040800, IIS-2416896, and CCF-2023495. Additionally, Pang and Qian were partially supported by NSF Grants 2322919, 2420632, 2426031, and 2114113.

Broader Impact

This paper presents work whose goal is to advance the field of fairness in machine learning. There are many potential societal consequences of our work. While the proposed algorithm does intend to infer sensitive attributes of data examples that may be protected by privacy regulations, it does not necessitate direct access to such sensitive information. On the other hand, our work can serve as an effective approach leading to mitigating the disparity with a limited annotation budget. We have thoroughly examined the potential ethical implications of our work and, based on our assessment, do not identify any issues that we deem necessary to emphasize here specifically.

References

Pareto effiency and production possibility frontier. https://en.wikipedia.org/wiki/ Production%E2%80%93possibility_frontier .

Alekh Agarwal, Alina Beygelzimer, Miroslav Dudík, John Langford, and Hanna Wallach. A reductions approach to fair classification. In Proceedings of the 35th International Conference on Machine Learning (ICML '18) , 2018.

Alekh Agarwal, Miroslav Dudík, and Zhiwei Steven Wu. Fair regression: Quantitative definitions and reduction-based algorithms. In International Conference on Machine Learning , pages 120-129. PMLR, 2019.

Hadis Anahideh, Abolfazl Asudeh, and Saravanan Thirumuruganathan. Fair active learning. Expert Systems with Applications , 199:116981, 2022.

McKane Andrus, Elena Spitzer, Jeffrey Brown, and Alice Xiang. What we can't measure, we can't understand: Challenges to demographic data procurement in the pursuit of fairness. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency , pages 249-260, 2021.

Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. Machine bias. 2016.

Abolfazl Asudeh, Zhongjun Jin, and HV Jagadish. Assessing and remedying coverage for a given dataset. In 2019 IEEE 35th International Conference on Data Engineering (ICDE) , pages 554-565. IEEE, 2019.

Arthur Asuncion and David Newman. Uci machine learning repository, 2007.

Fabio Azzalini, Chiara Criscuolo, and Letizia Tanca. Fair-db: Functional dependencies to discover data bias. In EDBT/ICDT Workshops , 2021.

Ananth Balashankar, Alyssa Lees, Chris Welty, and Lakshminarayanan Subramanian. What is fair? exploring pareto-efficiency for fairness constrained classifiers. arXiv preprint arXiv:1910.14120 , 2019.

Solon Barocas, Moritz Hardt, and Arvind Narayanan. Fairness and machine learning: Limitations and opportunities . MIT Press, 2023.

Frédéric Branchaud-Charron, Parmida Atighehchian, Pau Rodríguez, Grace Abuhamad, and Alexandre Lacoste. Can active learning preemptively mitigate fairness issues? arXiv preprint arXiv:2104.06879 , 2021.

Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency , pages 77-91. PMLR, 2018.

Réal André Carbonneau, Kevin Laframboise, and Rustam M. Vahidov. Application of machine learning techniques for supply chain demand forecasting. Eur. J. Oper. Res. , 184:1140-1154, 2008.

L Elisa Celis, Vijay Keswani, and Nisheeth Vishnoi. Data preprocessing to mitigate bias: A maximum entropy based approach. In International conference on machine learning , pages 1349-1359. PMLR, 2020.

Junyi Chai, Taeuk Jang, and Xiaoqian Wang. Fairness without demographics through knowledge distillation. Advances in Neural Information Processing Systems , 35:19152-19164, 2022.

Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research , 16:321357, 2002.

Irene Chen, Fredrik D Johansson, and David Sontag. Why is my classifier discriminatory? Advances in neural information processing systems , 31, 2018.

Wenlong Chen, Yegor Klochkov, and Yang Liu. Post-hoc bias scoring is optimal for fair classification. arXiv preprint arXiv:2310.05725 , 2023.

Yatong Chen, Reilly Raab, Jialu Wang, and Yang Liu. Fairness transferability subject to bounded distribution shift. Advances in Neural Information Processing Systems , 35:11266-11278, 2022.

Kristy Choi, Aditya Grover, Trisha Singh, Rui Shu, and Stefano Ermon. Fair generative modeling via weak supervision. In International Conference on Machine Learning , pages 1887-1898. PMLR, 2020.

Sanjoy Dasgupta and Daniel Hsu. Hierarchical sampling for active learning. In Proceedings of the 25th international conference on Machine learning , pages 208-215, 2008.

Maurice Diesendruck, Ethan R Elenberg, Rajat Sen, Guy W Cole, Sanjay Shakkottai, and Sinead A Williamson. Importance weighted generative networks. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2019, Würzburg, Germany, September 16-20, 2019, Proceedings, Part II , pages 249-265. Springer, 2020.

Michele Donini, Luca Oneto, Shai Ben-David, John S Shawe-Taylor, and Massimiliano Pontil. Empirical risk minimization under fairness constraints. Advances in neural information processing systems , 31, 2018.

Sanghamitra Dutta, Dennis Wei, Hazar Yueksel, Pin-Yu Chen, Sijia Liu, and Kush Varshney. Is there a trade-off between fairness and accuracy? a perspective using mismatched hypothesis testing. In International conference on machine learning , pages 2803-2813. PMLR, 2020.

Michael Feldman. Computational fairness: Preventing machine-learned discrimination . PhD thesis, 2015.

Michael Feldman, Sorelle A Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubramanian. Certifying and removing disparate impact. In proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining , pages 259-268, 2015.

Vitaly Feldman. Does learning require memorization? a short tale about a long tail. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing , pages 954-959, 2020.

Lei Feng, Senlin Shu, Zhuoyi Lin, Fengmao Lv, Li Li, and Bo An. Can cross entropy loss be robust to label noise? In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence , pages 2206-2212, 2021.

Stephen Giguere, Blossom Metevier, Yuriy Brun, Bruno Castro da Silva, Philip S Thomas, and Scott Niekum. Fairness guarantees under demographic shift. In Proceedings of the 10th International Conference on Learning Representations (ICLR) , 2022.

Zayd Hammoudeh and Daniel Lowd. Training data influence analysis and estimation: A survey. arXiv preprint arXiv:2212.04612 , 2022.

Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. Advances in neural information processing systems , 29, 2016.

Tatsunori Hashimoto, Megha Srivastava, Hongseok Namkoong, and Percy Liang. Fairness without demographics in repeated loss minimization. In International Conference on Machine Learning , pages 1929-1938. PMLR, 2018.

Steven CH Hoi, Rong Jin, and Michael R Lyu. Large-scale text categorization by batch mode active learning. In Proceedings of the 15th international conference on World Wide Web , pages 633-642, 2006.

Kenneth Holstein, Jennifer Wortman Vaughan, Hal Daumé III, Miro Dudik, and Hanna Wallach. Improving fairness in machine learning systems: What do industry practitioners need? In Proceedings of the 2019 CHI conference on human factors in computing systems , pages 1-16, 2019.

Heinrich Jiang and Ofir Nachum. Identifying and correcting label bias in machine learning. In International Conference on Artificial Intelligence and Statistics , pages 702-712. PMLR, 2020.

Faisal Kamiran and Toon Calders. Data preprocessing techniques for classification without discrimination. Knowledge and information systems , 33(1):1-33, 2012.

Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re-training is sufficient for robustness to spurious correlations. arXiv preprint arXiv:2204.02937 , 2022.

Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning , pages 1885-1894. PMLR, 2017.

Preethi Lahoti, Alex Beutel, Jilin Chen, Kang Lee, Flavien Prost, Nithum Thain, Xuezhi Wang, and Ed Chi. Fairness without demographics through adversarially reweighted learning. Advances in neural information processing systems , 33:728-740, 2020.

David D Lewis and Jason Catlett. Heterogeneous uncertainty sampling for supervised learning. In Machine learning proceedings 1994 , pages 148-156. Elsevier, 1994.

Peizhao Li and Hongfu Liu. Achieving fairness at no utility cost via data reweighing with influence. In International Conference on Machine Learning , pages 12917-12930. PMLR, 2022.

Tian Li, Maziar Sanjabi, Ahmad Beirami, and Virginia Smith. Fair resource allocation in federated learning. In International Conference on Learning Representations , 2019.

Yi Li and Nuno Vasconcelos. Repair: Removing representation bias by dataset resampling. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9572-9581, 2019.

Michael Lindenbaum, Shaul Markovitch, and Dmitry Rusakov. Selective sampling for nearest neighbor classifiers. Machine learning , 54:125-152, 2004.

Richard G Lipsey. An introduction to positive economics-4. 1975.

Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In International Conference on Machine Learning , pages 6781-6792. PMLR, 2021.

Zhuoming Liu, Hao Ding, Huaping Zhong, Weijia Li, Jifeng Dai, and Conghui He. Influence selection for active learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9274-9283, 2021.

Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision , pages 3730-3738, 2015.

Natalia Martinez, Martin Bertran, and Guillermo Sapiro. Fairness with minimal harm: A pareto-optimal approach for healthcare. arXiv preprint arXiv:1911.06935 , 2019.

Natalia Martinez, Martin Bertran, and Guillermo Sapiro. Minimax pareto fairness: A multi objective perspective. In International conference on machine learning , pages 6755-6764. PMLR, 2020.

Aditya Krishna Menon and Robert C Williamson. The cost of fairness in binary classification. In Conference on Fairness, accountability and transparency , pages 107-118. PMLR, 2018.

Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. In Advances in neural information processing systems , pages 1196-1204, 2013.

Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 1944-1952, 2017.

Geoff Pleiss, Manish Raghavan, Felix Wu, Jon Kleinberg, and Kilian Q Weinberger. On fairness and calibration. In Advances in Neural Information Processing Systems , pages 5680-5689, 2017.

Flavien Prost, Hai Qian, Qiuwen Chen, Ed H Chi, Jilin Chen, and Alex Beutel. Toward a better trade-off between performance and fairness with kernel-based distribution matching. arXiv preprint arXiv:1910.11779 , 2019.

Maan Qraitem, Kate Saenko, and Bryan A Plummer. Bias mimicking: A simple sampling approach for bias mitigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 20311-20320, 2023.

Yuji Roh, Kangwook Lee, Steven Euijong Whang, and Changho Suh. Improving fair training under correlation shifts. arXiv preprint arXiv:2302.02323 , 2023.

Nicholas Roy and Andrew McCallum. Toward optimal active learning through monte carlo estimation of error reduction. ICML, Williamstown , 2:441-448, 2001.

Shubham Sharma, Yunfeng Zhang, Jesús M Ríos Aliaga, Djallel Bouneffouf, Vinod Muthusamy, and Kush R Varshney. Data augmentation for discrimination prevention and bias disambiguation. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society , pages 358-364, 2020.

Changjian Shui, Gezheng Xu, Qi Chen, Jiaqi Li, Charles X Ling, Tal Arbel, Boyu Wang, and Christian Gagné. On learning fairness and accuracy on multiple subgroups. Advances in Neural Information Processing Systems , 35:34121-34135, 2022.

Naeem Siddiqi. Credit risk scorecards: developing and implementing intelligent credit scoring , volume 3. John Wiley & Sons, 2012.

Harvineet Singh, Rina Singh, Vishwali Mhasawade, and Rumi Chunara. Fairness violations and mitigation under covariate shift. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency , pages 3-13, 2021.

Nimit Sohoni, Jared Dunnmon, Geoffrey Angus, Albert Gu, and Christopher Ré. No subclass left behind: Fine-grained robustness in coarse-grained classification problems. Advances in Neural Information Processing Systems , 33:19339-19352, 2020.

Ki Hyun Tae and Steven Euijong Whang. Slice tuner: A selective data acquisition framework for accurate and fair machine learning models. In Proceedings of the 2021 International Conference on Management of Data , pages 1771-1783, 2021.

Michael Veale and Reuben Binns. Fairer machine learning in the real world: Mitigating discrimination without collecting sensitive data. Big Data & Society , 4(2):2053951717743530, 2017.

Akshaj Kumar Veldanda, Ivan Brugere, Sanghamitra Dutta, Alan Mishler, and Siddharth Garg. Hyper-parameter tuning for fair classification without sensitive attribute access. arXiv preprint arXiv:2302.01385 , 2023.

Jialu Wang, Yang Liu, and Caleb Levy. Fair classification with group-dependent label noise. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency , FAccT '21, page 526-536, New York, NY, USA, 2021. Association for Computing Machinery.

Jialu Wang, Xin Eric Wang, and Yang Liu. Understanding instance-level impact of fairness constraints. In International Conference on Machine Learning , pages 23114-23130. PMLR, 2022.

Ran Wang, Sam Kwong, and Degang Chen. Inconsistency-based active learning for support vector machines. Pattern Recognition , 45(10):3751-3767, 2012.

Tianyang Wang, Xingjian Li, Pengkun Yang, Guosheng Hu, Xiangrui Zeng, Siyu Huang, Cheng-Zhong Xu, and Min Xu. Boosting active learning via improving test performance. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 8566-8574, 2022.

Yuyan Wang, Xuezhi Wang, Alex Beutel, Flavien Prost, Jilin Chen, and Ed H Chi. Understanding and improving fairness-accuracy trade-offs in multi-task learning. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , pages 1748-1757, 2021.

Hongxin Wei, Lei Feng, Xiangyu Chen, and Bo An. Combating noisy labels by agreement: A joint training method with co-regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13726-13735, 2020.

Susan Wei and Marc Niethammer. The fairness-accuracy pareto front. Statistical Analysis and Data Mining: The ASA Data Science Journal , 15(3):287-302, 2022.

Blake Woodworth, Suriya Gunasekar, Mesrob I Ohannessian, and Nathan Srebro. Learning non-discriminatory predictors. In Conference on Learning Theory , pages 1920-1953. PMLR, 2017.

Yichen Xie, Han Lu, Junchi Yan, Xiaokang Yang, Masayoshi Tomizuka, and Wei Zhan. Active finetuning: Exploiting annotation budget in the pretraining-finetuning paradigm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 23715-23724, 2023.

Shen Yan, Hsien-te Kao, and Emilio Ferrara. Fair class balancing: Enhancing model fairness without observing sensitive attributes. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management , pages 1715-1724, 2020.

Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, and Krishna P Gummadi. Fairness constraints: Mechanisms for fair classification. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS) , 2017.

Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez-Rodriguez, and Krishna P Gummadi. Fairness constraints: A flexible approach for fair classification. The Journal of Machine Learning Research , 20(1):2737-2778, 2019.

Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. Learning fair representations. In International conference on machine learning , pages 325-333. PMLR, 2013.

Han Zhao and Geoffrey J Gordon. Inherent tradeoffs in learning fair representations. The Journal of Machine Learning Research , 23(1):2527-2552, 2022.

Tianxiang Zhao, Enyan Dai, Kai Shu, and Suhang Wang. Towards fair classifiers without sensitive attributes: Exploring biases in related features. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining , pages 1433-1442, 2022.

Zhaowei Zhu, Tianyi Luo, and Yang Liu. The rich get richer: Disparate impact of semisupervised learning. arXiv preprint arXiv:2110.06282 , 2021.

Zhaowei Zhu, Yuanshun Yao, Jiankai Sun, Hang Li, and Yang Liu. Weak proxies are sufficient and preferable for fairness with missing sensitive attributes. In International Conference on Machine Learning , pages 43258-43288. PMLR, 2023.

Appendix

The Appendix is organized as follows.

Section A provides more details of the related work.

Section B explores the relationship between our proposed fairness notion risk disparity and common fairness metrics, such as DP and EOd. In particular, we provide the full proof for Proposition 3.1.

Section C provides a detailed analysis of the FIS algorithm including 1) evaluating first-order influence estimations against real influence, 2) a comparative analysis of computational costs, and 3) the exploration of the labeling strategies.

Section D presents the full proofs for the Lemmas and Theorems shown in Section 4 and Section 5.

Section E presents detailed descriptions of all datasets, corresponding parameter settings, and full version of the experimental results. In particular, to demonstrate FIS's advantage at the same levels of information, we introduce a new baseline called Random+Val .

Section F discusses the privacy concern potentially caused by the correlations between nondemographic and demographic features.

A More details of related work

Active learning The core idea of active learning is to rank unlabeled instances by developing specific significant measures, including uncertainty [41, 45], representativeness [22], inconsistency [70], variance [34], and error [59]. Each of these measures has its criterion to determine the importance of instances for enhancing classifier performance. For example, uncertainty considers the most important unlabeled instance to be the nearest one to the current classification boundary. A related line of work [48, 71, 31, 76] concentrates on ranking unlabeled instances based on the influence function. Compared to these studies with a focus on prediction accuracy, our work poses a distinct challenge taking into account fairness violations. We note that adopting a particular sampling strategy can lead to distribution shifts between the training and testing data. What's worse, even though fairness is satisfied within the training dataset, the model may still exhibit unfair treatments on the test dataset due to the distribution shift. Therefore, it becomes imperative for the sampling approach to also account for its potential impacts on fairness.

Pareto optimality In the field of fairness in machine learning, Pareto optimality indicates the theoretical frontier of fairness accuracy tradeoff, meaning that fairness can not be improved without worsening model accuracy. Existing methods primarily focus on seeking the Pareto frontier of fairness accuracy tradeoff for the neural network classifier, instead of reaching a better one. For example, [10] first analyzes the Pareto optimality for classifiers within the context of fairness constraints. [72] explores the multi-dimensional Pareto frontiers of the fairness-accuracy tradeoff in the multi-task setting. [51, 50] target to obtain a Pareto efficient classifier to reduce worst-case group risks by formulating group fairness as a multiple-objective optimization problem, where each group risk is an objective function.

Fair classification The fairness-aware learning algorithms, in general, can be categorized into preprocessing, in-processing, and post-processing methods. Pre-processing methods typically reweigh or distort the data examples to mitigate the identified biases [7, 9, 65, 60, 15, 17, 80, 18]. More relevant to us is the importance reweighting , which assigns different weights to different training examples to ensure fairness across groups [37, 36, 23, 21, 57, 44]. Our proposed algorithm bears similarity to a specific case of importance reweighting, particularly the 0-1 reweighting applied to newly added data. The main advantage of our work, however, lies in its ability to operate without needing access to the sensitive attributes of either the new or training data. Other parallel studies utilize importance weighting to learn a complex fair generative model in a weakly supervised setting [23, 21], or to mitigate representation bias in training datasets [44]. Post-processing methods typically enforce fairness on a learned model through calibration [26, 27, 32]. Although this approach is likely to decrease the disparity of the classifier, by decoupling the training from the fairness enforcement, this procedure may not lead to the best trade-off between fairness and accuracy [75, 55]. In contrast, our work can achieve a better trade-off between fairness and accuracy, because we reduce the fairness disparity by mitigating the adverse effects of distribution shifts on generalization error. Additionally, these post-processing techniques necessitate access to the sensitive attribute during the inference phase, which is often not available in many real-world scenarios.

B Understanding risk disparity through common fairness metrics

Here, we provide a brief proof to illustrate Proposition 3.1, that is, the relationship between risk parity and common fairness metrics, such as DP and EOd. For completeness, here we provide a more detailed version of Proposition 3.1, as well as its proof.

Proposition 3.1. Consider a binary classification scenario involving two demographic groups S ∈ { k, k ′ } . When two groups are balanced, i.e., P ( Y = y | S = k ) P ( Y = y | S = k ′ ) = 1 , where ˆ Y denotes the predicted label, risk disparity can serve as a lower bound for fairness disparities based on EOd. Similarly, if the group sufficiency ratio can be calibrated by 1, i.e., P ( Y = y | S = k, ˆ Y = y ) = P ( Y = y | S = k ′ , ˆ Y = y ) = R where R is calibration score, risk disparity can be formulated as a lower bound for DP-based fairness disparity.

Proof. Consider a scenario involving two demographic groups S ∈ { k, k ′ } , alongside a 0-1 loss function.

Fairness metric EOd For EOd, the risk disparity can be reformulated as

̸

̸

where we define ω EOd ( y ) := P ( Y = y | S = k ) P ( Y = y | S = k ′ ) , serving as the bias weight. Here, we make a mild assumption that the two demographic groups are balanced, i.e., ω EOd ( y ) = 1 . Then, we can see that the last item measures the fairness disparity based on EOd for binary classification problems. Thus, we can claim that reducing risk disparity can promote the fairness metric EOd.

̸

̸

Fairness metric DP Similarly, for DP, we formulate the risk disparity as

̸

̸

̸

̸

where we define ω DP ( y ) := P ( Y = y | S = k, ˆ Y = y ) P ( Y = y | S = k ′ , ˆ Y = y ) . In fact, ω DP ( y ) measures the group sufficiency ratio [61]. Note that the group sufficiency is closely related to the idea of calibration [11]. Thus, we make a mild assumption that the group sufficiency ratio ω DP ( y ) can be calibrated by 1, i.e., P ( Y = y | S = k, ˆ Y = y ) = P ( Y = y | S = k ′ , ˆ Y = y ) = R , where R is a calibration score. When two demographic groups are balanced, i.e., ω DP ( y ) = 1 , we can see that the last inequality indicates the DP-based fairness disparity for binary classification problems. Therefore, we can also claim that the fairness metric DP can also be encouraged when reducing the risk disparity.

C Detailed analysis of the fair influential sampling algorithm

C.1 Evaluating first-order influence estimations against real influence

Recall that the influence score derived in this paper primarily utilizes a first-order approach. Here, we will demonstrate how accurate the first-order estimation of the influence is in comparison to the real influence.

C.2 Comparative analysis of computational costs

Recall that the proposed algorithm FIS needs to pre-calculate the accuracy loss and fairness loss for evaluating the performance of a certain example. However, the extra computation cost is comparable to the cost of traditional model training. Note that the main extra computation cost in FIS (Algorithm

0.0

0.2

0.4

0.6

0.8

1.0

actual influence per sample

0.0

0.2

0.4

0.6

0.8

1.0

first order estimation per sample

1) mainly results from model gradients. Let p denote the number of model parameters, then the cost for computing the gradients is O ( p ) per sample. Specifically, in each round that involves sampling, we need to calculate three parts of gradients: the gradients of | U | unlabeled instances, the average gradient of | Q v | validation instances w.r.t. accuracy loss, and the gradient of | Q v | validation instances w.r.t. fairness loss. Note that in general, | Q v | ≪| U | . In practical implementation, to speed up the calculation of gradients over | U | instances, we randomly sample 0.2-0.5% of the unlabeled dataset in each sampling batch. Additionally, we can increase the number of newly selected examples for each round ( r ) to save computation costs. In our experiments, we usually have 10-20 sampling rounds. For instance, running one experiment for the CelebA dataset on a single GPU roughly requires about 4 hours.

C.3 Exploration of the labeling strategies

Note that we provide a strategy that employs lowest-influence labels for annotating labels. This section will explore an alternative strategy that employs model predictions for the purpose of labeling. For completeness, we outline the two proposed labeling strategies as follows.

Strategy I Use low-influence labels. That is, ˆ y = arg min k ∈ [ K ] | Infl acc ( x ′ , k ) | , which corresponds to using the most uncertain point.

Strategy II Rely on model prediction. That is, ˆ y = arg max k ∈ [ K ] f ( x ; w )[ k ] , where f ( x ; w )[ y indicates the model's prediction probability on label y .

]

Proof. Based on the definition of the influence of the prediction component, as delineated in Eq. (2), it becomes evident that the most uncertain points are obtained when the proxy labels closely align with the true labels. Consequently, the model predictions used in Strategy II also approximate the true labels to minimize the cross-entropy loss. Thus, in a certain sense, Strategy I and Strategy II can be considered equivalent.

D Omitted proofs

In this section, we present complete proofs for the lemmas and theorems in Section 4 and 5, respectively.

D.1 Proof of Lemma 4.1

Lemma 4.1. The influence of predictions on the validation dataset Q v can be denoted by

Proof. Taking the first-order Taylor expansion, we will have

where we take this expansion with respect to f ( w , x ◦ n ) . Similarly, we have

where the last equality holds due to Eq. (1). Therefore,

Then the accuracy influence on the validation dataset V can be denoted by

D.2 Proof of Lemma 4.2

Lemma 4.2. The influence of fairness on the validation dataset Q v can be denoted by

Proof. By first-order approximation, we have

Recall by first-order approximation, we have

Therefore,

Note the loss function in the above equation should be ℓ since the model is updated with ℓ -loss. Therefore,

D.3 Proof of Theorem 5.1

Theorem 5.1. (Generalization error bound). Let dist ( P , Q ) , G P be defined therein. With probability at least 1 -δ with δ ∈ (0 , 1) , the generalization error bound of the model trained on dataset P is

Note that the generalization error bound is predominantly influenced by the shift in distribution when we think of an overfitting model, i.e., the empirical risk R P ( w P ) → 0 . The detailed proof is presented as follows.

Proof. The generalization error bound is

For the first term (distribution shift), we have

where we define dist ( P , Q ) = ∑ I i =1 | p ( P ) ( π = i ) -p ( Q ) ( π = i ) | and E z ∼ π i [ ℓ ( w P , z )] ≤ G P , ∀ i ∈ I because of Assumption 5.2. To avoid misunderstanding, we use a subscript P of the constant G to clarify the corresponding model w P . Then, for the second term (Hoeffding inequality), with probability at least 1 -δ , we have |R P ( w P ) -R P ( w P ) | ≤ √ log(4 /δ ) 2 | P | .

D.4 Proof of Theorem 5.2

Theorem 5.2. (Upper bound of fairness disparity). Suppose R Q ( · ) follows Assumption 5.1. Let dist ( P , Q ) , G P , dist ( P k , Q k ) and dist ( P k , P ) be defined therein. The initial learning rate η 2 0 < 1 √ 2 TL , where T denotes the number of training epochs. Given model w P and w k trained exclusively on group k 's data P k , with probability at least 1 -δ with δ ∈ (0 , 1) , then the upper bound of the fairness disparity is

where Υ = √ log(4 /δ ) 2 | P | + √ log(4 /δ ) 2 | P k | + ϖ + ϖ k . Note that E z ∼ π i [ ℓ ( w k , z )] ≤ G k , ϖ = R P ( w P ) -R ∗ Q ( w Q ) and ϖ k = R P k ( w k ) -R ∗ Q k ( w Q k ) . Specifically, ϖ and ϖ k can be regarded as constants because R P ( w P ) and R P k ( w k ) correspond to the empirical risks, R ∗ Q ( w Q ) and R ∗ Q k ( w Q k ) represent the ideal minimal empirical risk of model w Q trained on distribution Q and Q k , respectively. Moreover, these quantities ϖ and ϖ k are not taken into account during the training phase, but rather in relation to the final model.

Proof. First of all, we have

where w P k represents the model trained exclusively on group k 's data. For simplicity, when there is no confusion, we use w k to substitute w P k . The inequality R Q k ( w k ) -R Q ( w k ) ≥ 0 holds because the model tailored for a single group k can not generalize well to the entirety of the test set Q .

Then, for the first term, we have

where inequality (a) holds because of the L-smoothness of expected loss R Q ( · ) , i.e., Assumption 5.1. Specifically, inequality (b) holds because, by Cauchy-Schwarz inequality and AM-GM inequality, we have

Then, inequality (c) holds due to the L-smoothness of R Q ( · ) (Assumption 5.1), we can get a variant of Polak-Łojasiewicz inequality, which follows

where ( R ∗ Q ( w Q )) denotes the ideal minimal empirical risk of model w Q trained on distribution Q . Following a similar idea, for the second term, we also have

Combined with two terms, we have

Lastly, integrating with Lemmas D.1, D.2 and D.3, we can finish the proof.

Lemma D.1. (Train-test model gap) With probability at least 1 -δ , given the model w P trained on train set P , we have

where dist ( P , Q ) = ∑ I i =1 | p ( P ) ( π = i ) -p ( Q ) ( π = i ) | and E z ∼ π i [ ℓ ( w P , z )] ≤ G P , ∀ i ∈ I , and a constant ϖ := R P ( w P ) -R ∗ Q ( w Q ) .

Proof. First of all, we have,

For the first term (distribution shift), we have

where we define dist ( P , Q ) = ∑ I i =1 | p ( P ) ( π = i ) -p ( Q ) ( π = i ) | and E z ∼ π i [ ℓ ( w P , z )] ≤ G P , ∀ i ∈ I because of Assumption 5.2. For the second term, with probability at least 1 -δ , we have |R P ( w P ) -R P ( w P ) | ≤ √ log(4 /δ ) 2 | P | . Note that the third term R P ( w P ) -R ∗ Q ( w Q ) can be regarded as a constant ϖ .because R P ( w P ) is the empirical risk and R ∗ Q ( w Q ) is the ideal minimal empirical risk of model w Q trained on distribution Q .

Therefore, with probability at least 1 -δ , given model w P ,

Lemma D.2. (Group model gap) Suppose Assumptions 5.1 and 5.2 hold for empirical risk R P ( · ) . The initial learning rate η 2 0 < 1 √ 2 TL , where T denotes the number of training epochs. Then, we have

Proof. According to the above definition, we similarly define the following empirical risk R P k ( w ) over group k 's data P k by splitting samples according to their marginal distributions, shown as follows.

Let η t indicate the learning rate of epoch t . Then, for each epoch t , group k 's optimizer performs SGD as follows:

For any epoch t +1 , we have

where the third inequality holds since we assume that ∇ w E z ∼ π i [ ℓ ( w , z )] is L π i -Lipschitz continuous, i.e. , ||∇ w E z ∼ π i [ ℓ ( w k t , z )] -∇ w E z ∼ π i [ ℓ ( w P t , z )] || ≤ L π i || w k t -w P t || , and denote g max ( w P t ) = max I i =1 ||∇ w E z ∼ π i [ ℓ ( w P t , z )] || . The last inequality holds because the above-mentioned assumption that L = L π i = L π , ∀ i ∈ I , i.e. , Lipschitz-continuity will not be affected by the samples' classes. Then, g max ( w P t ) ≤ G because of Assumption 5.2.

For T training epochs, we have

where the last inequality holds because the initial models are the same, i.e., w 0 = w k 0 = w P 0 , ∀ k . When the condition η 2 t < η 2 0 < 1 √ 2 TL satisfies, 2 LG 2 ∑ T t =0 ( η 2 t (1 + 2 η 2 t L 2 )) T -t can be simplifited as 2 LG 2 , which is independent of the learning algorithm. This condition is easy to be satisfied since the learning rate η t is a small value ( < 0 . 0001 ) and usually set to be decay with the training epoch (i.e., η t +1 ≤ η t ).

Lemma D.3. With probability at least 1 -δ , given the model w k trained on group k 's dataset P k , we have

where dist ( P k , Q k ) = ∑ I i =1 | p ( P k ) ( π = i ) -p ( Q k ) ( π = i ) | and E z ∼ π i [ ℓ ( w k , z )] ≤ G k , ∀ i ∈ I , and ϖ k := R P k ( w k ) -R ∗ Q k ( w Q k ) .

Proof. Building upon the proof idea presented in Lemma D.1, for completeness, we provide a full proof here. Firstly, we have,

For the first term, we have

where dist ( P k , Q k ) := ∑ I i =1 | p ( P k ) ( π = i ) -p ( Q k ) ( π = i ) | and E z ∼ π i [ ℓ ( w k , z )] ≤ G k , ∀ i ∈ I due to Assumption 5.2. Recall that the constant G k clarifies the bound of loss on the corresponding model w k . For the second term, with probability at least 1 -δ , we have |R P k ( w k ) -R P k ( w k ) | ≤ √ log(4 /δ ) 2 | P k | . For the third term, we define ϖ k := R P k ( w k ) -R ∗ Q k ( w Q k ) , which can be regarded as a constant. This is because R P k ( w k ) represents empirical risk and R ∗ Q k ( w Q k ) is the ideal minimal empirical risk of model w Q k trained on sub-distribution Q k .

Therefore, with probability at least 1 -δ , given model w k ,

E More experimental results

E.1 Datasets and parameter settings

We empirically evaluate FIS on the CelebA dataset, an image dataset commonly used in the fairness literature [49]. We also evaluate FIS on two tabular datasets: UCI Adult [8] and Compas dataset [6].

E.1.1 CelebA dataset

Dataset details CelebA [49] is an image dataset with 202,599 celebrity face images annotated with 40 attributes, including gender, hair colour, age, smiling, etc. The sensitive attribute is gender: S = { Men, Women } . We select four binary classification targets, including smiling, attractive, young, and big nose. For example, the task is to predict whether a person in an image is young ( Y = 1 ) or non-young ( Y = 0 ), among other attribute predictions.

Hyper-parameter details In all our experiments using the CelebA dataset, we train a vision transformer with patch size (8 , 8) using SGD optimizer and a batch size of 128. The epochs are split into two phases: warm-up epochs (5 epochs) and training epochs (10 epochs). The default label budget per round, which represents the number of solicited data samples, is set to 256. Additionally, the default values for learning rate, momentum, and weight decay are 0.01, 0.9, and 0.0005, respectively. We initially allocate 2% of the training set for training purposes and the remaining 98% for sampling. Then, we randomly select 10% of the test data for validation. For JTT, we explore 10% data for training purposes with weights λ = 20 for retraining misclassified examples.

E.1.2 UCI Adult dataset

Dataset details. The Adult dataset [8] predicts whether an individual's annual income falls below or exceeds 50 K , denoted as Y = 0 and Y = 1 , respectively. This prediction is based on a variety of continuous and categorical attributes, including education level, age, gender, occupation, etc. The default sensitive attribute in this dataset is gender S = { Men, Women } [80]. In particular, we also group this dataset using age S = { Teenager, Non-teenager } . To achieve a balanced age distribution in the dataset, individuals with an age of less than 30 are grouped as 'Teenagers". The dataset contains a total of 45,000 instances. The dataset exhibits an imbalance: there are twice as many men as women, and only 15% of those with high incomes are women.

Hyper-parameter details. In the experiments using the Adult dataset, we train a two-layer ReLU network with a hidden size of 64. The epochs are split into two phases: warm-up epochs (100 epochs) and training epochs (60 epochs). The default label budget per round, which represents the number of solicited data samples, is set to 1024. Additionally, the default values for learning rate, momentum, and weight decay are 0.00001, 0.9, and 0.0005, respectively. We resample the datasets to balance the class and group membership [17]. The dataset is randomly split into a train and a test set in a ratio of 80 to 20. Then, we randomly re-select 20% of the training set for initial training and the remaining 80% for sampling. Also, 20% examples of the test set are selected to form a validation set. We utilize the whole model to compute the prediction influence and fairness for examples. Then, we randomly select 10% of the test data for validation. For JTT, we explore 30% data for training purposes with weights λ = 20 for retraining misclassified examples.

E.1.3 Compas dataset

Dataset details. Compas dataset, also known as the Correctional Offender Management Profiling for Alternative Sanctions dataset, is a collection of data related to criminal defendants. It contains information on approximately 6,172 individuals who were assessed for risk of re-offending. The primary task associated with this dataset is predicting whether a defendant will re-offend ( Y = 1 ) or not ( Y = 0 ) within a certain time frame after their release. The sensitive attribute is often considered to be race, specifically whether the individual is classified as African American or not.

Hyper-parameter details. In the experiments using the Compas dataset, we train a multi-layer neural network with one hidden layer consisting of 64 neurons. The epochs are split into two phases: warm-up epochs (20 epochs) and training epochs (50 epochs). The default label budget per round, which represents the number of solicited data samples, is set to 128. Furthermore, the default values for learning rate, momentum, and weight decay are 0.01, 0.9, and 0.0005, respectively. We resample the datasets to balance the class and group membership [17]. The dataset is initially split into training and test sets at an 80-20 ratio. Then, we further split 20% of the training set for initial training, reserving the remaining 80% for sampling. Additionally, 20% of the test set is selected to create a validation set. We use the entire model to calculate prediction influence and evaluate fairness for the dataset examples.

E.2 Full version of experimental results

E.3 Exploring the impact of label budgets

In our study, we examine how varying label budgets r influence the balance between accuracy and fairness. We present the results of test accuracy and fairness disparity across different label budgets on the CelebA, Compas, and Adult datasets. In these experiments, we use the demographics parity

(DP) as our fairness metric. For convenience, we maintain a fixed label budget per round, using rounds of label budget allocation to demonstrate its impact. The designated label budgets per round for the CelebA, Compas, and Adult are 256, 128, and 512, respectively. In the following figures, the x -axis is both the number of label budget rounds. The y -axis for the left and right sub-figures are test accuracy and DP gap, respectively. As observed in Figures 5-7, compared to the three baselines (BALD, JTT-20, and ISAL), our approach substantially reduces the DP gap without sacrificing test accuracy.

Specifically, on the Adult dataset, both accuracy and fairness violation converge to similar numerical values when the budget is lower than 20, suggesting a potential overfitting of the model to insufficient training examples. With a larger budget, our algorithms outperform other baseline methods, achieving higher accuracy and lower demographic disparity.

0

10

20

30

40

50

60

70

label budget

0.2

0.4

0.6

0.8

1.0

test accuracy

CelebA

BALD

JTT-20

ISAL

FIS

0

10

20

30

40

50

60

70

label budget

0.00

0.05

0.10

0.15

0.20

0.25

0.30

DP gap

CelebA

BALD

JTT-20

ISAL

FIS

0

20

40

60

80

label budget

0.50

0.55

0.60

0.65

0.70

0.75

0.80

test accuracy

Adult

BALD

JTT-20

ISAL

FIS

0

20

40

60

80

label budget

0.15

0.20

0.25

0.30

0.35

DP gap

Adult

BALD

JTT-20

ISAL

FIS

E.4 The role of validation dataset size

In this subsection, we explore the impact of adjusting the validation set size on our algorithm's performance. We present the test accuracy and fairness disparity across different validation set sizes on the CelebA, Compas, and Adult datasets. Note that the default validation set size for image and tabular datasets is set to 1% and 4% of the whole dataset size, respectively. That is, the default validation set sizes are 1996 (CelebA), 1800 (Adult), and 247 (Compas) instances, respectively. In particular, given the smaller size of the default validation set, the minimum scale of the validation set size is set to 1 / 5 × (nearly 400 CelebA images). Tables 5 and 6 present the performance results on the CelebA, UCI Adult, and Compas datasets, respectively.

E.5 Benchmarking model performance with validation set enhancements

Note that we resort to an additional validation set for developing FIS. To demonstrate FIS's advantage at the same levels of information, we introduce a new baseline called Random+Val . This method

0

20

40

60

80

label budget

0.66

0.68

0.70

0.72

0.74

test accuracy

Compas

BALD

JTT-20

ISAL

FIS

0

20

40

60

80

label budget

0.26

0.28

0.30

0.32

0.34

DP gap

Compas

BALD

JTT-20

ISAL

FIS

involves continuing to train the model with a randomly sampled validation set. Specifically, we start with the Random 's last saved checkpoint and train it further using the validation set. In particular, we would incorporate a fairness regularizer with dynamic weight to train the model using validation data, considering its sensitive attributes to reduce fairness disparity. Due to its small size, we limit training to 10 epochs to avoid overfitting. The performance results of Random , Random+Val , and FIS are given in Table 7.

F Potential privacy leakage from non-demographic and demographic feature correlations

Note that some non-demographic features may correlate with sensitive attributes (demographic information), which could lead to potential privacy leakage issues [82]. However, it is crucial to clarify that our method specifically limits its query to the data. We only query the true labels for a selected subset of unlabeled examples. While correlations between non-demographic variables and sensitive attributes may exist in the underlying data, our method itself does not introduce any additional privacy leakage beyond what is inherent in the original dataset. On the other hand, our

work's primary focus is not on addressing privacy concerns related to demographic information. Rather, our main objective is to reduce fairness disparities while maintaining a favorable utility trade-off. We achieve this without explicitly incorporating any additional sensitive information into the process and without directly engaging with the complex privacy implications of demographic data usage.

̸

To address this potential privacy concern, we can analyze it via differential privacy. Consider a function ϕ ( · ) that maps non-demographic information (feature x and label y ) to a sensitive attribute s . Due to insufficient information, ϕ ( · ) is not deterministic, making it unable to precisely estimate s . If a deterministic mapping function existed, it would unavoidably lead to privacy leakage. Therefore, we assume that given the same features and label, the function ϕ ( · ) output might vary. For example, in the Adult dataset, x includes age and credit history, y is the income, and s is the gender. In practice, both men and women have a probability of earning more than 50K (i.e., y = 1 ) given the same age, credit history, etc (same feature x ). Suppose P ( ϕ ( x , y ) = s | S = s, x , y ) ≤ 1 -ϵ 0 and P ( ϕ ( x , y ) = s | S = s ′ , x , y ) ≥ ϵ 1 for all x , y, s, s ′ (where s = s ′ ). Here, S represents the sensitive attribute variable, which is unknown to ϕ ( · ) . Then, we have

where ε = ln ( 1 -ϵ 0 ϵ 1 ) . In practice, if the mapping function is too strong, i.e. ln ( 1 -ϵ 0 ϵ 1 ) is too large, we can add additional noise to reduce their informativeness and therefore better protect privacy.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The claims in the abstract reflect our primary contributions that we propose an active sampling approach to mitigate unfairness without compromising accuracy. One notable advantage is our approach does not require eliciting sensitive attributes from the unlabeled data.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discussed the limitations in the concluding remarks section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We list a series of assumptions we used in Assumption 5.1 and Assumption 5.2.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We include all the experimental details in the Appendix. We also include the source code in the supplementary material for reproducing the experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We include the README file for the instructions of reproducing the results. Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have included these details in the setup as well as the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report the standard deviations in the main result table.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We report the computational resource in the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conform with the NeurIPS Code of Ethics. Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We have a dedicated broader impact section that addressed the ethical concerns. Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not release any data or models.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have credited the datasets we used for the experiments.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our work does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our work does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose Fair Influential Sampling(FIS) to achieve a better Pareto Frontier of the fairness-accuracy tradeoff through sampling training data. To maintain fairness without diminishing the accuracy, training data examples are validated through a validation dataset. In this case, privacy is maintained by not annotating sensitive attributes in the large training dataset, but doing so for the validation dataset. An example data is selected as training data based on its influence on the accuracy and fairness using the validation dataset. This research also claims that by acquiring new data the proposed algorithm can decrease the group gap by utilizing the fairness components and can also prevent the negative impact of distribution shift by utilizing accuracy influence components, which in theory should improve fairness without harming the accuracy.\n\nSTRENGTHS:\nThe proposed sampling approach attempts to mitigate group fairness without compromising both accuracy and privacy.\n\nAs the FIS does not require annotation of sensitive attributes for the large training dataset a substantial amount of cost and labor is negated.\n\nThe work addresses potential distribution shift issues effectively to maintain fairness and accuracy.\n\nEven though the method is validated on some specific datasets the basic mechanism of the FIS has the potential to be applicable in different sectors where maintaining fairness is a crucial factor.\n\nThe maintenance of privacy supports ethical AI application development practices.\n\nWEAKNESSES:\nThe success of the FIS is heavily dependent on the refinement level of the validation set. If the validation set is not a proper representation of the large training dataset then it may compromise the accuracy and fairness.\n\nThe paper's sampling strategy mostly depends on particular influence measurements. The sensitivity of these matrices for different data distributions should be thoroughly explored.\n\nIf there is bias present in the validation dataset, unfairness may be amplified by the FIS method.\n\nThe proposed method relies substantially on a clean and informative validation dataset. Which might be hard to secure in certain cases.\n\nThe paper does not mention several recent studies in the related work section, which could provide additional context or alternative methodologies for addressing fairness without demographic data. Here are a few examples:\n\n[1] Zhao, Tianxiang, et al. \"Towards fair classifiers without sensitive attributes: Exploring biases in related features.\" Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining. 2022.\n\n[2] Yan, Shen, Hsien-te Kao, and Emilio Ferrara. \"Fair class balancing: Enhancing model fairness without observing sensitive attributes.\" Proceedings of the 29th ACM International Conference on Information & Knowledge Management. 2020.\n\n[3] Chai, Junyi, Taeuk Jang, and Xiaoqian Wang. \"Fairness without demographics through knowledge distillation.\" Advances in Neural Information Processing Systems 35 (2022): 19152-19164.\n\nThe authors' method overlooks the presence of non-demographic features that could still relate to demographic information, posing a privacy issue even without direct inclusion of demographic information.\n\nAlthough in “Empirical Results” section states that 4 datasets were used, according to Appendix E, only 3 datasets(CelebA dataset, Adult dataset, COMPAS dataset) were used for experimental purposes.\n \nIn 4.2, it is stated that line no 9 updates U and Soriginal, but according to the pseudo code this process is being executed in line no 11.\n\nQUESTIONS:\nPlease refer to the “Weaknesses” section.\n\nLIMITATIONS:\nIn this paper, the authors propose FIS to preserve the fairness of the model and the privacy of individuals. However, several limitations can be observed in the sampling process. Firstly, the authors should consider the computational expenses of the algorithm involving different scales of datasets. Moreover, as sampling relies on the validation dataset, its fairness may be jeopardized if the validation dataset is skewed. Additionally, it would be much more efficient to develop an algorithm to select a validation dataset that contains little to no bias. The sampling method also does not explain how fairness is maintained in the case of expanding datasets.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenge of achieving fairness in machine learning (ML) without compromising model accuracy. The authors propose a novel active data sampling algorithm that mitigates group fairness disparity by acquiring more data without requiring sensitive attribute annotations for training, thus protecting privacy. The algorithm scores each new example based on its influence on fairness and accuracy, evaluated on a small validation set, and selects examples for training accordingly. Theoretical analysis demonstrates how acquiring more data can improve fairness without degrading accuracy, and provides upper bounds for generalization error and risk disparity. Extensive experiments on real-world data validate the effectiveness of the proposed approach.\n\nSTRENGTHS:\nClear & meaningful motivation: The proposed fair influential sampling (FIS) method avoids the need for sensitive attribute annotations during the sampling or training phases, thereby protecting privacy.\n\nSound Theoretical Contributions: The paper includes a thorough theoretical analysis, showing how the proposed algorithm can improve fairness without harming accuracy. It also provides upper bounds for generalization error and risk disparity, enhancing the method's credibility.\n\nApplicability: Based on the method design & setup, it seems it's not limited to single sensitive variable, rather multiple discrete ones. Therefore, this renders the method considerably better applicability compared to most other fairness methods.\n\nWEAKNESSES:\n1) Insufficient discussion of fairness metrics: The choice and justification of the fairness metrics used in the evaluation could be more thoroughly discussed. A deeper explanation of why these particular metrics were selected and how they effectively measure fairness would provide a clearer understanding of how fairness is quantified in this study and the implications of these choices. For example, why choose risk disparity as the fairness definition, and how does it compare & contrast with other metrics, like equalized odds, or counterfactual fairness metrics.\n\n2) As the authors point out, The proposed algorithm relies on a clean and informative validation set that contains sensitive attributes of data examples. This dependency may not always be practical or feasible in real-world scenarios, where obtaining such validation sets can be challenging due to privacy concerns and data availability.\n\n3) Unclear experiment results presentation and significance: I find key result in table 1 confusing. For example, how is fairness_violation defined, and what is the highlighting scheme, and why is the row of DP of CelebA - Attractive not highlighted.\n\nQUESTIONS:\n1) Could you provide more discussion on the choice of fairness definition used in your evaluation? I have skimmed section B of your appendix, but why is risk disparity selected as the fairness definition throughout, and how does risk disparity compare with other fairness metrics like equalized odds or counterfactual fairness metrics?\n\n2) Your proposed algorithm relies on a clean and informative validation set that contains sensitive attributes. How do you address the potential challenges in obtaining such validation sets in practical scenarios, especially considering privacy concerns and data availability? What are the implications if the validation set is not fully representative or contains noisy labels, and how might this affect the performance and fairness of your algorithm?\n\n3) Based on my understanding of equation 5, your method is compatible with multiple (discrete) sensitive variables. It that correct?\n\n4) For table 1: How is fairness_violation defined? Why is the row of DP of CelebA - Attractive not highlighted? What if the highlighting rule?\n\nWriting suggestion: I am confused with the highlighting scheme of Table 1. It would be better if it's explained in the caption right above.\n\nLIMITATIONS:\nThe authors mentioned limitation that it needs validation set though did not attempt to address it.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nGroup/statistical notions of fairness are known to be at odds with the accuracy metric. The authors propose an active sampling strategy to improve the tradeoff between fairness and predictive performance. This sampling approach only requires the sensitive information to be known in a small validation set. They estimate the influence of a potential instance on improving the accuracy and fairness, and choose those that most positively influence both objectives.\n\nSTRENGTHS:\nS1 - Improving the accuracy fairness trade-off is a very relevant problem. Besides, doing it in the scenario when only minimal sensitive information requires to be known extends its applicability to real-world scenarios.\n\nS2 -  Figure 1 offers a nice overview to situate the problem and provide an intuition regarding the objective of the work. \n\nS3 - In general terms, the readability of the text is quite good. The ideas are presented clearly, with cohesive explanations throughout. The text is well divided into sections and paragraphs, which helps maintain a consistent flow and makes it easy to follow and understand. \n\nS4 - The paper demonstrates commendable attention to reproducibility by providing detailed information regarding the experimental setup.\n\nWEAKNESSES:\nW1 - The related works section needs improvement, starting from the most general topics (active learning and fair classification), to more specific areas of research. Additionally, the claim made by the authors in lines 107-108 is not true. Those approaches do not strive to achieve uniform accuracy over all groups, but rather to improve the predictive performance of the classifier on the worst performing groups. Indeed, it is in the metrics considered by the authors where parity among groups is required. Besides, the acronyms DP and EOp are used before defining them. Moreover, when talking about [39] in line 92 it is not which problem the recent work aims to address.   \n\nW2 - In section 4.1 you introduce the influence of the accuracy component but what you are looking into is the loss, and the loss need not be equal to accuracy. Maybe you could rename it and call it the loss/risk component, or change the definition otherwise.\n\nW3 - When active sampling is used the iid condition is not satisfied anymore, but the authors do not mention this in the work nor address this challenge. \n\nW4 - One of the main claims made in the work is that this strategy is able to improve the fairness guarantees of classifiers without sacrificing accuracy. The authors even claim the latter can be seen from Table 1. However, in the empirical validation there are no numerical results regarding the accuracy, only fairness-related metrics are shown. Consequently, this claim is impossible to validate from the results provided in the main paper. \n\nW5 - The comparison with respect to the JTT method is unfair. In fact, this procedure has a strategy to choose the optimal weight that needs to be given to the misclassified instances, and is learned in the training process. Thus determining the weight to be 20 is unfair. \n\nW6 - It is not very clear how much data/information is given to each of the methods, which makes it difficult to verify whether the empirical validation is fair.\n\nW7 - The main objective of the active sampling paradigm is to achieve superior performance (in this case better accuracy-fairness trade-off) with less datapoints, as they are chosen effectively. However, there is no empirical validation nor theoretical statement that supports the latter.\n\nAdditional minor comments:\n\nW - In the paragraph labelling from section 4.2, the explanation that introduces the paragraph is not very clear, and it is in fact misleading… it would be nice to improve its clarity. \n\nW - In line 235 you talk about ‘most confident labels’, but maybe employing the word confidence is not the most appropriate. \n\nW - The authors do not explicitly state that the fairness loss is something to be minimized. \n\nW - The authors do not clearly state how the hyperparameters of the rest of the methods are tuned. \nW - In the paragraph *proposed algorithm* from section 4.2, the authors talk about tolerance $\\epsilon$ but there is no such parameter in Algorithm 1. \n\nW - You are talking about the trade-off between accuracy and fairness but in line 89 you use the word generalization, which is not a synonym but a word with a completely different meaning.\n\nW - The text contains several typos. For instance, between lines 83-85, work(s) need to be plural, and train examples are better references as training examples. Also the authors employ ‘to us’ in line 113, and it would be better to replace it with a more formal phrase such as ‘to this work’.\n\nW - In line 86 when you talk about the trade-off it would be nice to include some references (as in the introduction).\n\nW - Check how the references are written, missing capital letters in [49] and [23]; no publish information in [57], etc. \n\nW - Regarding the titles of the sections, only the first letter of the first word needs to go in uppercase. \n\nW - $\\G_p$ is not defined.\n\nQUESTIONS:\nQ1 - What do > symbols refer to in equations (2) and (3)? Vectorial product?\n\nQ2 - According to the definitions of Lemma 4.1 and 4.2 the value of the influence depends on the size of the validation set. Why don’t you use an average so that this characteristic does not influence?\n\nQ3 - In the section *proposed algorithm* you describe an iterative procedure to find the top r instances whose fairness influence based on the true labels meet the conditions, but you are using proxy labels and not the true labels. Then, how do you know the latter?\n\nQ4 - Why do you need to make the assumption that the train/test distributions are drawn from a series of component distributions?\n\nQ5 - Regarding Theorem 5.2 you talk about the group gap, and claim that the more balanced the data is (...). But which gap/balance do you refer to? Is it in terms of representation or performance?\n\nQ6 - Regarding the tabular data, you resample the data to balance the class and group membership; but what is the reason behind this procedure? What happens if there is no such balance?\n\nLIMITATIONS:\nLimitations are clearly stated by the authors in Section 7.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a method called Fair Influential Sampling (FIS), which selects candidate data points for training based on their estimated improvement on both fairness and accuracy. This estimation considers the influence of a new data point as the average gain in accuracy and fairness achieved by performing a gradient descent step with the new data point. This average is empirically computed using a labeled validation dataset. The gain in loss (or fairness) is approximated as the inner product between the gradient of the loss (or fairness) with respect to the parameters evaluated on the new data point and each validation point.\n\nThe proposed influence metric does not require access to the sensitive attributes of the candidate data points. Additionally, when labels for the target variable are unavailable for candidate data points, the authors suggest producing proxy labels by annotating the most confident class. This allows the estimation of the accuracy influence score. If a data point is selected for further training, a user must then perform the actual labeling.\n\nThe selected data points are those that improve the fairness influence score without negatively impacting the average accuracy score on the validation set. Experimental results demonstrate that the proposed approach can select training data points that enhance fairness without reducing accuracy compared to a random sampling strategy. Furthermore, under the constraint of maintaining an accuracy equal to or better than a \"random\" model, FIS achieves superior fairness performance compared to other competing methods.\n\nSTRENGTHS:\nThe paper is well-written, addressing a relevant problem with clear formulation and proposed methods. The authors provide theoretical justification for the approximations used in computing their influence scores and offer formal analysis on the impact of adding more data on model generalization.\n\nExperimental results demonstrate that the proposed active sampling approach performs as expected. It does not harm accuracy compared to a base ERM model or random sampling. Moreover, the selected data points improve fairness under this constraint. Additionally, the proposed approach does not necessarily require labeled data, which is advantageous as only ground truth labels for the accepted data are needed. This can motivate the use of this method.\n\nThat being said, I am not entirely familiar with the extensive state-of-the-art literature on active sampling to fully evaluate the novelty of the proposed approach.\n\nWEAKNESSES:\nIt seems to me that the data selection strategy based on the proposed influence scores is approximating the process of training with the validation set. I believe it would be beneficial to compare the performance of a model trained with this labeled validation set (without sampling and with random sampling) to demonstrate the advantage of using a data sampling strategy based on the validation set instead of simply including it as part of the training data.\n\nQUESTIONS:\nCould the authors indicate if the results in Table 1 are computed with respect to a test dataset that does not include the one used for validation and sample selection? I ask this because in line 330, the paper states that \"10% of the test data is randomly set aside as a hold-out validation set.\" I assume this portion is not used for evaluation.\n\nTypo in Eq 4, shouldn’t \\ell be \\phi?\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-24"}