Title: Fast Model DeBias with Machine Unlearning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Model Debias

Bias Mitigation

Machine Unlearning

Counterfactual Fairness
Abstract: Recent discoveries have revealed that deep neural networks might behave in a biased manner in many real-world scenarios. For instance, deep networks trained on a large-scale face recognition dataset CelebA tend to predict blonde hair for females and black hair for males. Such biases not only jeopardize the robustness of models but also perpetuate and amplify social biases, which is especially concerning for automated decision-making processes in healthcare, recruitment, etc., as they could exacerbate unfair economic and social inequalities among different groups. Existing debiasing methods suffer from high costs in bias labeling or model re-training, while also exhibiting a deficiency in terms of elucidating the origins of biases within the model. To this respect, we propose a fast model debiasing method (FMD) which offers an efficient approach to identify, evaluate and remove biases inherent in trained models. The FMD identifies biased attributes through an explicit counterfactual concept and quantifies the influence of data samples with influence functions. Moreover, we design a machine unlearning-based strategy to efficiently and effectively remove the bias in a trained model with a small counterfactual dataset. 
Experiments on the Colored MNIST, CelebA, and Adult Income datasets demonstrate that our method achieves superior or competing classification accuracies compared with state-of-the-art retraining-based methods while attaining significantly fewer biases and requiring much less debiasing cost. Notably, our method requires only a small external dataset and updating a minimal amount of model parameters, without the requirement of access to training data that may be too large or unavailable in practice.

Fast Model Debias with Machine Unlearning

Abstract

Recent discoveries have revealed that deep neural networks might behave in a biased manner in many real-world scenarios. For instance, deep networks trained on a large-scale face recognition dataset CelebA tend to predict blonde hair for females and black hair for males. Such biases not only jeopardize the robustness of models but also perpetuate and amplify social biases, which is especially concerning for automated decision-making processes in healthcare, recruitment, etc., as they could exacerbate unfair economic and social inequalities among different groups. Existing debiasing methods suffer from high costs in bias labeling or model re-training, while also exhibiting a deficiency in terms of elucidating the origins of biases within the model. To this respect, we propose a fast model debiasing framework (FMD) which offers an efficient approach to identify, evaluate and remove biases inherent in trained models. The FMD identifies biased attributes through an explicit counterfactual concept and quantifies the influence of data samples with influence functions. Moreover, we design a machine unlearning-based strategy to efficiently and effectively remove the bias in a trained model with a small counterfactual dataset. Experiments on the Colored MNIST, CelebA, and Adult Income datasets along with experiments with large language models demonstrate that our method achieves superior or competing accuracies compared with state-of-the-art methods while attaining significantly fewer biases and requiring much less debiasing cost. Notably, our method requires only a small external dataset and updating a minimal amount of model parameters, without the requirement of access to training data that may be too large or unavailable in practice.

1 Introduction

Biased predictions are not uncommon in well-trained deep neural networks [1-3]. Recent findings indicate that many deep neural networks exhibit biased behaviors and fail to generalize to unseen data [4, 5], e.g., convolutional neural networks (CNNs) might favor texture over shape for object classification [6]. For instance, well-trained networks on a large-scale dataset (e.g. CelebA) tend to predict a female person to be with blonde hair, and a male to be with black hair [7, 8]. This is because the number of <blonder hair, female> and <black hair, male> image pairs is significantly higher than others, although there is no causal relationship between hair color and gender [9]. In this case, the model does not learn the correct classification strategy based on human appearance, but rather shows a preference for specific individuals or groups based on irrelevant attributes (error correlations) [2]. Such error correlations not only affect the model's ability to make robust predictions but also perpetuate and exacerbate social bias, resulting in potential risks in many real-world scenarios, such as racism, underestimating minorities, or social disparities among groups in crime prediction [10], loan assessment [11], and recruitment [12] etc.

Efforts have been made to remove bias in models based on innate or acquired characteristics of individuals or groups. Existing debiasing mechanisms could be categorized into three types depending on when debiasing is conducted: pre-processing, in-processing, and post-processing [2, 13, 14]. Pre-processing debiasing methods usually modify the dataset for fair learning, which often involve reweighing samples [15, 16], modifying feature representations [17, 18], changing labels [19] etc. Another line of research accounts for fairness during training, i.e., in-processing [20-24], including feature-level data augmentation or adversarial training [25, 26] etc. However, the aforementioned methods require expensive costs for human labeling of misleading biases or computationally-intensive debiased model retraining, resulting in unsatisfactory scalability over modern large-scale datasets or models. Few research explore post-processing strategies to achieve fairness with minimal cost [2729]. They ensure group fairness by alternating predictions of some selected samples, causing degraded accuracy or unfairness on individuals. Moreover, most methods assume that the biased attributes were known, while a generalized debiasing framework should be able to verify whether an attribute (e.g. shape, texture, and color in an image classification task) is biased or not as well [30].

y: blonde

a: female

y: dark

a: male

Train

Test

Trained

Model

Unlearn

Test

pred: blonde

prob: 0.94

pred: dark

prob: 0.83

pred: blonde

prob: 0.89

pred: dark

prob: 0.63

pred: blonde

prob: 0.91

pred: blonde

prob: 0.65

pred: blonde

prob: 0.83

pred: blonde

prob: 0.70

Unlearned

Model

Biased-Effect Evaluation

ARD

WARD!

In this paper, we propose FMD, an all-inclusive framework for fast model debiasing. As illustrated in Fig. 1, the FMD comprises three distinct steps: bias identification, biased-effect evaluation, and bias removal. In contrast to pre- or in-processing debiasing methods, our approach eliminates the need for supervised retraining of the entire model or additional labeling of bias attributes. Notably, FMD leverages only a small external dataset, thereby obviating the requirement for access to extensive or unavailable training data in practical scenarios. Furthermore, achieving fair outputs through FMD necessitates updating only a minimal number of parameters, such as the top MLP layers of pre-trained deep networks. Compared to post-processing debiasing methods, FMD yields superior debiasing performance and consistently enhances fairness across diverse bias metrics with little costs.

The FMD operates through the following procedure. Given an attribute and a well-trained model, our first step is to ascertain whether and to what extent the model exhibits bias towards the attribute. To achieve this, we construct a dataset comprising factual samples along with their corresponding counterfactual samples [31], wherein the attribute in question can be varied. By observing how the model's predictions change with the attribute variations, we can effectively identify any bias present. In the biased-effect evaluation phase, we quantitatively assess the extent to which a biased training sample contributes to the model's biased predictions. This evaluation entails measuring how the biased training sample misleads the model and influences its predictions. To this end, we extend the theory of influence functions [32], employing it to estimate the impact of perturbing a biased attribute within the training data on the model's prediction bias measurement. Finally, we introduce an unlearning mechanism that involves performing a Newton step [33] on the learned model parameters to remove the learned biased correlation. We further design an alternative strategy to unlearn biases with the counterfactual external dataset, avoiding hard requirements on access to the training data which might be unavailable in practice. Our unlearning strategy effectively eliminates the estimated influence of the biased attribute, leading to a more fair and unbiased model. Experiments on multiple datasets show that our method can achieve accuracies on par with bias-tailored training methods with a much smaller counterfactually constructed dataset. The corresponding biases and computational costs are significantly reduced as well. Our main contributions are summarized as:

We propose a counterfactual inference-based framework that can quantitatively measure the biased degree of a trained (black-box) deep network with respect to different data attributes with a novel influence function.

We propose an unlearning-based debiasing method that effectively and efficiently removes model biases with a small counterfactual dataset, getting rid of expensive network re-training or bias labeling. Our approach inherently applies to in-processing debiasing.

Extensive experiments and detailed analysis on multiple datasets demonstrate that our framework can obtain competing accuracies with significantly smaller biases and much fewer data and computational costs.

2 Related Works

2.1 Group, Individual and Counterfactual Fairness

The pursuit of fairness in machine learning has led to the proposal of fairness-specific metrics. These metrics have been mainly categorized into two types: metrics for group fairness that require similar average outputs of different demographic groups [34-38]; and metrics for individual fairness that necessitate similarity in the probability distributions of individuals that are similar in respect to a specific task, regardless of their demographic group [39-42]. Generally, statistical parity among protected groups in each class (group fairness) could be intuitively unfair at the individual level [43]. Moreover, existing fairness metrics put a heavy emphasis on model predictions, while underestimating the significance of sensitive attributes for decision-making and are insufficient to explain the cause of unfairness in the task [31, 44]. Recently, [31] introduces counterfactual fairness, a causal approach to address individual fairness, which enforces that the distribution of potential predictions for an individual should remain consistent when the individual's protected attributes had been different in a causal sense. In contrast to existing individual bias metrics, counterfactual fairness can explicitly model the causality between biased attributes and unfair predictions, which provides explainability for different biases that may arise towards individuals based on sensitive attributes [45-47].

2.2 Bias Mitigation

Proposed debiasing mechanisms are typically categorized into three types[2, 13, 14]: pre-processing, in-processing, and post-processing. Pre- and in-processing algorithms account for fairness before and during the training process, where typical techniques entail dataset modification [15-19] and feature manipulation [20-24, 26, 25]. Post-processing algorithms are performed after training, intending to achieve fairness without the need of modifying data or re-training the model. Current post-processing algorithms make more fair decisions by tweaking the output scores [48-50]. For instance, Hardt [27] achieves equal odds or equal opportunity by flipping certain decisions of the classifier according to their sub-groups. [29, 28] select different thresholds for each group, in a manner that maximizes accuracy and minimizes demographic parity. However, achieving group fairness by simply changing the predictions of several individuals is questionable, e.g., the process might be unfair to the selected individuals, leading to an unsatisfactory trade-off between accuracy and fairness.

2.3 Machine Unlearning

Machine unlearning [51-53] is a new paradigm to forget a specific data sample and remove its corresponding influence from a trained model, without the requirement to re-train the model from scratch. It fulfills a user's right to unlearn her private information, i.e., the right to be forgotten, in accordance with requests from the General Data Protection Regulation (GDPR) [54]. Existing unlearning approaches can be roughly categorized into two types: exact unlearning [55, 56] and approximate unlearning [57-60]. Data influence-based unlearning is a representative branch of approximate unlearning that utilizes influence functions [32] to approximate and remove the effect of a training sample on the model's parameters [61-63]. In this paper, we are inspired by the paradigm of machine unlearning and extend it to remove the model's bias from a deep network without retraining it from scratch.

3 Method

3.1 Overview and Preliminaries

Problem Formulation. Consider a supervised prediction task with fairness considerations that maps input attributes A (biased attribute) and X (other attributes except A ) to certain outputs Y (labels). The training dataset D tr can be represented as { z 1 , z 2 , ..., z n } where each training point z i = { ( a i , x i ) , y i } ∈ A × X × Y . Let f ˆ θ denote the trained model (predictor) with parameter ˆ θ . Let L ( z i , θ ) denote the loss on the training sample z i w.r.t. parameter θ . It is deemed biased if a biased attribute a is highly correlated but wrongly correlated to the prediction ˆ y = f ˆ θ ( x, a ) , e.g., a CNN is biased if it predicts hair color (black/blonde) with the biased attribute genders (male/female).

Motivation. In large part, existing works focused on measuring fairness with implicit quantitative values (e.g. accuracy). However, they do not provide explicit illustrations on whether the decisionmaking is based on sensitive/protected attributes. Furthermore, based on the bias identified, research on how such bias is learned from training samples is limited. Our proposed method bridges this gap with two components: identifying bias from different predictions with counterfactual samples and evaluating the biased-effect from training samples with a modified influence function. Furthermore, we propose a novel machine unlearning-based method to efficiently and effectively remove the biases.

Counterfactual Fairness. We identify the biases of trained models with the concept of counterfactual fairness [31, 46, 45] which better models the causality between biased attributes and unfair predictions. We detail the definition following [31]:

Definition 1 (Counterfactual fairness) . A trained model f ˆ θ is counterfactual fair on A if for any a, ¯ a ∈ A ,

for all x ∈ X attainable by X .

Note that y = f ¯ θ ( X,A ) , which implies the process of attribute changing. The definition suggests that, for any individual, changing a , i.e., from a to ¯ a , while holding other attributes x unchanged should not change the distribution of ˆ Y if a is a biased attribute.

Influence function. Influence functions, a standard technique from robust statistics, are recently extended to characterize the contribution of a given training sample to predictions in deep networks [32, 64, 65], e.g., identify whether a sample is helpful or harmful for model predictions. A popular implementation of influence functions is to approximate the effects by applying the perturbation z = ( x, y ) ↦→ z δ = ( x + δ, y ) [32] that define the parameters resulting from moving ϵ mass from z onto z δ : ˆ θ ϵ,z δ , -z = argmin θ ∈ Θ 1 n ∑ n i =1 L ( z i , θ ) + ϵL ( z δ , θ ) -ϵL ( z, θ ) . An approximated computation of the influence as in [32] can be defined as:

3.2 Bias Identification and Biased-Effect Evaluation

Counterfactual bias identification. Wefirst identify the biases in a trained model with counterfactual concepts. Given a trained model f ˆ θ and an attribute of interest A , a primary question is whether f ˆ θ is fair on A . We employ an external dataset D ex (can be constructed from the test set) to identify biases. To measure how prediction changes in accordance with the attribute, for each sample c i = ( x i , a i ) ∈ D ex , where a i ∈ A , we alter a i while keeping x i unchanged based on the requirements of counterfactual fairness. The generated counterfactual sample is denoted as ¯ c i = ( x i , ¯ a i ) , ¯ a i ∈ A . We further define the counterfactual bias of the model f ˆ θ on sample c i as the difference in predictions:

The counterfactual bias on the whole dataset D ex can be represented as the average of individual counterfactual biases:

The measured bias is a scalar normalized from 0 to 1. We set a bias threshold δ that if the measured B ( D ex , A , f ˆ θ ) is larger than δ , we regard f ˆ θ to be biased on A . Note that our method could also generalize to other individual bias metrics besides Eq. 3.

̸

Biased-Effect Evaluation. Based on the identified counterfactual bias, we then investigate how the bias on A is learned by the model from training samples. Considering B ( ˆ θ ) measured on any A with any D ex , our goal is to quantify how each training point z k in the training set D tr contributes to B ( ˆ θ ) . Let's denote the empirical risk minimizer as ˆ θ = arg min θ 1 n ∑ n i =1 L ( z i , θ ) , and assume that the empirical risk is twice-differentiable and strictly convex in θ . The influence function [64] provides an approximation on the updates to parameters if z k were removed from D tr with a small coefficient ϵ . The new parameters can be obtained as ˆ θ ϵ,z k = arg min θ 1 n ∑ n i =1 ,i = k L ( z i , θ ) + ϵL ( z k , θ ) . By doing so, the influence of removing z k on the bias B ( ˆ θ ) can be defined as:

where H ˆ θ def = 1 n ∑ n i =1 ∇ 2 θ L ( z k , ˆ θ ) is the positive definite (PD) Hessian, and the closed form expression of d ˆ θ ϵ,z k dϵ ∣ ∣ ∣ ϵ =0 , explaining the influence of z k to model parameters, is provided by the influence function [32]. Note that "up" denotes "upweight". Refer to Appendix A for the derivation. Intuitively, this equation can be understood in two parts: the latter part calculates the impact of removing on the parameters. The former part corresponds to the derivative of bias with respect to parameters, assessing how changes in parameters affect the bias. Hence, this equation quantifies the influence of removing on the bias. Note that B ( ˆ θ ) can be any bias measurement of interest. Taking B ( D ex , A , ˆ θ ) defined in Eq. 4 as an example, the influence on counterfactual bias can be boiled down as:

where I up,bias ( z k , B ) is a scalar that measures how each training sample contributes to B . If removing the point z k increases the bias, we regard z k as a helpful sample, or harmful otherwise. We provide an illustration of the helpful and harmful samples with a toy example in Section B.1.

3.3 Bias Removal via Machine Unlearning

After quantifying how biases are learned by the model from harmful samples, the next question is how to remove such biases. Here we propose a machine unlearning-based strategy to remove the biases caused by harmful samples. In particular, we exploit the powerful capability of machine unlearning paradigms for forgetting certain training samples [66, 62, 63, 61]. Specifically, for a bias measurement B ( ˆ θ ) ,we first rank the influence I up,bias ( z k , B ( ˆ θ )) of every training sample z k in D tr , and then select the topK harmful samples. Afterward, we unlearn, i.e., let the model forget, these samples by updating the model parameters θ with a Newton update step as in [63]:

where H -1 ˆ θ ∇ ˆ θ L ( z k , ˆ θ ) = I up,params ( z k ) is explained as the influence of z k on model parameter [32]. Note that I up,params ( z k ) share similar computation as in Eq. 6, while I up,params ( z k ) estimates the influence on model parameter and I up,bias ( z k , B ) focuses on influence on biases.

Our unlearning strategy is further refined following the observations from experiments in Section B.1. In particular, by ranking and visualizing the harmful and helpful samples on the biases (as shown in Fig. 5), we have observed that the harmful samples heavily lead to biased/error correlations (i.e., bias-aligned) while the helpful samples behave oppositely (i.e., bias-conflicting). Hence, we propose a straightforward solution that further mitigates the influence of a harmful sample with a

bias-conflicting sample. Consequently, we update the parameters to unlearn the harmful samples by:

where ¯ z k denotes the bias-conflicting sample of z k . Following the explanation in influence theory [32], our unlearn mechanism removes the effect of perturbing a training point (¯ a, x, y ) to ( a, x, y ) . In other words, we not only remove the influence caused by harmful sample z k , but further ensure fairness with the corresponding counterfactual sample ¯ z k , see more details in Section B.1, 4.4 and Appendix.

Alternative Efficient Unlearn with Cheap External Datasets. In the above sections, the unlearning process is based on the assumption that we could access the original training sample z k to identify and evaluate biases and then forget them. However, in practice, the training set might be too large or even unavailable in the unlearning phase. In response, we further propose to approximate the unlearning mechanism with a small external dataset. As the influence to be removed can be obtained from the change of the protected attribute, we can construct the same modification to the protected attribute on external samples. In particular, we employ the D ex as in Section 3.2 to construct counterfactual pairs for unlearning, which redefines Eq. 21 as:

As D ex can be easily obtained from an external dataset rather than the training set, the practical applicability of our method could be greatly enhanced, as demonstrated in the experiments.

3.4 Model Generalization

Extension to Different Biases. To fulfill different fairness demands, we further discuss the generalization of the bias function B ( ˆ θ ) in Eq. 6 to other bias measurements. We provide the extension to the most frequently used group fairness measurement demographic parity [34] which requires equal positive prediction assignment across subgroups (e.g. male and female). Eq. 6 can be rewritten as:

where G A =1 and G A =0 represents the subgroup with protected attribute A = 1 and 0 . The extension to equal opportunity [35], which requires the positive predictions to be equally assigned across positive classes, can be rewritten as:

where G 1 , 1 represents the sub-group where A = 1 and Y = 1 .

Extension to Deep Models. In the previous sections, it's assumed that ˆ θ could be the global minimum. However, if ˆ θ is obtained in deep networks trained with SGD in a non-convex setting, it might be a local optimum and the exact influence can hardly be computed. We follow the strategy in [32] to approximate the influence in deep networks, and empirically demonstrate the effectiveness of FMD in deep models. Moreover, for deep networks where a linear classifier is stacked on a backbone feature extractor, we apply our unlearning mechanism to the linear classifier or several top MLP layers.

Efficient Influence Computation. A critical challenge to compute the influence in Eq. 6 is to explicitly calculate the inverse Hessian. Here we employ the implicit Hessianvector products (HVPs) [32, 67] to efficiently approximate ∇ ˆ θ L ( z k , ˆ θ ) . Meanwhile, ∇ ˆ θ L ( z k , ˆ θ ) in Eq. 6 can be precalculated and applied to different ∇ ˆ θ B ( ˆ θ ) . To avoid the O ( d 3 ) computational cost to calculate the inverse Hessian in every step, we pre-calculate it before the removal and keep it constant during unlearning phase [63]. The alternative strategy which continuously updates the inversion Hessian is also analyzed in the Appendix.

4 Experiment

4.1 Experiment details

Dataset. Our experiments are conducted on three datasets. Colored MNIST is constructed by adding color bias to the MNIST dataset [68]. Bias-aligned samples are constructed by adding a particular color to a particular digit like {Digit1_Color1} while other colors are for bias-conflicting samples. Following [3, 69, 70], we build 3 different training sets by setting different biased ratios {0.995, 0.99, 0.95} for biased-aligned training samples, where a high ratio indicates a high degree of bias. CelebA [71] is a face recognition dataset with 40 types of attributes like gender, age (young or not), and lots of facial characteristics (such as hair color, smile, beard). We choose Gender as the bias attribute, and Blonde hair and Attractive as the outputs following [7, 8]. Adult Income Dataset is a publicly available dataset in the UCI repository [72] based on the 1994 U.S. census data. The dataset records an individual's income (more or less than $50,000 per year) along with features such as occupation, marital status, and education. In our experiment, we choose gender and race as biased attributes following [73, 74]. We follow the pre-processing procedures in [75]. As for the experiment on the language model, we use StereoSet [76] as our test set. StereoSet is a large-scale natural dataset to measure stereotypical biases in gender, profession, race, and religion.

Baselines. For the sanity check experiment on a toy Colored MNIST dataset, we use a vanilla logistic regression model as the baseline. For experiments with deep networks, we compare our method with one pre-processing baseline Reweigh [77], 6 in-processing debiasing baselines (LDR [25], LfF [78], Rebias [79], DRO [7], SenSEI [80], and SenSR [81]) and 4 post-processing baselines (EqOdd [35], CEqOdd [35], Reject [82] and PP-IF [83]). We compare our method on language model with five debiasing baselines: Counterfactual Data Augmentation (CDA) [84], Dropout [85], Iterative Null-space Projection (INLP) [86], Self-debias [87], and SentenceDebias [88]. Details can be referred to Appendix C.

Construction of Counterfactual Dataset D ex . We separately construct counterfactual sets for the three datasets, while bias-aligned samples in the small dataset D ex are all split from the test set. In the Colored MNIST dataset, we randomly add another color on the same digit image as the counterfactual sample. As for the Adult dataset, we flip the protected attribute to the opposite while keeping other attributes and target labels exactly the same. For the CelebA dataset, we select images with the same target labels but the opposite protected attribute. To fulfill the request for counterfactual, we rank the similarity between images by comparing the overlap of other attributes and choose the most similar pair to form the factual and counterfactual samples. Part of the generated sample pairs is visualized in Fig. 2. Note that for the CelebA dataset, the counterfactual data are not that strict as the gender attribute is not independent of other features in the natural human facial images. We use Crows-Pairs [89] as our external dataset for the language model. Each sample in Crows-Pairs consists of two sentences: one that is more stereotyping and another that is less stereotyping, which can be utilized as counterfactual pairs.

a

3

5

7

2

color

gender

a

3

7

2

ARD

(a)Colored MNIST

(b)CelebA

Age

Workclass

Education

Education-num

Marital-status

Occupation

Race

Sex

Capital-gain

Hours/week

Native-country

Label

65

Private

HS-grad

9

Married

Machine-op-inspct

White

Male

6418

40

United-States

>50K.

race

65

Private

HS-grad

6

Married

Machine-op-inspct

Black

Male

6418

40

United-States

>50K.

(c)Adult

Implementation details. We use multi-layer perceptron (MLP) with three hidden layers for Colored MNIST and Adult, and ResNet-18 [90] for CelebA following the setting in [8]. During training, we set the batch size of 256 for Colored MNIST and Adult, respectively, and 64 for CelebA following [25, 78, 7]. We use pre-trained BERT [91] and GPT-2 [92], provided by Huggingface. During unlearning, we freeze the parameters of all other layers except the last classifier layer. The running time of all baselines is evaluated on a single RTX3090 GPU for a fair comparison. In our experiment, we select the number of samples k=5000 for Colored MNIST, and k=200 for both Adult and CelebA. The bias threshold is set to 0.

4.2 Sanity Check on Logistic Regression with a Toy Dataset

We conduct an experiment on a logistic regression task to illustrate our method. We simplify the Colored MNIST classification task to a binary classification problem of distinguishing between only digits 3 and 8, on a training set with a bias ratio of 0.95. and a balanced test set. We trained a regularized logistic regressor: argmin w ∈ R d ∑ n i =1 l ( w T x i , y i ) + λ ∥ w ∥ 2 2 . Fig. 5(a) illustrates the classification results of the vanilla regressor on part of test samples. We denote Digit by shape (triangle and rectangle) and Color by color (red and blue). The solid line represents the learned classification boundary and the dotted line represents the expected classification boundary. The test accuracy is 0 . 6517 and it can be observed that most bias-conflict samples tend to be misclassified according to their colors. Moreover, we select and visualize the most helpful and harmful samples in Fig. 5(c) based on Eq. 6. We found that the most helpful samples are in the 5% bias-conflict samples while harmful samples are bias-aligned samples. The unlearning curve is provided in Fig. 5(b). With only 50 samples, the accuracy is improved amazingly by 25.71% and the counterfactual bias decreases by 0.2755, demonstrating the effectiveness of our method.

(a)

(b)

(c)

Digit3_Color8

Digit8_Color8

Digit3_Color3

Digit8_Color3

90

Counterfactual Bias

SE'0

0.40

3

8

3

8

(%) Axeun33y

S8

0.30

ounterfactual

e0

75

00-

70

0.15

8

3

3

3

CO

D

10

20

teration

30

40

50

0.10

4.3 Experiment on Deep Models

Results on Colored MNIST. Tab. 1 shows the comparisons on the Colored MNIST dataset. We reported test accuracy, counterfactual bias, debiasing time, and the number of samples used for all methods. Our approach demonstrates competing performance on accuracy and superior performance on bias compared with retraining baselines. Meanwhile, we only make use of one-tenth of unlearning samples and reduce the debiasing time by 1-2 magnitudes.

Results on Adult. The results are in Table 2. It can be observed that the vanilla method performed the best in accuracy on both tasks, since in the real-world dataset, race and gender are biased w.r.t income in both training and test set and the well-trained model fits this correlation. However, to achieve fair prediction, we would not expect biased attributes to dominate predictions. Compared with other debiasing methods, our method achieved the best results in both accuracy and bias, with much less debiasing time on a smaller dataset.

Results on CelebA. We compare on average accuracy (Avg.), Unbiased accuracy [8] (Unb.) tested on the balanced test set, and Worst-group accuracy [7] (Wor.) tested on the unprivileged group to illustrate the performance, as reported in Tab. 3. It can be observed that the vanilla model performs well on the whole dataset (Avg.) but scores a really low accuracy (Wor.) on the worst group, which means the learned model heavily relies on the bias attribute to achieve high accuracy. Our method obviously bridges this gap and outperforms all other debiasing baselines on Wor. and Unb. in the two experiments. The experiments also demonstrate that our method is consistently feasible even in the absence of perfect standardized counterfactual samples in real-world datasets, by selecting a certain amount of approximate counterfactual data.

Results on Large Language Models (LLM). We further extended our method to the LLM debiasing scenario. Results are presented in Tab. 4. We report two metrics: Language Modeling Score (LMS) measures the percentage of instances in which a language model prefers the meaningful over meaningless association. The LMS of an ideal language model is 100 (the higher the better). Stereotype Score

Attr.

Method

Unb.(%)

↑

Wor.(%)

↑

Avg.(%)

↑

↓

Time(s)

Blonde

Vanilla

66.27

47.36

94.90

0.4211

-

LfF

84.33

81.24

93.52

0.2557

67,620

LDR

85.01

82.32

86.67

0.3126

24,180

DRO

85.66

84.36

92.90

0.3206

28,860

Ours

89.73

87.15

93.41

0.0717

191

Attractive

Vanilla

63.17

40.59

77.42

0.3695

-

LfF

67.44

52.25

77.24

0.2815

67,560

LDR

68.14

54.47

81.70

0.2986

24,420

DRO

66.14

62.33

78.35

0.3004

30,540

Ours

72.18

68.16

80.99

0.1273

187

(SS) measures the percentage of examples in which a model prefers a stereotypical association over an anti-stereotypical association. The SS of an ideal language model is 50 (the closer to 50 the better). It shows that our method can outperform or achieve comparable performance with baseline methods. As for BERT, our method reaches the best (denoted by bold ) or second best (denoted by underline) performance in 5 of 6 metrics. Description of baselines can be referred to Appendix. C.2.

4.4 Analysis

Effectiveness on Different Bias Metrics. We validate the generalization ability of our unlearning method based on different fairness metrics on the Colored MNIST with bias severity 0 . 99 . In Tab. 5, we compare the performance of unlearning harmful samples based on three different biases: Counterfactual bias (Co.), Demographic parity bias (De.) [34], and Equal opportunity bias (Eo.) [35]. For each experiment, we report the changes in three biases. We can note that our method is consistently effective on all three bias metrics. Meanwhile, our counterfactual-based unlearning can significantly outperform the other two in terms of accuracy, Co., and De., and is comparable with them on Eo..

Effectiveness of Unlearn Strategies. We empirically investigate the feasibility of the unlearning mechanism on training and external samples on the Colored MNIST with bias severity 0 . 99 . In Tab. 6, we report the results of unlearning harmful training samples (Eq. 7), unlearning by replacing harmful

samples with their bias-conflict helpful samples (Eq. 21) and unlearning with external counterfactual sample pairs (Eq. 23). It can be observed that unlearning in the training dataset can achieve higher accuracy and less bias, and Eq. 21 excels on both metrics. But unlearning with training samples requires much more time and training samples might not be available in practice, while unlearning with external samples provides a satisfactory alternative.

Discussion on Post-processing Methods. We compare our method to post-processing methods, i.e., Equalized Odds Post-processing (EqOdd) [35], Calibrated Equalized Odds Post-processing (CEqOdd) [35] and Reject Option Classification (Reject) [82], as shown in Tab. 7. Note these methods only apply to logistic regression. Our method outperforms them in most cases on Adult. It is also worth noting that these post-processing methods aimed at a specific fairness measure tend to exacerbate unfairness under other measures while our method consistently improves the fairness under different measures.

Ablation on the Number of Samples. Fig. 4 demonstrates the sensitivity of our unlearning performance w.r.t. number of samples on Colored MNIST with a bias ratio of 0.99. The accuracy increases and bias decreases incrementally with more samples, and becomes steady after the number is beyond 5,000. On the other hand, the unlearning time increases linearly with the number of samples. Additionally, constructing a large number of counterfactual samples in practice might be time-consuming as well. Practical usage of the FMD would require a trade-off based on utility requirements.

Ablation on Number of Fine-tuning Layers. We explore the impact of unlearning different numbers of layers (i.e., the last (one), two, three MLP) on the Color MNIST, with results in Tab. 11. Interestingly, the accuracy excels with two layers but decreases with three layers. Additionally, fine-tuning multiple layers takes much longer time on computation on much more parameters. It is also worth noting that our method could achieve such superior or competing performance even only by updating the last layer in deep models, which calls for more in-depth analysis in the future.

5 Conclusion and Limitation

Biased behaviors in contemporary well-trained deep neural networks can perpetuate social biases, and also pose challenges to the models' robustness. In response, we present FDM, an all-inclusive framework for fast and effective model debiasing. We explicitly measure the influence of training samples on bias measurement and propose a removal mechanism for model debiasing. Comprehensive experiments on multiple datasets demonstrate that our method can achieve superior/competing accuracies with a significantly lower bias as well as computational cost.

Our work preliminarily explored the application of our method to large language models, as well as more analysis on model fairness from different perspectives, which will be in our future plan. In addition, our method is not applicable to black-box models, which are of high interest in real-world scenarios. Our proposed method requires generating counterfactual pairs with labeled sensitive attributes, while many datasets do not have enough labels. Research on fairness with few/no attribute labels is still in the infant stage [93], and we will further explore it.

A Influence Function on Bias and Extension to DNN.

A.1 Deriving the Influence Function on Bias.

In this part, we provide detailed derivation of the influence function on bias in Eq. 5 in the main work. We first start from the influence function on parameters, we can also be referred to [32, 64].

Assuming there are n training samples z 1 , z 2 ..., z n , where z i = ( x i , y i ) , and let L ( z, θ ) represent the loss function of sample z under the model parameters θ , then the trained ˆ θ is given by:

Study the impact of changing the weight of a training sample z on the model parameters θ . If we increase the weight of this sample z in the training set by ϵ , then the perturbed parameters ˆ θ ϵ,z obtained according to ERM (Empirical Risk Minimization) will be:

Define the parameter change ∆ ϵ = ˆ θ ϵ,z -ˆ θ , and note that, as ˆ θ doesn't depend on ϵ , the quantity we seek to compute can be written in terms of it:

Since ˆ θ ϵ,z is a minimizer of R ( θ ) , therefore it satisfies the first-order derivative condition, which means the first-order derivative with respect to θ is zero:

Next, since ˆ θ ϵ,z → ˆ θ as ϵ → 0 , we perform a Taylor expansion of the right-hand side:

where we have dropped o ( ∥ ∆ ϵ ∥ ) terms. Solving for ∆ ϵ , we get:

Since ˆ θ minimizes R , we have ∇ R ( ˆ θ ) = 0 . Dropping o ( ϵ ) terms, we have

Note that it is assumed that R is twice-differentiable and strongly convex in θ . we define:

exists and is positive definite. This guarantees the existence of H -1 ˆ θ . The final influence function can be written as:

Considering B ( ˆ θ ) measured on any A with any D ex , our goal is to quantify how each training point z in the training set D tr contributes to B ( ˆ θ ) . We apply the chain rule on Eq. 18:

Intuitively, this equation can be understood in two parts: the latter part calculates the impact of removing z on the parameters. The former part corresponds to the derivative of bias with respect to parameters, assessing how changes in parameters affect the bias. Hence, this equation quantifies the influence of removing z on the bias.

A.2 Influence at Non-Convergence

In this part, we provide the theoretical proof of the feasibility of the influence function for deep networks (non-convergent) in [32]. In the derivation of the influence function, it's assumed that ˆ θ could be the global minimum. However, if ˆ θ is obtained in deep networks trained with SGD in a non-convex setting, it might be a local optimum and the exact influence can hardly be computed. Here we provide the proof in [32] on how can influence function approximate the parameter change in deep networks.

Consider a training point z . When the model parameters ˜ θ are close to but not at a local minimum, I up,params ( z ) is approximately equal to a constant (which does not depend on z ) plus the change in parameters after upweighting z and then taking a single Newton step from ˜ θ . The high-level idea is that even though the gradient of the empirical risk at ˜ θ is not 0, the Newton step from ˜ θ can be decomposed into a component following the existing gradient (which does not depend on the choice of z ) and a second component responding to the upweighted z (which I up,params ( z ) tracks).

̸

Let g def = 1 n ∑ n i =1 ∇ θ L ( z i , ˜ θ ) be the gradient of the empirical risk at ˜ θ ; since ˜ θ is not a local minimum, g = 0 . After upweighting z by ϵ , the gradient at ˜ θ goes from g ↦→ g + ϵ ∇ θ L ( z, ˜ θ ) , and the empirical Hessian goes from H ˜ θ ↦→ H ˜ θ + ϵ ∇ 2 θ L ( z, ˜ θ ) . A Newton step from ˜ θ therefore changes the parameters by:

Ignoring terms in ϵg , ϵ 2 , and higher, we get N ϵ,z ≈ -H -1 ˜ θ ( g + ϵ ∇ θ L ( z, ˜ θ ) ) . Therefore, the actual change due to a Newton step N ϵ,z is equal to a constant -H -1 ˜ θ g (that doesn't depend on z ) plus ϵ times I up,params ( z ) = -H -1 ˜ θ ∇ θ L ( z, ˜ θ ) (which captures the contribution of z ).

B Bias Removal via Machine Unlearning

B.1 A Closer Look at the Toy Experiment

We conduct an experiment on a logistic regression task using Eq. 19. We simplify the Colored MNIST classification task to a binary classification problem of distinguishing between only digits 3 and 8, on a training set with a bias ratio of 0.95, 0.9 and 0.8, and a balanced test set. To be specific, a bias ratio of 0.95 means 95% bias-aligned samples <digit3_color3, digit8_color8> and 5% bias-conflicting samples <digit3_color8, digit8_color3> in the training set. We trained a regularized logistic regressor: argmin w ∈ R d ∑ n i =1 l ( w T x i , y i ) + λ ∥ w ∥ 2 2 . Fig. 5 (a) illustrates the classification results of the vanilla classifier (trained on the 0.95-biased train set) on part of test samples. We denote Digit by shape (triangle and rectangle) and Color by color (yellow and green). The solid line represents the learned classification boundary and the dotted line represents the expected classification boundary. It can

be observed that the learned classifier tends to classify digits according to their color. Based on the observed bias, we employ Eq. 19 to evaluate how each training sample contributes to the bias. In Fig. 5(b), we select and visualize the most helpful (reduce bias) and harmful (increase bias) samples. We found that the most harmful samples are bias-aligned while helpful samples are bias-conflicting. With this inspiration, We further visualize the influence distribution of training samples in Fig. 6. We denote the bias-conflicting sample with 'red dot' and the bias-aligned sample with 'blue dot'. We find that most bias-aligned samples tend to be harmful while bias-conflicting samples tend to be helpful. This pattern is consistent across different ratios of bias-conflicting samples. Additionally, the influences of helpful samples are larger than those of harmful ones. Visualizations are produced with randomly 500 samples from the training set.

(a)

(b)

Digit8_Color_8

Digit3 Color 3

3

8

38

8

3

Digit8_Color_3

Digit3_Color_8

8333

83

B

(a)

(b)

(c)

0.0030

0.0030

0.0025

Bias-conflicting Samples

Bias-align Samples

0.0025

Bias-conflicting Samples

Bias-align Samples

0.0025

Bias-conflicting Samples

Bias-align Samples

0.0020

0.0020

0.0015

0.0015

0.0015

0.0010

0.0010

0.0010

0.0005

0.0005

0.0005

0.0000

0.0000

0.0000

0.0005

0.0005

0.0005

0.0010

0

100

200

300

400

500

0

100

200

300

400

500

0.0010

0

100

200

300

400

500

Id of sample

Id of sample

Id of sample

Inspired by this observation, our unlearning strategy is further refined. Hence, we propose a straightforward solution that further mitigates the influence of a harmful sample with a bias-conflicting sample. Consequently, we update the parameters to unlearn the harmful samples by:

where ¯ z k denotes the bias-conflicting sample of z k . Following the explanation in influence theory [32], our unlearn mechanism removes the effect of perturbing a training point (¯ a, x, y ) to ( a, x, y ) . In other words, we not only remove the influence caused by harmful sample z k , but further ensure fairness with the corresponding counterfactual sample ¯ z k .

To further illustrate the functionality of Eq. 21, we measure the influences of the selected harmful and helpful sample pairs by:

with visualizations in Fig. 7. By calculating the difference between the harmful samples and helpful samples, the biased effect is significantly amplified. In this way, the unlearning becomes more effective.

B.2 Deriving Alternative Efficient Unlearn

In the above sections, the unlearning process is based on the assumption that we could access the original training sample z k to identify and evaluate biases and then forget them. However, in practice,

(a)

(b)

(c)

0.003

0.003

0.003

0.002

Bias-conflicting Samples

Bias-align Samples

Counterfactual Pairs

0.002

0.002

Bias-conflicting Samples

Bias-align Samples

Counterfactual Pairs

0.001

0.001

1000

0.000

L2

0.000

1

0.000

0.001

0.001

0.001

0.002

0.002

Bias-conflicting Samples

Bias-align Samples

0.002

Counterfactual Pairs

0.003

0

100

200

300

400

500

0.003

0

100

200

300

400

500

0.003

0

100

200

300

400

500

Id of sample

Id of sample

Id of sample

the training set might be too large or even unavailable in the unlearning phase. In response, we further propose to approximate the unlearning mechanism with a small external dataset. As the influence to be removed can be obtained from the change of the protected attribute, we can construct the same modification to the protected attribute on external samples. In particular, we employ an external dataset D ex as in Section 3.1 in the main work to construct counterfactual pairs for unlearning, which redefines Eq. 21 as:

As D ex can be easily obtained from an external dataset rather than the training set, e.g., the test set, the practical applicability of our method could be significantly enhanced.

We further visualize the influence of samples in the balanced external dataset in Fig. 8 (a). In the balanced dataset, the ratio of bias-aligned and bias-conflicting samples is about 50%. We can observe that the pattern of harmful bias-aligned samples and helpful bias-conflicting samples in the external dataset is similar to the training set. By comparing the influence of counterfactual pairs in the external dataset (Fig. 8 (b)) and the training set (Fig. 8 (c)), we can find the distributions are similar, which proves the feasibility of our alternative unlearning.

(a)

(b)

(c)

0.006

0.006

0.003

Bias-conflicting Samples

Bias-conflicting Samples

0.005

Bias-align Samples

0.004

Bias-align Samples

Counterfactual Pairs

0.002

0.004

0.002

0.001

000

V

0.000

V

=

0.000

0.002

0.001

0.002

0.001

0.000

0.004

0.002

Bias-conflicting Samples

Bias-align Samples

0.001

0

100

200

300

400

500

0.006

0

100

00Z

300

400

500

0.003

0

Counterfactual Pairs

100

200

300

400

500

Id of sample

Id of sample

Id of sample

B.3 Alternative Efficient Unlearn vs. Directly Unlearn Training Data.

Tackling the problem that, in practice, the training set might be too large or even unavailable in the unlearning phase, we propose an alternative unlearning strategy in Sec. 3.3 in the main work. We approximate the change of the protected attribute by constructing the same modification to the protected attribute on external samples. Then we unlearn the same perturbation from the model with the constructed external dataset. In Sec. 4.4 in the main work, we provide the performance comparison of alternative efficient unlearn (Ours) and directly unlearn training data (Eq. 7 and Eq. 8 in the main work).

In this section, we further compare the performance of alternative unlearning on Adult and simplified Colored MNIST on logistic regression, with results reported in Tab. 9 and Tab. 10. We can find that in five experiments, alternative learning achieves comparable performance with the two directly

unlearning strategies. Comparing Eq. 7 and Eq. 8, we can find that the modified Eq. 8 reaches convergence taking less iteration. The number of samples used is 200 for the two datasets.

B.4 Efficient Unlearning for Deep Networks.

In our experiment, We are inspired by Sec. 5.1 and Sec. 5.2 in [32] which keep all but the top layer in deep networks frozen and measure influence. We follow this setting so that the finetuning on deep networks can be simplified as logistic regression. In this part, we investigate the difference in finetuning different numbers of layers. The experiment is conducted on Colored MNIST with MLP with 3 hidden layers.

Discussion on Different Fine-tuning Strategies. Following Sec. 4.4 in the main work, we explore the impact of unlearning different numbers of layers (i.e., the top one, two, three MLP) on the Colored MNIST with three bias ratios, with results in Tab. 11. Interestingly, the accuracy excels with two layers but decreases with three layers. Additionally, fine-tuning multiple layers takes much longer time on computation on much more parameters. It is also worth noting that our method could achieve such superior or competing performance even only by updating the last layer in deep models, which calls for more in-depth analysis in the future.

B.5 Effectiveness of Pre-calculating Hessian.

In Sec. 3.4 in the main work, we propose to pre-calculate the inverse Hessian before performing unlearning. In this way, we approximate the Hessian as it should change with model parameters, however, we prevent the large computation cost of updating and inverting the Hessian at every iteration. In this part, we empirically illustrate the effectiveness of our approximation. Experiments are conducted on Colored MNIST and Adult datasets with logistic regression tasks, with results provided in Tab. 12 and Tab. 13. "wo/" denotes unlearning without pre-calculation. It can be observed that unlearning with or without can achieve comparative performance on bias and accuracy. However, our method can save about 40% run time on Adult and 97% run time on Colored MNIST. The reason is that the number of parameters for Colored MNIST is much larger than Adult, so that the calculation of inverse Hessian makes up a larger proportion of the total run time.

C Experiment Details

C.1 Dataset

Colored MNIST. Colored MNIST is constructed based on the MNIST dataset [68] designed for digit classification tasks. To build a biased correlation, ten distinct RGB values are applied on grayscale digit images [3, 69, 70]. Digit and color distribution are paired to build biased correlations in the training set. Bias-aligned samples are defined as fixed combinations of digit and color like Digit 1, Color 1 while bias-conflict samples are defined as other combinations like Digit 1, random Color in 2-10. In our Experiment, we use 3 different training sets by setting different bias ratios 0.995, 0.99, 0.95 for biased-aligned training samples where the ratio represents the partition of bias-aligned samples in the training set. The higher the ratio, the higher the degree of bias. The split of the training set, test set, and external set is 60000, 10000, and 10000.

CelebA. CelebA dataset [71] is a face recognition with 40 types of attributes like gender, age (young or not), and lots of facial characteristics (such as hair color, smile, beard). The dataset contains a total of 202,599 images which, following the official train validation split, consists of 162,770 images for training and 9,867 images for testing. We choose Gender as the protected attribute, Hair-color (blonde hair or not) and Attractive as the target attribute following [7, 8]. The number of selected samples for the two target attributes is 200 and 182, which are split from the test set.

Adult Income Dataset. The Adult dataset is a publicly available dataset in the UCl repository [72] based on 1994 U.S. census data. The goal of this dataset is to successfully predict whether an individual earns more or less than $50,000 per year based on features such as occupation, marital status, and education. We follow the processing procedures in [41]. In our experiment, we choose gender and race as protected attributes following [73, 74]. We split 200 samples from the test set as the external dataset.

C.2 Baselines

For the sanity check experiment on a toy Colored MNIST dataset, we use a vanilla logistic regression model as the baseline. For experiments with deep networks, we compare our method with one pre-processing baseline Reweigh [77], 6 in-processing debiasing baselines (LDR [25], LfF [78], Rebias [79], DRO [7], SenSEI [80], and SenSR [81]) and 4 post-processing baselines (EqOdd [35], CEqOdd [35], Reject [82] and PP-IF [83]). [77] utilizes the influence function to reweight the training sample, in order to re-train a fair model targeting group fairness metrics (equal opportunity and demographic parity). Among in-processing baselines, LDR, LfF, Rebias, and DRO are designed explicitly to target higher accuracy (on unbiased test set or worst-group test set) and implicitly target fairness, while SenSEI and SenSR are designed to target individual fairness. EqOdd, CEqOdd and Reject are designed to target different group fairness metrics (equal odd and demographic parity), while [83] proposes a post-processing algorithm for individual fairness.

Baselines for experiment on Large Language Model. We evaluate several baseline debiasing methods. Counterfactual Data Augmentation (CDA) [94] adjusts a corpus for balance by exchanging words indicative of bias (such as 'he/she') within the dataset. This newly balanced corpus is then typically utilized for additional model training to reduce bias. Dropout [95] suggests enhancing dropout rates and incorporating an extra pre-training stage for debiasing. SentenceDebias [88] aims to derive unbiased representations by removing biased projections on a presumed bias subspace from the original sentence representations. Iterative Nullspace Projection (INLP) [86] employs a projection-based approach to exclude protected attributes from representations. Finally, SelfDebias [87] advocates for the use of a model's intrinsic knowledge to avert the generation of biased text.

D Discussion

D.1 Dataset Generation

In our experiments, we utilize approximated counterfactual samples for CelebA due to the unavailability of strict counterfactual data. Based on attribute annotations, we select images with the same target attributes but opposite sensitive attributes, while maintaining other attributes as much as possible. Our method achieves the best results on the worst-case group, indicating that the approximated counterfactual samples can also effectively enhance fairness in predictions. Similar to our approach, [96] proposes to select pairs of counterfactual images based on attribute annotations on the CUB dataset to produce counterfactual visual explanations. Their experiments also show that neural networks can discern major differences (such as gender in our work) between images without strict control (such as background).

For real-world visual datasets (like facial dataset or ImageNet), the unavailability of strict counterfactual data is a common challenge. Existing methods propose to train a generative model to create counterfactual images with altered sensitive attributes [97-99], which seems to be a viable approach for obtaining counterfactual datasets for more diverse vision applications. Building upon these methods, we will extend our approach to more scenarios.

D.2 Influence Estimation

In our unlearning experiment, we freeze the parameters of all other layers except the top layer. Previous work investigates the estimation accuracy of the influence function on both multi-layer and single-layer setups [100]. It performs a case study on the MNIST. For each test point, they select 100 training samples and compute the ground-truth influence by model re-training. Results show that estimations are more accurate for shallow networks.

Our results in Tab. 7 in the main manuscript also validate this point. When applying FMD to a three-layer neural net, the performance on either accuracy or bias becomes worse. This could potentially be attributed to the inaccurate estimation of influence function on multi-layer neural nets. In our experiments, we adhere to the set-up in [32], where the influence function is only applied to the last layer of deep models, which proves to be effective.

As verified in [32, 100, 101], influence estimation matches closely to leave-one-out retraining for logistic regression model. As discussed in [97], measuring influence score for the last layer can be regarded as calculating influence from a logistic regression model on the bottleneck features (Sec. 5.1 in the main manuscript). The same setup is followed by many influence function-based works [102, 103] and proves to be effective.

D.3 Computational Complexity

As for bias-effect evaluation, with n training points and θ ∈ R d , directly computing Eq. 5 (in the main manuscript) requires O ( nd 2 + nd 3 ) operations. In our experiment, we only activate the last layer so that d is small. However, when the number of training samples is very large, performing Eq. 5 is expensive. As for the debiasing phase, it requires O ( nd 2 + kd 2 ) operations, where k is the number of samples to unlearn. Note that if hessian is calculated in the bias-effect evaluation phase, it can be directly used in the debiasing phase. Hence, the overall computational complexity using Eq. 7 and Eq. 8 is O ( nd 2 + kd 2 + nd 3 ) .

However, in our proposed alternative debiasing method, we only utilize an external counterfactual dataset with a small number of k. Hence, we can omit the O ( nd 3 ) operations to compute influences and rank the training samples. Hence, the overall computational complexity using Eq. 9 (Ours) is O ( nd 2 + kd 2 ) . Experimental comparison results can be referred to Tab. 5 (in the main manuscript). Debiasing with Eq. 8 takes about 500x more time than Eq. 9 (in the main manuscript).

E Preliminaries

E.1 Influence Function

The origins of influence-based diagnostics can be traced back to important research papers such as [64, 104, 105]. More recently, Koh and Liang [32] introduced the concept of influence functions to large-scale deep learning, which numerous publications have since followed up. In their work, [32] advocated for the use of an approximation, Eq. 13, to estimate the change in loss when a small adjustment is made to the weights of the dataset. In practical applications involving deep models, the Hessian matrix (H) cannot be stored in memory or inverted using standard linear algebra techniques. However, by considering a fixed vector (v), the Hessian vector product (HVP), Hv, can be computed in O(bp) time and memory [106], where b represents the batch size and determines the number of training examples used to approximate H (for a given loss function L). The iterative procedure LISSA [107], employed by [32], relies on repeated calls to the HVP to estimate the inverse HVP.

E.2 Counterfactual Fairness

Counterfactual fairness, a relatively new concept, has emerged as a means to measure fairness at an individual level [31]. The fundamental idea behind this approach is to determine the fairness of a decision for an individual by comparing it with the decision that would have been made in an alternate scenario where the individual's sensitive attributes possessed different values. This concept builds upon earlier work [108], which introduced a causal framework for learning from biased data by examining the relationship between sensitive features and the data. Recent advancements in deep learning have further contributed to this field, with novel approaches [46, 45, 109, 98] proposing methods to enhance the accuracy of decision-making models by improving the approximation of causal inference, particularly when dealing with unobserved confounding variables.

E.3 Demographic Parity and Equal Opportunity

Demographic Parity [34]: A predictor Y satisfies demographic parity if P ( Y | A = 0) = P ( Y | A = 1) , where A is the sensitive attribute. The likelihood of a positive outcome should be the same regardless of whether the person is in the protected (e.g., female) group.

Equal Opportunity [35]: 'A binary predictor Y satisfies equal opportunity with respect to A and Y if P ( Y = 1 | A = 0 , Y = 1) = P ( Y = 1 | A = 1 , Y = 1) ' . This means that the probability of a person in a positive class being assigned to a positive outcome should be equal for both protected and unprotected (female and male) group members.

References

M. Du, F. Yang, N. Zou, and X. Hu, 'Fairness in deep learning: A computational perspective,' IEEE Intelligent Systems , vol. 36, no. 4, pp. 25-34, 2020.

N. Mehrabi, F. Morstatter, N. Saxena, K. Lerman, and A. Galstyan, 'A survey on bias and fairness in machine learning,' ACM Computing Surveys (CSUR) , vol. 54, no. 6, pp. 1-35, 2021.

B. Kim, H. Kim, K. Kim, S. Kim, and J. Kim, 'Learning not to learn: Training deep neural networks with biased data,' in IEEE Conference on Computer Vision and Pattern Recognition , 2019.

R. Geirhos, J.-H. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge, and F. A. Wichmann, 'Shortcut learning in deep neural networks,' Nature Machine Intelligence , vol. 2, no. 11, pp. 665-673, 2020.

S. Sagawa, A. Raghunathan, P. W. Koh, and P. Liang, 'An investigation of why overparameterization exacerbates spurious correlations,' in International Conference on Machine Learning , pp. 8346-8356, PMLR, 2020.

R. Geirhos, P. Rubisch, C. Michaelis, M. Bethge, F. A. Wichmann, and W. Brendel, 'Imagenettrained cnns are biased towards texture; increasing shape bias improves accuracy and robustness,' arXiv preprint arXiv:1811.12231 , 2018.

S. Sagawa, P. W. Koh, T. B. Hashimoto, and P. Liang, 'Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization,' in International Conference on Learning Representations , 2020.

E. Tartaglione, C. A. Barbano, and M. Grangetto, 'End: Entangling and disentangling deep representations for bias correction,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 13508-13517, June 2021.

S. Sagawa, A. Raghunathan, P. W. Koh, and P. Liang, 'An investigation of why overparameterization exacerbates spurious correlations,' in International Conference on Machine Learning , pp. 8346-8356, PMLR, 2020.

T. Brennan, W. Dieterich, and B. Ehret, 'Evaluating the predictive validity of the compas risk and needs assessment system,' Criminal Justice and behavior , vol. 36, no. 1, pp. 21-40, 2009.

J. F. Mahoney and J. M. Mohen, 'Method and system for loan origination and underwriting,' Oct. 23 2007. US Patent 7,287,008.

M. Bogen and A. Rieke, 'Help wanted: An examination of hiring algorithms, equity, and bias,' 2018.

D. Pessach and E. Shmueli, 'A review on fairness in machine learning,' ACM Computing Surveys (CSUR) , vol. 55, no. 3, pp. 1-44, 2022.

O. Parraga, M. D. More, C. M. Oliveira, N. S. Gavenski, L. S. Kupssinskü, A. Medronha, L. V. Moura, G. S. Simões, and R. C. Barros, 'Debiasing methods for fairer neural models in vision and language research: A survey,' arXiv preprint arXiv:2211.05617 , 2022.

Y. Li and N. Vasconcelos, 'Repair: Removing representation bias by dataset resampling,' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pp. 9572-9581, 2019.

M. Tanjim, R. Sinha, K. K. Singh, S. Mahadevan, D. Arbour, M. Sinha, G. W. Cottrell, et al. , 'Generating and controlling diversity in image search,' in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pp. 411-419, 2022.

F. Calmon, D. Wei, B. Vinzamuri, K. Natesan Ramamurthy, and K. R. Varshney, 'Optimized pre-processing for discrimination prevention,' Advances in neural information processing systems , vol. 30, 2017.

M. Feldman, S. A. Friedler, J. Moeller, C. Scheidegger, and S. Venkatasubramanian, 'Certifying and removing disparate impact,' in proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining , pp. 259-268, 2015.

F. Kamiran and T. Calders, 'Data preprocessing techniques for classification without discrimination,' Knowledge and information systems , vol. 33, no. 1, pp. 1-33, 2012.

M. B. Zafar, I. Valera, M. Gomez Rodriguez, and K. P. Gummadi, 'Fairness beyond disparate treatment & disparate impact: Learning classification without disparate mistreatment,' in Proceedings of the 26th international conference on world wide web , pp. 1171-1180, 2017.

E. Adeli, Q. Zhao, A. Pfefferbaum, E. V. Sullivan, L. Fei-Fei, J. C. Niebles, and K. M. Pohl, 'Representation learning with statistical independence to mitigate bias,' in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pp. 2513-2523, 2021.

P. Dhar, J. Gleason, A. Roy, C. D. Castillo, and R. Chellappa, 'Pass: protected attribute suppression system for mitigating bias in face recognition,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , pp. 15087-15096, 2021.

M. Alvi, A. Zisserman, and C. Nellåker, 'Turning a blind eye: Explicit removal of biases and variation from deep neural network embeddings,' in Proceedings of the European Conference on Computer Vision (ECCV) Workshops , pp. 0-0, 2018.

H. Bahng, S. Chun, S. Yun, J. Choo, and S. J. Oh, 'Learning de-biased representations with biased representations,' in International Conference on Machine Learning , pp. 528-539, PMLR, 2020.

J. Lee, E. Kim, J. Lee, J. Lee, and J. Choo, 'Learning debiased representation via disentangled feature augmentation,' Advances in Neural Information Processing Systems , vol. 34, pp. 2512325133, 2021.

T. Wang, J. Zhao, M. Yatskar, K.-W. Chang, and V. Ordonez, 'Balanced datasets are not enough: Estimating and mitigating gender bias in deep image representations,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , pp. 5310-5319, 2019.

M. Hardt, E. Price, and N. Srebro, 'Equality of opportunity in supervised learning,' Advances in neural information processing systems , vol. 29, 2016.

A. K. Menon and R. C. Williamson, 'The cost of fairness in binary classification,' in Conference on Fairness, accountability and transparency , pp. 107-118, PMLR, 2018.

S. Corbett-Davies, E. Pierson, A. Feller, S. Goel, and A. Huq, 'Algorithmic decision making and the cost of fairness,' in Proceedings of the 23rd acm sigkdd international conference on knowledge discovery and data mining , pp. 797-806, 2017.

P. Saleiro, B. Kuester, L. Hinkson, J. London, A. Stevens, A. Anisfeld, K. T. Rodolfa, and R. Ghani, 'Aequitas: A bias and fairness audit toolkit,' arXiv preprint arXiv:1811.05577 , 2018.

M. J. Kusner, J. Loftus, C. Russell, and R. Silva, 'Counterfactual fairness,' Advances in neural information processing systems , vol. 30, 2017.

P. W. Koh and P. Liang, 'Understanding black-box predictions via influence functions,' in International conference on machine learning , pp. 1885-1894, PMLR, 2017.

Z. Cao, J. Wang, S. Si, Z. Huang, and J. Xiao, 'Machine unlearning method based on projection residual,' arXiv preprint arXiv:2209.15276 , 2022.

C. Dwork, M. Hardt, T. Pitassi, O. Reingold, and R. Zemel, 'Fairness through awareness,' in Proceedings of the 3rd innovations in theoretical computer science conference , pp. 214-226, 2012.

M. Hardt, E. Price, and N. Srebro, 'Equality of opportunity in supervised learning,' Advances in neural information processing systems , vol. 29, 2016.

S. Jung, S. Chun, and T. Moon, 'Learning fair classifiers with partially annotated group labels,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 10348-10357, 2022.

S. Verma and J. Rubin, 'Fairness definitions explained,' in Proceedings of the international workshop on software fairness , pp. 1-7, 2018.

M. Kearns, S. Neel, A. Roth, and Z. S. Wu, 'Preventing fairness gerrymandering: Auditing and learning for subgroup fairness,' in International conference on machine learning , pp. 25642572, PMLR, 2018.

C. Dwork, M. Hardt, T. Pitassi, O. Reingold, and R. Zemel, 'Fairness through awareness,' in Proceedings of the 3rd innovations in theoretical computer science conference , pp. 214-226, 2012.

M. Joseph, M. Kearns, J. Morgenstern, S. Neel, and A. Roth, 'Rawlsian fairness for machine learning,' arXiv preprint arXiv:1610.09559 , vol. 1, no. 2, p. 19, 2016.

C. Louizos, K. Swersky, Y. Li, M. Welling, and R. Zemel, 'The variational fair autoencoder,' arXiv preprint arXiv:1511.00830 , 2015.

W. Fleisher, 'What's fair about individual fairness?,' in Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society , pp. 480-490, 2021.

R. Binns, 'On the apparent conflict between individual and group fairness,' in Proceedings of the 2020 conference on fairness, accountability, and transparency , pp. 514-524, 2020.

S. Chiappa, 'Path-specific counterfactual fairness,' in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 33, pp. 7801-7808, 2019.

S. Garg, V. Perot, N. Limtiaco, A. Taly, E. H. Chi, and A. Beutel, 'Counterfactual fairness in text classification through robustness,' in Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society , pp. 219-226, 2019.

Y. Wu, L. Zhang, and X. Wu, 'Counterfactual fairness: Unidentification, bound and algorithm,' in Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence , 2019.

N. Kilbertus, P. J. Ball, M. J. Kusner, A. Weller, and R. Silva, 'The sensitivity of counterfactual fairness to unmeasured confounding,' in Uncertainty in artificial intelligence , pp. 616-626, PMLR, 2020.

F. Kamiran, A. Karim, and X. Zhang, 'Decision theory for discrimination-aware classification,' in 2012 IEEE 12th international conference on data mining , pp. 924-929, IEEE, 2012.

C. Dwork, N. Immorlica, A. T. Kalai, and M. Leiserson, 'Decoupled classifiers for group-fair and efficient machine learning,' in Conference on fairness, accountability and transparency , pp. 119-133, PMLR, 2018.

N. Kallus, X. Mao, and A. Zhou, 'Assessing algorithmic fairness with unobserved protected class using data combination,' Management Science , vol. 68, no. 3, pp. 1959-1981, 2022.

T. Baumhauer, P. Schöttle, and M. Zeppelzauer, 'Machine unlearning: Linear filtration for logit-based classifiers,' Machine Learning , vol. 111, no. 9, pp. 3203-3226, 2022.

Q. P. Nguyen, R. Oikawa, D. M. Divakaran, M. C. Chan, and B. K. H. Low, 'Markov chain monte carlo-based machine unlearning: Unlearning what needs to be forgotten,' arXiv preprint arXiv:2202.13585 , 2022.

A. Tahiliani, V. Hassija, V. Chamola, and M. Guizani, 'Machine unlearning: Its need and implementation strategies,' in 2021 Thirteenth International Conference on Contemporary Computing (IC3-2021) , pp. 241-246, 2021.

M. Magdziarczyk, 'Right to be forgotten in light of regulation (eu) 2016/679 of the european parliament and of the council of 27 april 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing directive 95/46/ec,' in 6th International Multidisciplinary Scientific Conference on Social Sciences and Art Sgem 2019 , pp. 177-184, 2019.

L. Bourtoule, V. Chandrasekaran, C. A. Choquette-Choo, H. Jia, A. Travers, B. Zhang, D. Lie, and N. Papernot, 'Machine unlearning,' in 2021 IEEE Symposium on Security and Privacy (SP) , pp. 141-159, IEEE, 2021.

J. Brophy and D. Lowd, 'Machine unlearning for random forests,' in International Conference on Machine Learning , pp. 1092-1104, PMLR, 2021.

A. Golatkar, A. Achille, and S. Soatto, 'Eternal sunshine of the spotless net: Selective forgetting in deep networks,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 9304-9312, 2020.

A. Mahadevan and M. Mathioudakis, 'Certifiable machine unlearning for linear models,' arXiv preprint arXiv:2106.15093 , 2021.

Y. Wu, E. Dobriban, and S. Davidson, 'Deltagrad: Rapid retraining of machine learning models,' in International Conference on Machine Learning , pp. 10355-10366, PMLR, 2020.

S. Neel, A. Roth, and S. Sharifi-Malvajerdi, 'Descent-to-delete: Gradient-based methods for machine unlearning,' in Algorithmic Learning Theory , pp. 931-962, PMLR, 2021.

Z. Cao, J. Wang, S. Si, Z. Huang, and J. Xiao, 'Machine unlearning method based on projection residual,' arXiv preprint arXiv:2209.15276 , 2022.

Z. Izzo, M. A. Smart, K. Chaudhuri, and J. Zou, 'Approximate data deletion from machine learning models,' in International Conference on Artificial Intelligence and Statistics , pp. 20082016, PMLR, 2021.

C. Guo, T. Goldstein, A. Hannun, and L. Van Der Maaten, 'Certified data removal from machine learning models,' arXiv preprint arXiv:1911.03030 , 2019.

R. D. Cook and S. Weisberg, Residuals and influence in regression . New York: Chapman and Hall, 1982.

X. Han, B. C. Wallace, and Y. Tsvetkov, 'Explaining black box predictions and unveiling data artifacts through influence functions,' arXiv preprint arXiv:2005.06676 , 2020.

A. Peste, D. Alistarh, and C. H. Lampert, 'Ssse: Efficiently erasing samples from trained machine learning models,' arXiv preprint arXiv:2107.03860 , 2021.

B. A. Pearlmutter, 'Fast exact multiplication by the hessian,' Neural computation , vol. 6, no. 1, pp. 147-160, 1994.

Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, 'Gradient-based learning applied to document recognition,' IEEE , vol. 86, no. 11, pp. 2278-2324, 1998.

Y. Li and N. Vasconcelos, 'Repair: Removing representation bias by dataset resampling,' in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pp. 95729581, 2019.

H. Bahng, S. Chun, S. Yun, J. Choo, and S. J. Oh, 'Learning de-biased representations with biased representations,' in International Conference on Machine Learning , 2020.

Z. Liu, P. Luo, X. Wang, and X. Tang, 'Deep learning face attributes in the wild,' in IEEE International Conference on Computer Vision , 2015.

A. Frank, A. Asuncion, et al. , 'Uci machine learning repository, 2010,' URL http://archive. ics. uci. edu/ml , vol. 15, p. 22, 2011.

R. Zemel, Y. Wu, K. Swersky, T. Pitassi, and C. Dwork, 'Learning fair representations,' in International conference on machine learning , pp. 325-333, PMLR, 2013.

T. Kamishima, S. Akaho, and J. Sakuma, 'Fairness-aware learning through regularization approach,' in 2011 IEEE 11th International Conference on Data Mining Workshops , pp. 643650, IEEE, 2011.

R. K. Bellamy, K. Dey, M. Hind, S. C. Hoffman, S. Houde, K. Kannan, P. Lohia, J. Martino, S. Mehta, A. Mojsilovi´ c, et al. , 'Ai fairness 360: An extensible toolkit for detecting and mitigating algorithmic bias,' IBM Journal of Research and Development , vol. 63, no. 4/5, pp. 4-1, 2019.

M. Nadeem, A. Bethke, and S. Reddy, 'Stereoset: Measuring stereotypical bias in pretrained language models,' arXiv preprint arXiv:2004.09456 , 2020.

P. Li and H. Liu, 'Achieving fairness at no utility cost via data reweighing with influence,' in International Conference on Machine Learning , pp. 12917-12930, PMLR, 2022.

J. Nam, H. Cha, S. Ahn, J. Lee, and J. Shin, 'Learning from failure: Training debiased classifier from biased classifier,' in Advances in Neural Information Processing Systems , 2020.

H. Bahng, S. Chun, S. Yun, J. Choo, and S. J. Oh, 'Learning de-biased representations with biased representations,' in International Conference on Machine Learning (ICML) , 2020.

M. Yurochkin and Y. Sun, 'Sensei: Sensitive set invariance for enforcing individual fairness,' arXiv preprint arXiv:2006.14168 , 2020.

M. Yurochkin, A. Bower, and Y. Sun, 'Training individually fair ml models with sensitive subspace robustness,' arXiv preprint arXiv:1907.00020 , 2019.

F. Kamiran, A. Karim, and X. Zhang, 'Decision theory for discrimination-aware classification,' in 2012 IEEE 12th International Conference on Data Mining , pp. 924-929, 2012.

F. Petersen, D. Mukherjee, Y. Sun, and M. Yurochkin, 'Post-processing for individual fairness,' Advances in Neural Information Processing Systems , vol. 34, pp. 25944-25955, 2021.

R. Zmigrod, S. J. Mielke, H. Wallach, and R. Cotterell, 'Counterfactual data augmentation for mitigating gender stereotypes in languages with rich morphology,' arXiv preprint arXiv:1906.04571 , 2019.

K. Webster, X. Wang, I. Tenney, A. Beutel, E. Pitler, E. Pavlick, J. Chen, E. Chi, and S. Petrov, 'Measuring and reducing gendered correlations in pre-trained models,' arXiv preprint arXiv:2010.06032 , 2020.

S. Ravfogel, Y. Elazar, H. Gonen, M. Twiton, and Y. Goldberg, 'Null it out: Guarding protected attributes by iterative nullspace projection,' arXiv preprint arXiv:2004.07667 , 2020.

T. Schick, S. Udupa, and H. Schütze, 'Self-diagnosis and self-debiasing: A proposal for reducing corpus-based bias in nlp,' Transactions of the Association for Computational Linguistics , vol. 9, pp. 1408-1424, 2021.

P. P. Liang, I. M. Li, E. Zheng, Y. C. Lim, R. Salakhutdinov, and L.-P. Morency, 'Towards debiasing sentence representations,' arXiv preprint arXiv:2007.08100 , 2020.

N. Nangia, C. Vania, R. Bhalerao, and S. R. Bowman, 'Crows-pairs: A challenge dataset for measuring social biases in masked language models,' arXiv preprint arXiv:2010.00133 , 2020.

K. He, X. Zhang, S. Ren, and J. Sun, 'Deep residual learning for image recognition,' arXiv preprint arXiv:1512.03385 , 2015.

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, 'Bert: Pre-training of deep bidirectional transformers for language understanding,' arXiv preprint arXiv:1810.04805 , 2018.

A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. , 'Language models are unsupervised multitask learners,' OpenAI blog , vol. 1, no. 8, p. 9, 2019.

S. Seo, J.-Y . Lee, and B. Han, 'Unsupervised learning of debiased representations with pseudoattributes,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 16742-16751, 2022.

R. Zmigrod, S. J. Mielke, H. Wallach, and R. Cotterell, 'Counterfactual data augmentation for mitigating gender stereotypes in languages with rich morphology,' arXiv preprint arXiv:1906.04571 , 2019.

K. Webster, X. Wang, I. Tenney, A. Beutel, E. Pitler, E. Pavlick, J. Chen, E. Chi, and S. Petrov, 'Measuring and reducing gendered correlations in pre-trained models,' arXiv preprint arXiv:2010.06032 , 2020.

Y. Goyal, Z. Wu, J. Ernst, D. Batra, D. Parikh, and S. Lee, 'Counterfactual visual explanations,' in International Conference on Machine Learning , pp. 2376-2384, PMLR, 2019.

S. Dash, V. N. Balasubramanian, and A. Sharma, 'Evaluating and mitigating bias in image classifiers: A causal perspective using counterfactuals,' in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pp. 915-924, 2022.

H. Kim, S. Shin, J. Jang, K. Song, W. Joo, W. Kang, and I.-C. Moon, 'Counterfactual fairness with disentangled causal effect variational autoencoder,' in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 35, pp. 8128-8136, 2021.

J. Cheong, S. Kalkan, and H. Gunes, 'Counterfactual fairness for facial expression recognition,' in European Conference on Computer Vision , pp. 245-261, Springer, 2022.

S. Basu, P. Pope, and S. Feizi, 'Influence functions in deep learning are fragile,' arXiv preprint arXiv:2006.14651 , 2020.

J. Bae, N. Ng, A. Lo, M. Ghassemi, and R. B. Grosse, 'If influence functions are the answer, then what is the question?,' Advances in Neural Information Processing Systems , vol. 35, pp. 17953-17967, 2022.

G. Pruthi, F. Liu, S. Kale, and M. Sundararajan, 'Estimating training data influence by tracing gradient descent,' Advances in Neural Information Processing Systems , vol. 33, pp. 1992019930, 2020.

C.-K. Yeh, J. Kim, I. E.-H. Yen, and P. K. Ravikumar, 'Representer point selection for explaining deep neural networks,' Advances in neural information processing systems , vol. 31, 2018.

R. D. Cook and S. Weisberg, 'Characterizations of an empirical influence function for detecting influential cases in regression,' Technometrics , vol. 22, no. 4, pp. 495-508, 1980.

R. D. Cook and S. Weisberg, Residuals and influence in regression . New York: Chapman and Hall, 1982.

B. A. Pearlmutter, 'Fast exact multiplication by the hessian,' Neural computation , vol. 6, no. 1, pp. 147-160, 1994.

N. Agarwal, B. Bullins, and E. Hazan, 'Second-order stochastic optimization for machine learning in linear time,' The Journal of Machine Learning Research , vol. 18, no. 1, pp. 41484187, 2017.

J. Pearl et al. , 'Models, reasoning and inference,' Cambridge, UK: CambridgeUniversityPress , vol. 19, no. 2, 2000.

S. R. Pfohl, T. Duan, D. Y. Ding, and N. H. Shah, 'Counterfactual reasoning for fair clinical risk prediction,' in Machine Learning for Healthcare Conference , pp. 325-358, PMLR, 2019.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an approach to removing biases from trained models without necessitating full retraining and without necessitating access to a large dataset that has annotations w.r.t biases. The proposed method consists of 3 steps: firstly, identifying which attributes are ‘biased’ (according to a counterfactual definition of bias: the more the predictions differ between the factual and counterfactual examples, the more biased the model is). Second, identify which examples are those contributing to biased behavior, using influence functions. Finally, once the (top K) harmful samples are identified, they run an unlearning procedure (via a Newton step) to remove the influence of those examples.\n\nSTRENGTHS:\n- This paper tackles an important problem and provides useful motivation and intuition for understanding the issues with biases in neural networks\n\n- The proposed approach makes fewer limiting assumptions than some of the previous methods, and the authors show that empirically, it outperforms certain baselines on certain metrics on some (simple) datasets / models\n\n- The paper is well-structured and mostly clear, with some exceptions (see weaknesses section below)\n\nWEAKNESSES:\nA) Clarity can be significantly improved in some sections. Some particular examples:\n\na) Some imprecise statements, e.g. what does it mean to be “closely tied” (line 136)\n\nb) In line 158, what is z_k? Should that have been z_i? (i is the variable that the sum is over, but it’s not used, unless I’m missing it?). Also, need to define n in that equation.\n\nc) In Equation 3 (which defines the counterfactual bias), how do we choose the value to modify a_i with? Does the equation need some expectation over which alteration to make to the affected attribute?\n\nd) in the “Biased-effect evaluation section”, the authors say that they investigate how the bias is learned. More precisely, though, what they do is estimate the influence of specific training examples, so a more appropriate description is to identify which examples lead to learning the bias, not *how* the bias is learned.\n\ne) What does “up” mean / stand for, in e.g. Equation 6.\n\nf) Generally, the Bias-effect evaluation paragraph is hard to understand without a background in influence functions. It would be great to give more intuition around equations.\n\ng) In the experiments section, the authors don’t describe the baselines they use, making it harder for someone unfamiliar with this literature to assess their appropriateness and the value of the associated comparisons\n\n\nB) [Terminology] Unlearning refers to removing the influence of a subset of training examples from a trained model. Therefore, doing “unlearning” on external samples (e.g. from a test set) doesn’t make sense according to that definition. Would be useful to clarify this point and perhaps use a different term instead. Debiasing sounds like a more appropriate word to me rather than unlearning (though inspiration may be drawn from unlearning methods). \n\nC) Issues with the experimental setup: \n\na) there are no confidence intervals in the tables, making it hard to assess whether the proposed method’s improvement over the considered baselines in each case is statistically significant. Sometimes the results are quite similar.\n\nb) In Table 1, why is DRO excluded from comparisons? In Table 3, why don’t the authors also show the Unb. and Wor. columns, as in Table 2? Those seem like a more direct and appropriate way to assess the degree of biasing in a way that actually matters / affects the model’s behavior, rather than only looking at the Bias column.\n\nc) Overall, the experiments are conducted on simple datasets with relatively small neural networks, and it’s unclear how this affects the conclusions. For instance, it’s surprising that unlearning only the top layer is successful, as in general this is not a good unlearning method (examples typically influence the trained model throughout layers, in a highly non-convex manner)  \n\nd) It would also be nice to compare against the oracle for unlearning, to get an upper bound. Specifically, once the harmful examples are identified, the oracle for unlearning would simply remove them from the training set and retrain the model without them (this of course only applies to the case where unlearning is done using training examples, as is the setup in unlearning). How well does the proposed approach debias, compared to this oracle for the unlearning step? \n\ne) Looking at Table 5, counterintuitively, Eq. 9 (“unlearning” using external samples) leads to smaller bias than Eq. 7 (unlearning using training samples). This seems to contradict the narrative that the former should do worse than the latter but is adopted instead due to not wanting to assume access to the training set? Do the authors have an explanation for this? Or is it the case that all these results are statistically indistinguishable from each other? (again, there are no confidence intervals in the tables, making it hard to tell.)\n\nQUESTIONS:\n- How did the authors pick the value of K (number of harmful samples to unlearn)? Is this different for each dataset / setting?\n\n- When computing the time (for quantifying the efficiency of the method), which steps are taken into account for the proposed method? Are all steps 1-3 counted towards this? If not, why not? Is this done fairly w.r.t how it’s counted for other methods?\n\n- Follow-up: what is the computational complexity of running the proposed method, in all 3 steps? Computing Hessians is expensive, and it sounds like the proposed method is to compute the influence of each example, in order to then rank them and find the top K.\n\n- How well does the method scale to deeper neural networks? All architectures used in the experiments are very small.\n\n- When using an external dataset for “unlearning”, how similar does that dataset need to be to the training set, in order for that to be beneficial? What if the external dataset is distributionally-shifted in different ways?\n\n- Could some of the baselines considered also be (easily) modified to use external datasets and / or fewer samples?\n\nLIMITATIONS:\n- The proposed method requires datasets with access to attributes, so that counterfactuals can be created by e.g. changing the value of one attribute while keeping the values of other attributes fixed. But in practice, we typically don’t have attribute information for datasets of interest.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the issue of biases in trained models, which are often caused by imbalances in the dataset used for training. The conventional approach to mitigating biases involves finetuning the model, but this incurs additional training costs. The paper proposes an alternative method called Fast Model Debiasing (FMD) that does not require finetuning.\n\nThe FMD method consists of three steps: identifying the bias attributes, identifying the top samples that contribute to the bias, and removing the bias. The key contribution of the paper is the use of Influence Functions (IF) and its extension to identify and eliminate biases. This approach is interesting and appears to be reasonable.\n\nSTRENGTHS:\nThe main contribution of the proposed method is to leverage the IF (Implicit Fairness) technique and its extension to identify and remove bias in models. This approach offers a practical solution for making models debiased.\n\nCompared to typical fine-tuning solutions, this method avoids the need for additional training effort, making it a more efficient procedure. By directly removing bias through the IF technique, the model can be made fairer without requiring extensive retraining.\n\nFurthermore, this method can be implemented as a separate API, allowing for easy integration into existing systems. This API can be used to remove bias from models, ensuring fairness in various applications.\n\nWEAKNESSES:\nHere, the general idea is simple: to identify the samples, and further enhance the model to debias.\nMore experiments should be added to verify this.\nrefer to the following: limitation parts.\n\nQUESTIONS:\nIn the approach, several layers in the model are updated based on certain rules to remove the bias. To further extend this work, it would be interesting to explore which modules are sensitive to bias and contribute to the debiasing process. \nFor example, when considering model adaptation during the inference stage, we can optimize the Batch Normalization (BN) module by re-estimating the mean and variance based on the fresh samples from new scenarios in just a few steps. HEre, BN is the sensitive for model adaptation. \nBack to the debias, which module is the key?\n\nThere are some other questions:\n1. Select K top harmful samples. what is the suggested value for K? \n2. Such harmful samples highly depends on the testing samples. what is such testing samples from? \n-if testing samples is from one subset of the training/validation set, how does the proposed not involve new bias after removing the influence of K top harmful samples? \n3. Not find the bassline accuracy of the model without the debias, and how to guarantee that the updated parameters would not impact the model accuracy? \n4. Have you tried other complex or big model?\n\nLIMITATIONS:\n1. In the proposed FMD,  based on the Newton update, the model is updated. The model accuracy generally would be impacted. But, most of the given experiments show that both of accuracy and debias performance are best. Such results might not be convinced. All of the experiments have been conducted using small or simple models on easy tasks. It is important to verify whether such methods can be applied to other typical or complicated tasks/datasets, such as ImageNet/COCO or segmentation tasks. Please enable more experiments or cite more other reference.\n* Firstly, add one line or column to show the baseline performance, e.g., the accuracy without the debias. \n* Secondly, extend to more scenarios, e.g., imageNet dataset.\n* Thirdly, maybe one flow figure is helpful to understand the overall FMD procedure.\n* Model debias is partially related to other topics, e.g., OOD, model, domain adaptation or others, where some debias method are also introduced. if time and space are ok, survey and add more reference. \n\n2. Implementing such a scheme requires building a specific dataset (even if it is small) and updating the model layers, which can be a time-consuming process. It might not be easier than performing a quick finetuning for debiasing. Therefore, it would be beneficial to compare the complexity between the proposed approach and quick finetuning, as well as evaluate their performance.\n* In general, quick finetuning does not necessarily require a larger dataset. A small specific dataset with data augmentation might be sufficient for finetuning to address bias.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

comments: SUMMARY:
In this paper, the authors propose Fast Model Debiasing (FMD), which targets solving biased prediction problems. The proposed FMD can effectively and efficiently remove model bias by leveraging only a small external dataset by altering the attribute, then evaluating the biased-effect of each sample, and finally using the most harmful samples to unlearn the model bias with the Newton update step. Experiments on Colored MNIST, CelebA, and Adult datasets demonstrate the effectiveness of the proposed method.

STRENGTHS:
The motivation and the idea proposed in this paper are reasonable. The unlearning progress is efficient since the model parameters only need to be updated in several steps. The execution time is greatly shorter than the compared methods. The experimental results are sufficient to support the effectiveness of the proposed method.

WEAKNESSES:
1. Some writing issues. "Fast Model Debias" in the title, but "fast model debiasing" in the abstract. The full name of "FMD" is not explained when it first appeared in Sec 1. 
2. In Eq 3, reduant ")". 
3. Some usage of notation is not standard. For example, the function "B" accepts three kinds of input. 
4. In line 231, the notation $C_i$ should be $c_i$.
5. The number of K when selecting harmful samples, and the bias threshold seems not explained in the experiments.

QUESTIONS:
N/A

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel method for model debiasing, based on the techniques of counterfactual biases and machine unlearning. The method designs an all-inclusive debiasing pipeline: bias identification, biased-effect evaluation, and bias removal. The proposed method can get rid of time-consuming human labeling or re-training, which is appealing to practical applications. Experimental results and ablation analysis suggest that the proposed method is effective and more efficient on multiple datasets for fairness evaluation. Overall, I appreciate the idea of using machine unlearning and counterfactual samples for efficient model debiasing, though some concerns still need to be addressed for improved quality.\n\nSTRENGTHS:\nStrengths:\n1. Model debiasing and fairness are critical for the deployment of large AI models in practice. The proposed method can identify and remove the bias in a cost-effective manner, which is more suitable for practical applications than previous retraining methods. \n2. The idea is novel and interesting. Employing counterfactual samples to identify biases seems quite helpful. Leveraging the machine unlearning techniques with counterfactually constructed datasets for bias removal shows a new promising direction for efficient model debiasing. \n3. The sanity check is clear, experiments in the main paper and appendix are quite comprehensive, and the performance in terms of accuracy, fairness and efficiency are promising.\n\nWEAKNESSES:\n1. The post-processing baselines are included on the Colored MNIST. I suggest including them for other datasets, i.e., CelebA and Adult. \n2. More discussions on how the proposed method can be generalized for large-scale networks or tasks. Existing fairness studies are mainly conducted on small datasets like C-MNIST and networks such as MLP/ResNet. It would be great if the authors can investigate how the proposed unlearning method can be extended to large-scale datasets or models, for examples, is it possible to extend this idea to  large-scale language models. Even some preliminary results or discussions would be highly appreciated. \n3. Besides quantitative results in Table 1-3, more straightforward visualizations of experimental results on how the proposed method can give rise to better fairness (less bias) are highly encouraged  to better verify the effectiveness.\n\nQUESTIONS:\nSee weakness. \nMinor issues: \n1. Writing need to be improved, e.g., typos and grammar errors: line 40 involves (involve), line 35 “or”, etc. \n2. Paper title and manuscript title are not the same. Title should be \"Fast model debiasing with machine unlearning\"?\n3. What are the definitions of  Demographic parity bias (De.) [34], and Equal opportunity bias, and how they relate to the counterfactual bias? The Eq 10 and 11 need to be further clarified.\n\nLIMITATIONS:\nAuthors addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel approach to addressing bias in models by utilizing a small external dataset for de-biasing. The proposed method effectively identifies and mitigates inherent biases by analyzing the disparities between original images and counterfactual images. Initially, the method determines the specific type of bias present in the model by computing these differences. Subsequently, it identifies samples with the highest influence scores in terms of counterfactual bias when a particular bias type is detected. To mitigate this bias, the method updates the model by unlearning these influential samples, using the difference between the gradients of the original and counterfactual samples. The method is evaluated using one synthetic dataset and two real-world datasets. The results demonstrate that the proposed method achieves state-of-the-art performance in mitigating counterfactual bias, while also maintaining low computational costs.\n\nSTRENGTHS:\n* The intuition of the proposed method seems clear and intuitive. Also, the proposed method is novel and well-designed two recent techniques in other domains, influence function and unlearning.\n\n* The proposed method can mitigate the bias of a model efficiently. It only needs a small amount of group-labeled data points. Furthermore, the method requires much less training time than the existing de-biasing methods based on re-training.\n\n* The method achieves good performance on CelebA and Colored MNIST.\n\nWEAKNESSES:\n* Lack of some baselines. Given that the objective of this method is to achieve counterfactual fairness or individual fairness, it is important for the authors to compare their approach with existing methods [1,2] specifically designed for individual fairness. This would ensure a fair and comprehensive evaluation of their method's performance in terms of counterfactual bias. It should be noted that Group DRO and LfF, the authors adopted as baselines, were originally designed for group robustness, rather than individual-level fairness.\n\n* Challenges in obtaining counterfactual samples. One drawback of the proposed method is that it requires counterfactual samples for each instance in the external dataset. However, acquiring such counterfactual samples for image datasets is not a trivial task. For instance, assuming that images are annotated with 40 attributes, as done for the CelebA dataset, may not always be feasible.\n* There should be a more detailed explanation of why the experiment results turned out the way they were presented. Why does your method that archives the counterfactual bias also mitigate the worst-case group accuracy or any other group fairness metrics?\n* Improving the writing clarity. \n  - It is suggested to provide clearer distinctions between the problems of model bias and fairness, emphasizing that while there is overlap between the two, they are not synonymous. \n  - Additionally, the definition of counterfactual fairness in Equation (1) may be incorrect, compared to the definition introduced in [31]. \n  - Some typos in equations. In lines 158 and 183, is it z_i instead of z_k?\n\n[1] Yurochkin and Sun. SENSEI: Sensitive set invariance for enforcing individual fairness, ICLR, 2021.\n[2] Yurochkin et al. Training individually fair ML models with sensitive subspace robustness, ICLR, 2020.\n\nQUESTIONS:\n* Why is the worst-case accuracy better than the baselines in Table 2? Why does unlearning by Co. outperform unlearning by De. in terms of De? \n\n* Why does unlearning by De. fail to better demographic parity bias in Table 4?\n\n* In Table 7, unlearning more layers of neural networks does not lead to better performance. Why?\n\nLIMITATIONS:\nThe author has addressed their limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}