Fast Model DeBias with Machine Unlearning
Authors: Ruizhe Chen
Jianfei Yang
Huimin Xiong
Jianhong Bai
Tianxiang Hu
Jin Hao
YANG FENG
Joey Tianyi Zhou
Jian Wu
Zuozhu Liu
Venue: NeurIPS 2023 poster (2023)
Keywords: Model Debias
Bias Mitigation
Machine Unlearning
Counterfactual Fairness

Abstract: Recent discoveries have revealed that deep neural networks might behave in a biased manner in many real-world scenarios. For instance, deep networks trained on a large-scale face recognition dataset CelebA tend to predict blonde hair for females and black hair for males. Such biases not only jeopardize the robustness of models but also perpetuate and amplify social biases, which is especially concerning for automated decision-making processes in healthcare, recruitment, etc., as they could exacerbate unfair economic and social inequalities among different groups. Existing debiasing methods suffer from high costs in bias labeling or model re-training, while also exhibiting a deficiency in terms of elucidating the origins of biases within the model. To this respect, we propose a fast model debiasing method (FMD) which offers an efficient approach to identify, evaluate and remove biases inherent in trained models. The FMD identifies biased attributes through an explicit counterfactual concept and quantifies the influence of data samples with influence functions. Moreover, we design a machine unlearning-based strategy to efficiently and effectively remove the bias in a trained model with a small counterfactual dataset. 
Experiments on the Colored MNIST, CelebA, and Adult Income datasets demonstrate that our method achieves superior or competing classification accuracies compared with state-of-the-art retraining-based methods while attaining significantly fewer biases and requiring much less debiasing cost. Notably, our method requires only a small external dataset and updating a minimal amount of model parameters, without the requirement of access to training data that may be too large or unavailable in practice.

Fast Model Debias with Machine Unlearning
Ruizhe Chen', Jianfei Yang', Huimin X Jianhong Bai', Tianxiang Hu', Jin Hao', Yang Feng', Joey Tianyi Zhou', Jian Wu', Zuozhu Liu Zhejiang University 2 Nanyang Technological University Stanford University Angelalign Technology Inc. Centre for Frontier Al Research ruizhec. 21@int1...... zju. edu.
Abstract
Recent discoveries have revealed that deep neural networks might behave in a biased manner in many real-workd scenarios. For instance, deep networks trained on a large-scale face recognition dataset CelebA tend to predict blonde hair for females and black hair for males. Such biases not only jeopardize the robustness of models but also perpetuate and amplify social biases, which is especially concerning for automated decision-make making processes in healthcare, recruitment, etc., they could exacerbate unfair economic and social inequalities among different groups. Existing debiasing methods suffer from high costs in bias labeling or model re-training, while also exhibiting a deficiency in terms of elucidating the origins of biases within the model. To this respect, propose fast model debiasing framework (FMD) which offers an efficient approach to identify, evaluate and remove biases inherent trained models. The FMD identifies biased attributes through an explicit counterfactual concept and quantifies the influence of data samples with influence functions. Moreover, design a machine unlearming-based strategy to efficiently and effectively remove the bias in trained model with a small counterfactual dataset. Experiments on the Colored MNIST, CelebA, and Adult Income datasets along with experiments with large language models demonstrate that our method achieves superior or competing accuracies compared with state-of-the-ae methods while attaining significantly fewer biases and requiring much less debiasing cost. Notably, our method requires only a small external dataset and updating minimal amount of model parameters, without the requirement of access to training data that may too large or unavailable in practice.
Introduction
Biased predictions are not uncommon in well-trainee deep neural networks [13]. Recent findings indicate that many deep neural networks exhibit biased behaviors and fail to generalize to unseen data 44 [5) c.g., convolutional neural networks (CNNs) might favor texture over shape for object large-scale dataset (e.g. CelebA) tend to
classification [6]. For instance, well-trained networks predict a female person to be with blonde hair, and a male to be with black hair 7 This because the number of <blonder hair, female> and <black hair, male> image pairs is significantly higher than others, although there is no causal relationship between hair color and gender [9]. In this case, the model does not learn the correct classification strategy based human appearance, but rather shows a preference for specific individuals or groups based on irrelevant attributes (error correlations) (2]. Such error correlations not only affect the model's ability make robust predictions but also perpetuate and exacerbate social bias, resulting in potential risks in many real-world scenarios, such as racism, underestimating minorities, social disparities among groups in crime prediction (10) loan assessment [.] and recruitment [12] etc.
"Coreesponiig author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Efforts have been made to remove bias in models based on innate or acquired characteristics of individuals or groups. Existing debiasing mechanisms could be categorized into three types depending on when debiasing is conducted: pre-processing, in-processing, and post-processing [2] 13 14
Pre-processing debiasing methods usually modify the dataset for fair learning, which often involve
reweighing samples [5 16], modifying feature representations [17] [8], changing labels [9] etc.
Another line of research accounts for fairness during training, i.c., un-poocessing [20H24], including feature-level data augmentation or adversarial training [25] 26 etc. However, the aforementioned methods require expensive costs for human labeling of misleading biases or campuuttionally-inensiit debiased model retraining, resulting in unsatisfactory scalability over modern large-scale datasets or models. Few research explore proscesosssiigg strategies to achieve fairness with minimal cost [27]- 29]. They ensure group fairness by alternating predictions of some selected samples, causing degraded accuracy or unfairness on individuals. Moreover, most methods assume that the biased attributes were known, while generalized debiasing framework should be able to verify whether attribute (e.g. shape, texture, and color in an image classification task) is biased or not as well (30]
Common
Counterfactual Dalasel
Trainsl
Unleamed Model
Model
MARIO
poodi
pool Blande
Blas
predi
pezz: Monal prods
pools
poto:
Unkkem
proks
raae
Blaued Eificrt Paskuttion
post dork
poad blondd
pesto
pesst Nenee
proks
poeer Nonde mrck 0.83
Figure 1: Pipeline of our proposed FMD.
In this paper, wC propose FMD, an all-inclusive framework for fast model debiasing. As illustrated in Fig. [1. the FMD comprises three distinct steps: bias identification, biased-effect evaluation, and bias removal. In contrast to pre- or in- pr-prceessing debiasing methods, our approach eliminates the need for supervised retraining of the entire model additional labeling of bias attributes. Notably, FMD leverages only small external dataset, thereby obviating the requirement for access to extensive or unavailable training data in practical scenarios. Furthermore, achieving fair outputs through FMD necessitates updating only a minimal number of parameters, such as the top MLP layers of pre-trained deep networks. Compared post-processing debiasing methods, FMD yields superior debiasing performance and consistently enhances fairness across diverse bias metrics with little costs.
The FMD operates through the following procedure. Given an attribute and well-trained model, our first step is to ascertain whether and to what extent the model exhibits bias towards the attribute. To achieve this, construct dataset comprising factual samples along with their corresponding counterfactual samples (31, wherein the attribute in question can be varied. By observing how the model's 's predictions change with the attribute variations, we can effectively identify any bias present. In the biased-effect evaluation phase, we quantitatively assess the extent to which a biased training sample contributes the model's biased predictions. This evaluation entails measuring how the biased training sample misleads the model and influences its predictions. To this end, we extend the theory of influence functions (32), employing estimate the impact of perturbing biased attribute within the training data on the model's 's prediction bias measurement. Finally, we introduce an unlearning mechanism that involves performing Newton step 333 on the learned model parameters to remove the learned biased correlation. We further design an alternative strategy to unlearn biases with the counterfactual external dataset, avoiding hard requirements on access to the training data which might be unavailable in practice. Our unlearning strategy effectively eliminates the estimated influence of the biased attribute, leading to a more fair and unbiased model. Experiments on multiple datasets show that our method can achieve accuracies a much smaller counterfactually constructed dataset. The corresponding biases and computational costs are significantly reduced well. Our main contributions are summarized as:
par with bias-tailored training methods with

---

We propose counterfactual inference-basee framework that can quantitatively measure the biased degree of trained (black-box) deep network with respect to different data attributes with novel influence function.
We propose an unlearning-based debiasing method that effectively and efficiently removes model biases with a small counterfactual dataset, getting rid of expensive network re-training bias labeling. Our approach inherently applies to in-processing debiasing.
Extensive experiments and detailed analysis on multiple datasets demonstrate that our framework can obtain competing accuracies with significantly smaller biases and much fewer data and computational costs.
2 Related Works
2.1 Group, Individual and Counterfactual Fairness
The pursuit of fairness in machine learning has led to the proposal of fairness-specific metrics. These metries have been mainly categorized into two types: metrics for group fairness that require similar average outputs of different demographic groups [4:38; ; and metrics for individual fairness that necessitate similarity in the probability distributions individuals that are similar in respect to specific task, regardless of their demographic group (39/42]. Generally, statistical parity among protected groups in each class (group fairness) could be intuitively unfair at the individual level [43]. Moreover, existing fairness metries put heavy emphasis on model predictions, while underestimating the significance of sensitive attributes for decision- making and are insufficient to explain the cause of unfaireess in the task B1 (44 Recently, [1 introduces counterfactual fairness, a causal approach to address individual fairness, which enforces that the distribution of potential predictions for an individual should remain consistent when the individual's protected attributes had been different in a causal sense. In contrast to existing individual bias metrics, counterfactual fairness can explicitly model the causality between biased attributes and unfair predictions, which provides explainability for different biases that may arise towards individuals based on sensitive attributes 445.
2.2 Bias Mitigation
Proposed debiasing mechanisms are typically categorized into three types 13 14 pre-processing, in- pr-processing, and post-processing. Pre- and in-processing algorithms account for fairness before and during the training process, where typical techniques entail dataset modification [9] and feature manipulation 24 26 255. Post-processing algorithms are performed after training, intending to achieve fairness without the need modifying data or re-training the model. Current post-processing algorithms make more fair decisions by tweaking the output scores [0]. For instance, Hardt [27 achieves equal odds or equal opportunity by flipping certain decisions of the classifier according to
their tob-groups. 229 28 select different thresholds for each group, in manner that maximizes accuracy and minimizes demographic parity. However, achieving group fairness by simply changing the predictions of several individuals is questionable, e.g., the process might be unfair to the selected individuals, leading to an unsatisfactory trade-off between accuracy and fairness.
2.3 Machine Unlearning
new paradigm forget specific data sample and remove its corresponding influence from a trained model, without the requirement to re-train the model from scratch. It fulfills user's right unlearn her private information, i.e., the right be forgotten,
Machine unlearning (51553]
in accordance with requests from the General Data Protection Regulation (GDPR) [54] Existing
unlearning approaches can be roughly categorized into two types: exact unlearning [55] 566 and approximate unlearning 60]. Data influence- approximate unlearning that utilizes influence functions 32) to approximate and remove the effect of a training sample on the model's 's parameters 63] In this paper, we are inspired by the paradigm of machine unlearning and extend it to remove the model's 's bias from a deep network without retraining it from scratch.
unlearning is representative branch of
3

---

3 Method
3.1 Overview and Preliminaries
Problem Formulation. Consider supervised prediction task with fairness considerations that maps input attributes A (biased attribute) and X' (other attributes except A) to certain outputs y (labels). The training dataset Dtr can be represented as (21, 22,-.,, 2n where each training point 2, = {(a,, zz), yi E A x X' x V. Let fg denote the trained model (predictor) with parameter A. Let [zz,,) denote the loss on the training sample 2; w.r.t. parameter 0. It is deemed biased if biased attribute is highly correlated but wrongly correlated to the prediction ý = ʃg (x. a), e.g., a CNN is biased it predicts hair color (black/blonde) with the biased attribute genders (male/female).
Motivation. In large part, existing works focused on measuring fairness with implicit quantitative values (e.g. accuracy). However, they do not provide explicit illustrations on whether the decision- making is based on sensitive/protected attributes. Furthermore, based on the bias identified, research on how such bias learned from training samples limited. Our proposed method bridges this gap with two components: identifying bias from different predictions with counterfactual samples and evaluating the biased-effect from training samples with a modified influence function. Furthermore, we propose novel machine unlearning-basse -based method to efficiently and effectively remove the biases.
Counterfactual Fairness. We identify the biases of trained models with the concept of counterfactual fairness [31 45 which better models the causality between biased attributes and unfair predictions.
We detail the definition following (31:
Definition 1 (Counterfactual fairness). A trained model fg is counterfactual fair on A if for any a, A.
PiA-_ y X = x, A=
(I)
for all x E X attainable by X.
Note that [x( A), which implies the process of attribute changing. The definition suggests that, for any individual, changing a, i.c., from a to a while holding other attributes x unchanged should not change the distribution Y if a is a biased attribute.
Influence function. Influence functions, a standard technique from robust statistics, are recently extended to characterize the contribution of a given training sample to predictions in deep net- e.g.. identify whether a sample is helpful or harmful for model predictions. A popular implementation of influence functions is to approximate the effects by applying the perturba- tion (z, y) + = (x: 8, y) (32 that define the parameters resulting from moving mass from onto zạ: 0 0,. argmin Σ L(zi, 0) eL(zs, 0) eL(z,, 0). An approximated computation of the influence as in 332 can be defined as:
works 32 64 65
d9
c,26,
(Vol(s))) voL(2,0))
(2)
de
c==
3.2 Bias Identification and Biased-Effes Evaluation
Counterfactual bias identification. We first identify the biases in a trained model with counterfactual concepts. Given trained model ʃg and an attribute of interest A. primary question is whether Sg is fair on A. We employ an external dataset Drz (can be constructed from the test set) to identify biases. To measure how prediction changes in accordance with the attribute, for each sample G = (I,, a.) E Dea, where a; E A, we alter a, while keeping unchanged based on the requirements of counterfactual fairness. The generated counterfactual sample is denoted as &, = (I;, az), a, E A. We further define the counterfactual bias of the model fa on sample c; as the difference predictions:
B(q, = p( = fo(X,A)) X z,, 2, = P( = = fx(X,4) X = 24, A o.)
The counterfactual bias on the whole dataset Daz can represented as the average of individual counterfactual biases:

---

B(D ez: A.0)
Σ B(c A.0).
(4)
|Deu
The measured bias is a scalar normalized from 0 to 1. We set bias threshold å that if the measured B(Dew. A, fo) is larger than 8, we regard fg to be biased on A. Note that our method could also generalize to other individual bias metries besides Eq.
Biased-Effect Evaluation. Based on the identified counterfactual bias, we then investigate how the bias on is learned by the model from training samples. Considering B(0) measured on any A with any Den. our goal is to quantify how each training point in the training set Dte contributes to B(O). Let's denote the empirical risk minimizer as 0 = arg ming Li=LLZz, 0). and assume that the empirical risk is diccedifferentiable and strictly convex in 0. The influence function [64] provides an approximation on the updates to parameters if 24.
removed from Drr with small coefficient
The new parameters can be obtained as 0z.z, = arg ming 1 Di=1ii [zz,, 0) + el(z,, 0). By
doing the influence of removing 2 the bias B(0) can be defined as:
biae (Zk, B(Ò)) dB((e..e)
dB((..:.) dfc..
-V BO)HH (000)H15''LL000 VoL(Zz, (5)
de
c=0
d0,,z.
de
where Hi def
t VEL(E,, 0) is the positive definite (PD) Hessian, and the closed form expres-
sion of db explaining the influence of zn to model parameters, is provided by the influence Note that "up" denotes "upweight". Refer to Appendix A for the derivation. Intuitively, this equation be understood in two parts: the latter part calculates the impact of removing on the parameters. The former part corresponds to the derivative of bias with respect to parameters, assessing how changes in parameters affect the bias. Hence, this equation quantifies the influence of removing on the bias. Note that (0) can be any bias measurement of interest. Taking B(Dea. emy A, defined in Eq. 4 as an example, the influence on counterfactual bias can be boiled down as:
de
function 32
hup.bias(2z: B(Dez; A,0) = Dee
Σ c.DD..
- Vela( (Vola(c)-Voaalae V.L 0),
(6)
where Tup.bias (Ek, B) scalar that measures how each training sample contributes to B. removing the point 2ę increases the bias, we regard =g as a helpful sample, or harmful otherwise. We provide an illustration of the helpful and harmful samples with a toy example in Section B.I
3.3 Bias Removal via Machine Unlearning
After quantifying how biases are learned by the model from harmful samples, the next question is how to remove such biases. Here we propose machine unlearning-based strategy to remove the biases caused by harmful samples. particular, we exploit the powerful capability of machine unlearning paradigms for forgetting certain training samples [66] 62 63] Specifically, for bias measurement B (f).we we first rank the influence Iup.biae B()) of every training sample 24 in Der. and then select the top-K harmful samples. Afterward, we unlearn, i.c., let the model forget, these samples by updating the model parameters 0 with Newton update step as in [63
K
nne = 0 + Σ 6 (VL(
km
where H,J ' VoL(ZK,)) Tup.parems (2x) is explained as the influence of z on model parameter 32
Note that Lup. Axp.params (Z+) share similar computation in Eq. 6 while Lup upp.arams (zx) estimates the influence on model parameter and Iup.bias(24 2k, B) focuses on influence on biases.
Our unlearning strategy is further refined following the observations from experiments in Section B. In particular, by ranking and visualizing the harmful and helpful samples on the biases (as shown in Fig. 5) we have observed that the harmful samples heavily lead to biased/error correlations (i.e., bias-eligned) while the helpful samples behave oppositely (i.e., bias-conflicting). Hence, we propose straightforward solution that further mitigates the influence of a harmful sample with
5

---

ciasfiinfii sample. Consequently, we update the parameters to unlearn the harmful samples by:
K (Vg - VzL(2,,0))
Bnew A
Σ
(8)
where Eg denotes the bias-conflicting sample of zg. Following the explanation in influence theory [22 our unlearn mechanism removes the effect of perturbing training point (á, v. y) to (a, w. y)- In other words, we not only remove the influence caused by harmful sample 2k. but further ensure fairness with the corresponding counterfactual sample 2g, see more details in Section B.I 4.4 and Appendix.
Alternative Efficient Unlearn with Cheap External Datasets. In the above sections, the unlearning process is based on the assumption that we could access the original training sample = to identify and evaluate biases and then forget them. However, in practice, the training set might be too large unavailable in the unlearning phase. In response, we further propose to approximate the unlearning mechanism with small external dataset. As the influence to be removed can be obtained from the change of the protected attribute, we can construct the same modification to the protected attribute on external samples. In particular, we employ the Dew in Section 3.2 for unlearning, which redefines Eq. 21 as:
to construct counterfactual pairs
Frww = 0
As Dex can be easily obtained from an external dataset rather than the training set, the practical applicability of our method could be greatly enhanced, as demonstrated the experiments.
3.4 Model Generalization
Extension to Different Biases. To fulfill different fairness demands, we further discuss the general-
ization of the bias function B(²) in Eq. to other bias measurements. We provide the extension to the most frequently used group faimess measurement demographic parity [34] which requires equal
positive prediction assignment across subgroups (e.g. male and female). Eq. 6 can be rewritten as:
Lup biae (iia(Zz)) (zx) =
Σ A
falc.) Vo G
Σ fo(c)) ( VgL(ZN. 0), (10)
GA
CyESA=D
where GA=1 and G represents the subgroup with protected attribute A = 1 and 0. The extension to equal opportunity [35), which requires the positive predictions be equally assigned across positive classes, can be rewritten as:
Jup,bias (2x) (V GI.1
1
folc.) V 4EE1..1
Σ
Σ cjE0o.1
(Voe)))(VVoL(z) (oL(Zz, (1I)
Go
where G1,1 represents the sub-group where A 1 and Y =
Extension to Deep Models. In the previous sections, it's assumed that ô could be the global minimum. However, if is obtained deep networks trained with SGD in a non-convex setting, it might be local optimum and the exact influence can hardly be computed. We follow the strategy in [32] to approximate the influence in deep networks, and empirically demonstrate the effectiveness of FMD in deep models. Moreover, for deep networks where linear classifier is stacked on a backbone feature extractor, we apply our unlearning mechanism to the linear classifier or several top MLP layers.
Efficient Influence Computation. A critical challenge to compute the influence in Eq. i to explicitly calculate the inverse Hessian. Here we employ the implicit Hessian- vector products (HVPs) [3, 67 to efficiently approximate
Algorithm 1: The FMD framework.
Input: dataset De., loss C, attribute of interest A, Hessian matrix H, bias threshold parameter B, |DD
LgL(zz, 0). Meanwhile, VIL(ZE, 0) Eq. 6 can be pre- calculated and applied to different V, ((). To avoid the (dd) computational cost to calculate the inverse Hessian in every step, we pre-calculate it before the removal and keep it constant during unlearning phase (63]. The alter- native strategy which continuously updates the inversion Hessian is also analyzed in the Appendix.
B + B(D... H Inverse(H))
B>5 then
for i 1.2.3....II do
A V VoL(c, 8) VyL(e,
8
end
end Output: 0

---

Experiment
4.1 Experiment details
Dataset, Our experiments are conducted on three datasets. Colored MNIST'S is constructed by adding color bias to the MNIST dataset [68]. Bias-aligned samples are constructed by adding particular color to a particular digit like [Digitl [ while other colors are for bias-conflicting samples. Following 3 69 70,, we build different training sets by setting different biased ratios (0.995, 0.99, 0.95) for biased-aligned training samples, where a high ratio indicates a high degree bias. CelebA / is face recognition dataset with 40 types of attributes like gender, age (young not), and lots of facial characteristics (such as hair color, smile, beard). We choose Gender as the bias attribute, and Blonde hair and Attractive the outputs following 7 8 Adult Income Dataset is publicly available dataset in the UCI repository [72) based on the 1994 U.S. census data. The dataset records an individual's income (more or less than $50,000 per year) along with features such as occupation. marital status, and education. In our experiment, we choose gender and race as biased attributes following 73 74). We follow the pre-processing procedures in (75). As for the experiment on the language model, we use StereoSet [76] as our test set. StereoSet is large-scale natural dataset to measure stereotypical biases in gender, profession, race, and religion.
Colorl)
Baselines. For the sanity check experiment on a toy Colored MNIST dataset, we use a vanilla logistic regression model the baseline. For experiments with deep networks, we compare our method with one pre-processing baseline Reweigh (77), 6 in-processing debiasing baselines (LDR (25),
LfF (78.. Rebias 779), DRO 7 SenSEI 88 and SenSR (8I) and 4 post-processing baselines
(EqOdd [35]. CEqOdd [35). Reject [82] and PP-IF (83). We compare our method on language
model with five debiasing baselines: Counterfactual Data Augmentation (CDA) (84) Dropout (85),
Iterative Null-space Projection (INLP) [86]. Self-debias 87, and SentenceDebias
88 Details can
be referred Appendix C.
Construction of Counterfactual Dataset Dex- We separately construct counterfactual sets for the three datasets, while bias-aligned samples in the small dataset Der are all split from the test set. In the Colored MNIST dataset, we randomly add another color on the same digit image as the counterfactual sample. As for the Adult dataset, we flip the protected attribute to the opposite while keeping other attributes and target labels exactly the same. For the CelebA dataset, we select images with the same target labels but the opposite protected attribute. To fulfill the request for counterfactual, we rank the similarity between images by comparing the overlap of other attributes and choose the most similar pair to form the factual and counterfactual samples. Part of the generated sample pairs is visualized in Fig. 2 Note that for the CelebA dataset, the counterfactual data are not that strict as the gender attribute not independent of other features in the natural human facial images. We use Crows-Pairs [99] as our external dataset for the language model. Each sample in Crows-Paire consists of two sentences: one that more stereotyping and another that is less stereotyping, which can be utilized as counterfactual pairs.
J
3
4
5
7
8
9
a
3
4/-
5
4
7
q
(a) Colord MNIST
Workstee
Inteetion
Sdecatoo-vuu
Martel-rana
Oociprtee
Pice
Hearswwek
Natee-ceeity
Latel
Folara
Hö--gadd
Marrad
Mlaciree op-rn-----ppp
White
Mide
Unitad Staae
Pnote
Machee--p--rapt
Mide
Unitad- States
(c) Adutt
Figure 2: Visualization of factual and counterfactual pairs for three datasets.
Implementation details. We use multi-layer perceptron (MLP) with three hidden layers for Colored MNIST and Adult, and ResNet- 18 990 for CelebA following the setting in [8. During training, we set the batch size of 256 for Colored MNIST and Adult, respectively, and 64 for CelebA following 7]. We use pre-trained BERT 91 and GPT-2 [92], provided by Huggingface. During
[25] 78, unlearning, we freeze the parameters of all other layers except the last classifier layer. The running time of all baselines is evaluated on a single RTX3090 GPU for fair comparison. In our experiment, we select the number samples k=5000 for Colored MNIST, and k=200 for both Adult and CelebA. The bias threshold is set to 0.
7

---

4.2 Sanity Check on Logistic Regression with a Toy Dataset
We conduct an experiment on logistic regression task to illustrate our method. We simplify the Colored MNIST classification task to binary classification problem of distinguishing between only digits 3 and 8, on a training set with a bias ratio of 0.95, and a balanced test set. We trained regularized logistic regressor: argmin R4 Σi=1 [w 24, 21) + A||w||? 3. Fig. 5(a) illustrates the classification results of the vanilla regressor on part of test samples. We denote Digit by shape (triangle and rectangle) and Color by color (red and blue). The solid line represents the learned classification boundary and the dotted line represents the expected classification boundary. The test accuracy is 0.6517 and it can be observed that most bias-conflict samples tend to be misclassified according to their colors. Moreover, we select and visualize the most helpful and harmful samples in Fig. (5c) based on Eq. 6 We found that the most helpful samples are in the 5% bias-conflict samples while harmful samples are bias-aligned samples. The unlearning curve is provided in Fig. (b).. With only 50 samples, the accuracy improved amazingly by 25.71 55,71% % and the counterfactual bias decreases by 0.2755, demonstrating the effectiveness of our method.
3
8838
3
8
3
8
8
3
3
3
8
8
(a)
(b)
(c)
Figure 3: (a) Illustration of the learned pattern on our toy dataset.(b) Accuracy and bias curves during
unlearning, (b) Visualization of helpful samples (top row) and harmful samples (bottom row).
4.3 Experiment Deep Models
Results on Colored MNIST. Tab. [ shows the comparisons on the Colored MNIST dataset. We reported test accuracy, counterfactual bias, debiasing time, and the number of samples used for all methods. Our approach demonstrates competing performance on accuracy and superior performance on bias compared with retraining baselines. Meanwhile, we only make use of one-tenth of unlearning samples and reduce the debiasing time by 1-2 magnitudes.
Bins Ratio
Method
Acc-(%) f
Blas J
Time(s)
# Samp.
Attr.
Method
Acc:(5) 1
Bias
Timets)
Samp.
Vanilla
38.59
0,5863
Vanilla
85.40
0.0195
LDR
66.76
0.4144
1,261
50 k
LDR
26,904
0.995
LIF
56.45
0.3675
661
50 k
LIF
73.08
0.0036
525
26,904
Rebias
Ours
71.70
21.24
0.3428
0.3027
1,799
59
k
Gender
Roweigh
Rehias
76.57
0.0041
1292
26,904
26,904
SenSR
84.09
0.0049
571
26,904
Vanilla
51.34
0.4931
SenSel
83.91
0.0016
692
26,904
IDDR
0,2511
1,330
PP-JF
26,904
0.99
LIF
64.71
0.2366
726
50 k
Ours
81.89
0.0005
2.49
500
Rebias
80.41
0.3302
1,658
50 k.
Ours
80.04
0,2042
Vanilla
84.57
0.0089
LDR
26,904
Vanilla
77.63
0.2589
LIF
75.16
0.0024
501
26,904
0.95
LDR
LIF 85.55
90.42
0,2333
0.1264
1,180
724
50 k
Race
Reweigh
Rehias
77.89
0.0038
1304
26,904
26,904
Rebias 89.63
0.1205
1,714
50
SenSR
84.09
0.0036
571
26,904
Ours
89.26
0.1189
56
SenSel
83.91
0.0015
692
26,90
PP-IF
82.37
0.0015
26,904
Table 1: Results on Colored MNIST. (bold: best performance, underline: second best performance.)
Ours
83.80
0.0013
2.54
500
Table 2: Results on Adult.
Results on Adult. The results are in Table It can be observed that the vanilla method performed the best in accuracy on both tasks, since in the real-world dataset, race and gender are biased w.r.t income in both training and test set and the well-trained model fits this correlation. However, to achieve fair prediction, we would not expect biased attributes to dominate predictions. Compared with other debiasing methods, our method achieved the best results in both accuracy and bias, with much less debiasing time on smaller dataset.
8

---

Results on CelebA, We compare on average accuracy (Avg.), Unbiased accuracy 8 (Unb.) tested on the balanced test set, and Worst Worst-group group accuracy 7 (Wor.) tested on the unprivileged group to illustrate the performance, as reported in Tab. 3 It can be observed that the vanilla model performs well on the whole dataset (Avg.) but scores a really low accuracy (Wor.) on the worst group, which means the learned model heavily relies on the bias attribute to achieve high accuracy. Our method obviously bridges this gap and outperforms all other debiasing baselines on Wor. and Unb. in the two experiments. The experiments also demonstrate that our method is consistently feasible even in the absence of perfect standardized counterfactual samples in real-world datasets, by selecting a certain amount of approximate counterfactual data.
Results on Large Language Models (LLM). We further extended our method to the I.L debiasing scenario. Results are presented in Tab. 4 We report two metrics: Language Modeling Score (LMS) measures the percentage of instances in which a language model prefers the mean- ingful over meaningless association. The LMS of an ideal language model is 100 (the higher the better). Stereotype Score
Altr.
Mothod
Unb:(%)
Wor.(k)
Avg.(%)
Bias
Timels)
Vanilla
66.27
47.36
94,90
0.1211
LF
84.33
81.24
93.52
0.2557
67.620
Blonde
85.01
N2.32
86.67
24,180
DRO
85.66
81.36
92.90
0.3206
28,860
Ours
89.73
87.15
93.41
0
191
Vanilla
63.17
40.59
77,42
0.3695
LF
71:24
67.56l
Atmective
68.14
81.70
24,420
DRO
66.14
62.33
78.35
0.3004
30,540
Ours
72.18
68.16
80.99
0.1273
187
Table 3: Results on CelebA.
(SS) measures the percentage of examples in which a model prefers a stereotypical association over an stti-stereot association. The SS of ideal language model is 50 (the closer to 50 the better). It shows that our method can outperform or achieve comparable performance with baseline methods. As for BERT, our method reaches the best (denoted by bold) or second best (denoted by underline) performance in 5 of 6 metrics. Description of baselines can be referred to Appendix. C.2
Backbone
Attribute
Method
SS
LMS
Attribute
Method
sS
LMS
Attribute
Method
SS
LMS
Vanilla
60.28
84.17
Vanilla
$7.03
84.17
Vanilla
59.7
84.17
CDA
59.61
83.08
CDA
56.73
83.41
CDA
58.37
83.24
Dropout
60.66
83.04
Dropout
57.07
83.04
Dropout
59.13
83.04
BERT
gender
INLP
$7.25
80.63
Face
INI.
57.29
83.12
religion
INLP
60.31
83.36
Self-dehias
59.34
84.09
Self-dehius
54.30
84.24
Self debias
57,26
84.23
SentDebias
59.37
84,20
SentDbiiss
57.78
83.95
SentDebias
58.73
84.26
Ours
$2.27
85.45
Qurs
57.24
84.19
Ours
57.85
84.90
Vanilla
62.65
91.01
Vanilla
58.9
91.01
Vanilla
63,26
91.01
CDA
64.02
90.36
CDA
57.31
90.36
CDA
63.55
91..66
Dropout
63.35
90.40
Dropout
57.50
90.40
Dropout
64.17
90.40
GPT-2
gender
INLP
91.62
race
INLP
58.96
91.066
religion
63.95
Self-debias
60.84
89.07
Self-debiar
57.33
89.53
Self-debias
60.45
89.36
SentDebias
56.05
87.43
SeniDe
56.43
91.38
SemDas
59.62
90.53
Ours
60.42
91.01
Ours
60.42
91.01
Ours
58.43
86.13
Table 4: Results with Large Language Models (BERT and GPT-2),
4.4 Analysis
Effectiveness on Different Bias Metrics. We validate the generalization ability of our unlearning method based on different fairesss metrics on the Colored MNIST with bias severity 0.99. In
Tab. [ we compare the performance of unlearning harmful samples based on three different biases:
Counterfactual bias (Co.), Demographic parity bias (De.) 34). and Equal opportunity bias (Fo.) (35)
For each experiment, we report the changes in three biases. We can note that our method is consistently effective on all three bias metrics. Mcanwhile, our counterfactual-asee unlearning can significantly outperform the other two in terms of accuracy, Co., and De., and is comparable with them on Eo..
Asc 1
Co. I
De. I
Eo. J
Avc.
Bias 1
Time(s)]
Vanilla Unlearn by De.
65.17
0.3735
0.5895
0.2235
Vanilla
65.17
0.3735
71.52
0.1796
0.4026
0.0116
Unlearn by Eq. 7
36.87
Unleam by Eo.
71.12
0.1826
0.4217
0.0103
Unlearn by Eq, 8
91.18
0.1023
39.63
Unleam by Co. (Ours)
87.90
0.1051
0.1498
0.0108
Unlearn by Fq. 9 (Ours)
90.42
0.1051
0.059
Table 5: Ablation on Different Biases.
Table 6: Ablation on Unlearning Strategies.
Effectiveness of Unlearn Strategies. We empirically investigate the feasibility of the unlearning mechanism on training and external samples on the Colored MNIST with bias severity 0.99. In Tab. [ we report the results of unlcarning harmful training samples (Eq. 7 unlearning by replacing harmful

---

samples with their bias-conflic helpful samples (Eq. and unlearning with external counterfactual sample pairs (Eq. 23). It can be observed that unlearning in the training dataset can achieve higher accuracy and less bias, and Eq. 21 excels on both metrics. But unlearning with training samples requires much more time and training samples might not be available practice, while unlearning with external samples provides a satisfactory alternative.
Attr:
Method
(9c
Co.
De.
Time(s)
CEqOád
EyChid
83.22
82.71
0.0047
0.0247
0.4469
0.5991
0.0021
0.0125
0.5583
0.0113
Mothod
+Lay
A PPara.
Acc({)) ?
Bias
Time(s)
Gender
Reject
74.63
0.0676
0.2744
0.3140
14.420
Vanilla
51.34
0,4931
Ours
0.0019
0.1438
0.0460
0.0389
Ours
1K
71.19
0.2757
3,75
EqChdd
83.22
0.01.8 39
0.7288
0.0021
0.0105
Our
11K
74.18
0.3134
432.86
CEqOúd
82.88
0.0012
0.6803
0.0051
3,6850
Ours
21
61.45
0.2949
496.44
Reject
0.1156
0.4349
14,290
Durs
83.12
0.0006
0.4219
0.0167
DJ0360
Table Ablation on # MLP Layers.
Table 7: Discussion on Post-processing Methods.
Discussion on Post-processing Methods. We compare our method to post-processing methods, Le., Equalized Odds Post-processing (EqOdd) 35 Calibrated Equalized Odds Post-processing (CEqOdd) 355 and Reject Option Classification (Reject) [82), as shown in Tab. 7 Note these methods only apply to logistic regression. Our method outperforms them in most cases on Adult. is also worth noting that these post-processing methods aimed at a specific fairness measure tend exacerbate unfairness under other measures while our method consistently improves the fairness under different measures.
Ablation the Number of Samples. Fig, demon- strates the sensitivity our unlearning performance w.r.t number of samples on Colored MNIST with bias ratio of 0.99. The accuracy increases and bias decreases incre- mentally with more samples, and becomes steady after the number is beyond 5,000. On the other hand, the unlearn- ing time increases linearly with the number of samples. Additionally, constructing a large number of counterfac- tual samples in practice might be cone-conu as well. Practical usage of the FMD would require a trade-off based on utility requirements.
2000
4505
0303
30303
Figure 4: Ablation on # Samples.
Ablation on Number of Fine-tuning Layers. We explore the impact of unlearning different numbers of layers (i.e., the last (one), two, three MLP) on the Color MNIST, with results in Tab. [1] Interestingly, the accuracy excels with two layers but decreases with three layers. Additionally, fine-tuning multiple layers takes much longer time on computation on much more parameters. is also worth noting that our method could achieve such superior or competing performance even only by updating the last layer in deep models, which calls for more in-depth analysis in the future.
Conclusion and Limitation
Biased behaviors in contemporary well-trained deep neural networks can perpetuate social biases, and also pose challenges to the models' robustness. In response, we present FDM, an all-inclusive framework for fast and effective model debiasing. We explicitly measure the influence of training samples on bias measurement and propose removal mechanism for model debiasing. Comprchensive experiments on multiple datasets demonstrate that our method can achieve superior/competing accuracies with a significantly lower bias well computational cost.
Our work preliminarily explored the application of our method to large language models, as well as more analysis on model faimess from different perspectives, which will be in our future plan. In addition, our method is not applicable to black-box models, which are of high interest in real-world scenarios. Our proposed method requires generating counterfactual pairs with labeled sensitive attributes, while many datasets do not have enough labels. Research on fairness with few/no attribute labels is still in the infant stage [93], and we will further explore it.
10

---

Acknowledgements
This work is supported by the National Natural Science Foundation of China (Grant No. 62106222), the Natural Science Foundation of Zhejiang Province, China No. LZ23F020008) and the Zhejiang University-Angelalign Inc. R&D Center for Intelligent Healthcare.
A Influence Function on Bias and Extension to DNN.
A.1 Deriving the Influence Function Bias.
In this part, we provide detailed derivation of the influence function on bias in Eq. 5 in the main work. We first start from the influence function on parameters, we can also be referred to [4).
Assuming there are training samples 2 #2..., Zn. where 2; (I,, vt). and L(z, 0) represent loss function of sample z under the model parameters 0, then the trained / is given by:
argmin, R(0) argming
L(2,
(12)
Study the impact of changing the weight of a training sample 2 on the model parameters 0. If we increase the weight of this sample in the training set by then the perturbed parameters Q..- obtained according to ERM (Empirical Risk Minimization) will be:
ò,. arg min R(0) + eL,
(13)
Define the parameter change A, 8.. 0, and note that, as 0 doesn't depend on c, the quantity we seek to compute can be written in terms of it:
di dA. de de
(14)
Since Òr,2 minimizer of R(0), therefore it satisfies the first-order derivative condition, which means the first-order derivative with respect to 0 zero:
VR(O..)) + (z,
(15)
Next, since òr,z > Ô as e > 0, we perform a Taylor expansion of the right-hand side:
V R + + + v + +
where we have dropped of [| terms. Solving for A., get:
~ eV
+ eV 0)}.
Since i minimizes Rt we have VR(0) = 0. Dropping o(e) terms, we have
A. -2R(0))
(16)
Note that it is assumed that R twice-diffierentialee and strongly convex in 0. we define:
Hg - n2
(2,
(17)
exists and is positive definite. This guarantees the existence of H The final influence function can be written as:

---

cd0r.ze de
Jup.params (2)
= - V
(18)
l==0
Considering B()) measured on any AA with any Den, our goal is to quantify how each training point 2 in the training set Drr contributes to B(0). We apply the chain rule on Eq. 18
Tup.bias (Zk, B(O)) dB((e...)) dBc,=n.
= -VVBB(O)H (Vo0)HT'VoLaa0 VoL(EE.
(19)
d0c,zu de
c=0
Intuitively, this equation can be understood in parts: the latter part calculates the impact of removing on the parameters. The former part corresponds to the derivative of bias with respect to parameters, assessing how changes in parameters affect the bias. Hence, this equation quantifies the influence of removing on the bias.
A.2 Influence at Non-Convergence
In this part, we provide the theoretical proof of the feasibility of the influence function for deep networks (non- con-eennerg)nt) in [32]. In the derivation of the influence function, it's assumed that could be the global minimum. However, if 0 obtained in deep networks trained with SGD in a non-convex setting, it might be local optimum and the exact influence can hardly be computed. Here we provide the proof in [32] on how can influence function approximate the parameter change in deep networks.
Consider a training point z. When the model parameters are close to but not at local minimum, Iup.params(? (=) approximately equal constant (which does not depend on 2) plus the change in parameters after upweighting & and then taking a single Newton step from The high-level idea that even though the gradient of the empirical risk at is not O, the Newton step from 0 can be decomposed into component following the existing gradient (which does not depend on the choice of 2) and a second component responding to the upweighted! = (which T Tup.params tracks).
Let def Σ (oL 8) be the gradient of the empirical risk 0; since 0 is not local minimum, 7
* O. After upweighting z by e, the gradient at 0 goes from + + eV,L(2, 0), and the empirical Hessian goes from H3 + H + + LZ, 0). A Newton step from @ therefore changes the parameters by:
Nr, - H [2T+,V)L(220)
"[
(20)
Ignoring terms in cg. 2², and higher, we get Ne,: -H,² (g
( (VL( 0)) Therefore, the actual
change due to a Newton step Ne,; is equal to a constant - - 9 (that doesn't depend on 2) plus c times Lup.porami (2) H VeL(z, 0) (which captures the contribution of 2).
B Bias Removal via Machine Unlearning
B.1 A Closer Look the Toy Experiment
We conduct an experiment on a logistic regression task using Eq. 19 We simplify the Colored MNIST classification task to binary classification problem of distinguishing between only digits 3 and 8, on a training set with a bias ratio of 0.95, 0.9 and 0.8, and balanced test set. To be specific, a bias ratio color3, digit8 color8> and 5% bias-conf
of 0.95 means 95% bias-aligned samples <digit3_color3, samples <digit3 color8, digit8 _oceell__ol33___1111 in the training set. We trained a regularized logistic regressor: argmin Σi= 1w xr, y/ + A|[w|| Fig. 5 (a) illustrates the classification results of the vanilla classifier (trained on the 0.95-biased train set) part of test samples. We denote Digit by shape (triangle and rectangle) and Color by color (yellow and green). The solid line represents the learned classification boundary and the dotted line represents the expected classification boundary. It can
12

---

be observed that the learned classifier tends to classify digits according to their color. Based on the observed bias, we employ Eq. I9 to evaluate how each training sample contributes the bias. In Tig. we select and visualize the most helpful (reduce bias) and harmful (increase bias) samples. We found that the most harmful samples are bias-aligned while helpful samples are bias-conflicting. With this inspiration, We further visualize the influence distribution of training samples in Fig. [ We denote the bias-conflcting sample with "red dot" and the bias-aligned sample with "blue dot". We find that most bias-aligned samples tend to be harmful while bias-conflicting samples tend to be helpful. This pattern is consistent across different ratios of bias-confiictnn samples. Additionally, the influences of helpful samples are larger than those of harmful ones. Visualizations produced with randomly 500 samples from the training set.
Dijna Coler
Digid D
8
3
8
3
8
3
Digitt 3
83
3
3
3
8
8
3
8
(a)
(b)
Figure 5: (a) Illustration of the learned pattern on our toy dataset. (b) Visualization of helpful samples (top row) and harmful samples (bottom row).
1001
Gai La (hinng SAEIOON aergee
Boiing Sarapii
2005
Kran cal
100K
0204
10
20:0
(a)
(b)
(c)
Figure 6: Influences of training samples with bias ratios of (a) 0.8, (b) 0.9, (c) 0.95.
Inspired by this observation, unlearning strategy is further refined. Hence, propose straight- forward solution that further mitigates the influence of a harmful sample with a bias-conflicting sample. Consequently, update the parameters unlearn the harmful samples by:
Onew 0+ Σ H (V 0) - v ( VoL(z,,0), 0)),
(21)
where Σạ denotes the bias-conflicting sample of zg. Following the explanation in influence theory (32], our unlearn mechanism removes the effect of perturbing training point (ā, x, y) to (a,x, x. y). In other words, we not only remove the influence caused by harmful sample zg. but further ensure fairness with the corresponding counterfactual sample Σμ.
To further illustrate the functionality of Eq. 21 measure the influences of the selected harmful and helpful sample pairs by:
Lup,bias (2k, B(θ) V)
-V,L(E,))))
(22)
with visualizations in Fig. 7 By calculating the difference between the harmful samples and helpful samples, the biased effect is significantly amplified. In this way, the unlearning becomes more effective.
B.2 Deriving Alternative Efficient Unlearn
In the above sections, the unlearning process based on the assumption that we could access the original training sample 2 identify and evaluate biases and then forget them. However, in practice,
13

---

Nos Sompln
Bei Setee Sarpiin
201
6408
1004
6802
CX
0402
(a)
Figure 7: Influences of selected training sample (counterfactual) pairs in Eq. with bias ratios of (a)
0.8, (b) 0.9, (c) 0.95.
the training set might be too large or even unavailable in the unlearning phase. In response, we further propose to approximate the unlearning mechanism with small external dataset. As the influence to be removed can be obtained from the change of the protected attribute, we can construct the same modification to the protected attribute external samples. In particular, we employ external dataset Dew as in Section 3.1 in the main work to construct counterfactual pairs for unlearning, which redefines Eq. 21 as:
0 fnww = 0 "H VgL(e, 0))
(23)
As Dew can be easily obtained from an external dataset rather than the training set, e.g., the test set, the practical applicability of our method could be significantly enhanced.
We further visualize the influence of samples in the balanced external dataset in Fig. (a). In the balanced dataset, the ratio of bias-aligned and bias-conflicting samples is about 50%. We can observe that the pattern of harmful bias-aligned samples and helpful biascconfiicting samples in the external dataset is similar to the training set. By comparing the influence of counterfactual pairs in the external dataset (Fig. [ (b)) and the training set (Fig. 8 (c)), , we can find the distributions are similar, which proves the feasibility of our alternative unlearning.
S01 langle
2014
Biscang kinges
0.00
24
0000
(a)
(b)
Figure 8: Influences of samples in (a) external dataset, (b) external dataset (with counterfactual sample pairs), (c) training set.
B.3 Alternative Efficient Unlearn vs. Directly Unlearn Training Data.
Tackling the problem that, in practice, the training set might be too large or even unavailable in the unlearning phase, propose alternative unlearning strategy in Sec. 3.3 in the main work. We approximate the change of the protected attribute by constructing the same modification to the protected attribute on external samples. Then we unlearn the same perturbation from the model with the constructed external dataset. In Sec. 4.4 in the main work, we provide the performance comparison of alternative efficient unlear (Ours) and directly unleam training data (Eq. 7 and Eq. 8 in the main work).
In this section, we further compare the performance of alternative unlearning on Adult and simplified Colored MNIST on logistic regression, with results reported in Tab. and Tab. TO] We can find that in five experiments, alternative learning achieves comparable performance with the two directly
14

---

unlearning strategies. Comparing Eq. 7 and Eq. 8, we can find that the modified Eq. 8 reaches convergence taking less iteration. The number of samples used is 200 for the two datasets.
Attr.
Method
Bias 4
Time(s)
# Iter.
Acc.(%) f
Vanilla
0.0134
0.8259
Fq. 7
0.0002
1394
39
0.8249
race
Eq. 8
0.0002
1398
10
0.8311
Ours
0.0002
0.0039
0.8229 46
Vanilla
0.0494
0.8259
Eq.
0.0001
1386
212
0.8234
gender
Eq.
0.0001
1390
186
0.8252
Ours
0.0006
0.0038
252
0.8232
Table 9: Altemnatvee Efficient Unlearn on Adult.
Attr.
Method
Bias
Time(s)
Iter.
Acc.(%) f
Vanilla
0,4624
0.5922
Eq.
0.1642
183
201
0,8548
0.95
Ey.
0. 1624
183
157
0.8617
Ours
0.1496
0.0017
0.8594 74
Vanilla
0.4086
0.6517
Eq.
0,1599
183
212
0,9102
0.9
Eq. 0.1562
183
185
0,9211
Ours
0.1658
0.0018
0,9113
Vanilla 0.3735
0.6517
Eq. 0,1622
183
187
0.9241
0.8
Eq. 0,1617
183
169
0,9312
Ours
0,1611
0.0017
67
0.9244
Table 10: Alternative Efficient Unlearn on Colored MNIST.
B.4 Efficient Unlearning for Deep Networks.
In our experiment, We are inspired by Sec. 5.1 and Sec. 5.2 in [32 which keep all but the top layer deep networks frozen and measure influence. We follow this setting so that the finetuning on deep networks can be simplified logistic regression. In this part, investigate the difference in finetuning different numbers layers. The experiment is conducted on Colored MNIST with MLP with 3 hidden layers.
Discussion on Different Fine-tuning Strategies. Following Sec. 4.4 in the main work, we explore the impact of unlearning different numbers of layers (i.e., the top one, two, three MLP) on the Colored MNIST with three bias ratios, with results in Tab. [T Interestingly, the accuracy excels with two layers but decreases with three layers. Additionally, fine-tuning multiple layers takes much longer time on computation on much more parameters. It also worth noting that our method could achieve such superior or competing performance even only by updating the last layer in deep models, which calls for more in-depth analysis in the future.
B.5 Effectiveness of Pre-calculating Hessian.
In Sec. 3.4 in the main work, we propose pre-calculate the inverse Hessian before performing unlcarning. In this way, we approximate the Hessian as it should change with model parameters, however, we prevent the large computation cost of updating and inverting the Hessian at every iteration. In this part, we empirically illustrate the effectiveness of our approximation. Experiments are conducted on Colored MNIST and Adult datasets with logistic regression tasks, with results provided in Tab. and Tab. [3 "wo/" denotes unlearning without pre-calculation. It can be observed that unlearning with or without can achieve comparative performance on bias and accuracy. However, our method can save about 40% run time on Adult and 97% run time on Colored MNIST. The reason that the number of parameters Colored MNIST much larger than Adult, so that the calculation of inverse Hessian makes up larger proportion of the total run time.
15

---

Ratio
Method
Lay.
# Para.
Acc(%)) 1
Bias |
Time(s)
Vanilla
38.59
0.5863
Ours
1000
62.34
0.3415
3.750
0.995
Ours²
11000
64.18
0.3378
439.34
Ours
21000
55.32
0,3519
504. 12
Vanilla
$1.34
0.4931
Ours'
1000
71.19
0.2757
3,750
0.99
Ours
11000
74
0.3134
432.86
Ours
21000
61.45
0.2949
496.44
Vanilla
77.63
0.2589
0.95
Ours'
1000
86.39
0.1849
3.975
Ours
11000
87.34
0.1902
434.25
Ours 3
21000
86.47
0.1914
501.24
Table 11: Ablation on MLP Layers.
Attr.
Method
Bias 1
Time(s)
# Iter.
Acc.(%)
Vanilla
0.0134
0.8259
wo/
0.0002
0.0064
42
0.8229
Ours
0.0002
0.0039
0.8229 46
Vanilla
0.0494
0.8259
gender
wo/
0.0006
0.0066
149
0.8243
Ours
0.0006
0.0038
252
0.8232
Table 12: Efficient Hessian Computation on Adult.
C
Experiment Details
C.1
Dataset
Colored MNIST. Colored MNIST is constructed based on the MNIST dataset [68] designed for digit classification tasks. To build a biased correlation, ten distinct RGB values are applied on grayscale digit images 3 69 70). Digit and color distribution are paired to build biased correlations in the training Bias-aligned samples
defined as fixed combinations of digit and color like Digit Color while bias-conflict samples are defined as other combinations like Digit I, random Color in 2-10. our Experiment, we use 3 different training sets by setting different bias ratios 0.995, 0.99, 0.95 for biased-aligned training samples where the ratio represents the partition of bias-aligned samples the training set. The higher the ratio, the higher the degree of bias. The split of the training set, test set, and external set is 60000, 10000, and 10000.
CelebA. CelebA dataset 77 is face recognition with 40 types of attributes like gender, age (young or not). and lots of facial characteristics (such as hair color, smile, beard). The dataset contains total of 202.599 images which, following the official train validation split, consists of 162.770 images for training and 9,867 images for testing. We choose Gender as the protected attribute, Hair-color (blonde hair not) and Attractive the target attribute following 87. The number of selected samples for the two target attributes is 200 and 182, which are split from the test set.
Ratio
Method
Bias 1
Time(s)
# Iter.
Acc.(96) f
Vanilla
0.4624
0.5922
0.95
wn/
0.1490
0.0556
59
0.8674
Ours
0.1496
0.0017
0.8594 74
Vanilla 0.6517
0.4086
0.9
wo/ 0.1698
0.0498
46
0.9093
Ours
0.1658
0.0018
0.9113 77
Vanilla 0.2857
0.6915
0.8
wo/ 0.1689
0.0517
34
0.9264
Ours
0.1611
0.0017
67
0.9244
Table 13: Efficientt Hessian Computation Colored MNIST.
16

---

Adult Income Dataset. The Adult dataset is publicly available dataset in the UCI repository 72 based on 1994 U.S. census data. The goal of this dataset is to successfully predict whether an individual earns more or less than $50,000 per year based on features such as occupation, marital status, and education. We follow the processing procedures in [41]. In our experiment, we choose gender and race as protected attributes following 73 74]. We split 200 samples from the test set as the external dataset.
C.2 Baselines
For the sanity check experiment on a toy Colored MNIST dataset, we use a vanilla logistic regression model as the baseline. For experiments with deep networks, we compare method with one pre-processing baseline Reweigh (77), 6 in-processing debiasing baselines (LDR 25 LIF
(78).
Rebias 779 DRO 77 SenSEI [80). and SenSR [8]) and 4 post-processing baselines (EqOdd [5
utilizes the influence function to reweight the training sample. in order to re-train fair model targeting group fairness metries (equal opportunity and demographic parity), Among in processing basclines, LDR, LIF, Rebias, and DRO are designed explicitly target higher accuracy (on unbiased test set worst-group test set) and implicitly target fairness, while SenSEI and SenSR are designed to target individual fairness, EgOdd, CEqOdd and Reject are designed to target different group fairness metries (equal odd and demographic parity), while 83 proposes post- post-processing algorithm for individual fairness.
CEqOdd 35 Reject (2) and PP-IF
(3
Baselines for experiment on Large Language Model. We evaluate several baseline debiasing meth- ods. Counterfactual Data Augmentation (CDA) [94] adjusts a corpus for balance by exchanging words indicative of bias (such as "he/she") within the dataset. This newly balanced corpus is then typically utilized for additional model training to reduce bias. Dropout [95] suggests enhancing dropout rates and incorporating an extra pre- pre-training stage for debiasing. SentenceDebias [8] aims to derive unbiased representations by removing biased projections on presumed bias subspace from the original sentence representations. Iterative Nullspace Projection (INLP) [86] employs a projection- based approach to exclude protected attributes from representations. Finally, Self-
Debias [87] advocates for the use of model's intrinsic knowledge avert the generation of biased text.
D Discussion
D.1 Dataset Generation
In our experiments, utilize approximated counterfactual samples for CelebA due the unavailabil- ity of strict counterfactual data. Based on attribute annotations, we select images with the same target attributes but opposite sensitive attributes, while maintaining other attributes much as possible. Our method achieves the best results on the worst-case group, indicating that the approximated counterfactual samples can also effectively enhance fairness predictions. Similar to our approach, (66] proposes to select pairs of counterfactual images based on attribute annotations on the CUB dataset to produce counterfactual visual explanations. Their experiments also show that neural networks can discern major differences (such as gender in our work) between images without strict control (such as background).
For real-world visual datasets (like facial dataset or ImageNet), the unavailability of strict coun- terfactual data is common challenge. Existing methods propose train generative model to create counterfactual images with altered sensitive attributes (97-99), which seems to be a viable approach for obtaining counterfactual datasets for more diverse vision applications. Building upon these methods, we will extend our approach to more scenarios.
D.2 Inffuence Estimation
In our unlearning experiment, we freeze the parameters of all other layers except the top layer. Previous work investigates the estimation accuracy of the influence function on both multi-layer and single-layer setups [100]. It performs case study on the MNIST. For each test point, they select 100 training samples and compute the ground-truth influence by model re-training. Results show that estimations are more accurate for shallow networks.
17

---

Our results in Tab. 7 in the main manuseript also validate this point. When applying FMD to three-layer neural net, the performance on either accuracy or bias becomes worse. This could potentially be attributed to the inaccurate estimation of influence function on multi-layer neural nets. In our experiments, we adhere to the set-up in [2] where the influence function is only applied to the last layer of deep models, which proves to be effective.
As verified in 33 100 [011, influence estimation matches closely to leave-one-out retraining for logistic regression model. As discussed in 197], measuring influence score for the last layer can be regarded as calculating influence from a logistic regression model on the bottleneck features (Sec. 5.1 in the main manuscript). The same setup is followed by many influence function- based works [02 103 and proves to be effective.
D.3
Computational Complexity
As for bias-effect evaluation, with n training points and 0 E Hd directly computing Eq. 5 (in the main manuscript) requires O(nd + ndd)) operations. In our experiment, only activate the last layer so that is small. However, when the number of training samples is very large, performing Eq. 5 is expensive. As for the debiasing phase, it requires Onnd kd?) operations, where k is the number of samples to unlearn. Note that hessian is calculated in the bias-effect evaluation phase, can be directly used in the debiasing phase. Hence, the overall computational complexity using Eq. 7 and Eq. 8 is Onnd2 + kd? + ndd).
However, in our proposed alternative debiasing method, we only utilize an external counterfactual dataset with small number of k. Hence, we can omit the (nd2) operations to compute influences and rank the training samples. Hence, the overall computational complexity using Eq. 9 (Ours) is O(nd² + kef) Experimental comparison results can referred to Tab, 5 (in the main manuscript). Debiasing with Eq. 8 takes about 500x more time than Eq. 9 (in the main manuscript).
E
Preliminaries
E.1
Influence Function
The origins of influence-based diagnostics can be traced back to important research papers such as 664 [04 [05]. More recently, Koh and Liang [32] introduced the concept of influence functions to large-scale deep learning, which numerous publications have since followed up. In their work,
32 advocated for the use of an approximation, Eq. [3 to estimate the change in loss when a small adjustment is made to the weights of the dataset. In practical applications involving deep models, the
Hessian matrix (H) cannot be stored in memory or inverted using standard linear algebra techniques. However, by considering a fixed vector (v), the Hessian vector product (HVP), Hv, can be computed in O(bp) time and memory [106), where represents the batch size and determines the number of training examples used to approximate H (for given loss function L.. The iterative procedure LISSA 107.. employed by (32, relies on repeated calls to the HVP to estimate the inverse HVP.
E.2
Counterfactual Fairness
Counterfactual fairness, a relatively new concept, has emerged as a means to measure fairness at an individual level [31]. The fundamental idea behind this approach is to determine the fairness of a decision for an individual by comparing with the decision that would have been made in an alternate scenario where the individual's sensitive attributes possessed different values. This concept builds upon earlier work [08]. which introduced a causal framework for learning from biased data by examining the relationship between sensitive features and the data. Recent advancements in deep learning have further contributed to this field, with novel approaches [46] 45 [09 98) proposing methods to enhance the accuracy of decision-making models by improving the approximation of causal inference, particularly when dealing with unobserved confounding variables.
E.3 Demographic Parity and Equal Opportunity
Demographic Parity 34 A predictor Y satisfies demographic parity if P(Y A = 0) = P(Y]A= 1). where the sensitive attribute. The likelihood of positive outcome should be the same regardless of whether the person is in the protected (c.g., female) group.
18

---

Equal Opportunity [35] "A binary predictor Y satisfies equal opportunity with respect to A and Y if P(Y 1 1) P 1| = = =) This means that the probability of a person in positive class being assigned to positive outcome should be equal for both protected and unprotected (female and male) group members.
References
[1] M. Du, F. Yang, N. Zou, and X. Hu, "Faimess in deep learning; A computational perspective," IEEE Intelligent Systems, vol. 36, no. 4, pp. 25-34, 2020.
[2] N. Mehrabi, F. Morstatter, N. Saxena, K. Lerman, and A. Galstyan, "A survey on bias and fairness in machine learning," ACM Computing Surveys (CSUR), vol. 54, no. 6, pp. 1-35, 2021.
[3] B. Kim, H. Kim, K. Kim, S. Kim, and J. Kim, "Learning not to learn: Training deep neural networks with biased data," in IEEE Conference Computer Vision and Pattern Recognition, 2019.
[4] R. Geirhos, J.-H. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge, and F. A. Wichmann, "Shortcut learning in deep neural networks," Nature Machine Intelligence, vol. 2, no. 11, pp. 665-673, 2020.
[5] S. Sagawa, A. Raghunathan, P. W. Koh, and P. Liang, "An investigation of why overparameter- ization exacerbates spurious correlations," in International Conference on Machine Learning, pp. 8346-836 PMLR, 2020.
[6] Geirhos, P. Rubisch, Michaelis, M. Bethge, A. Wichmann, and W. Brendel, "Imagenet- trained cnns are biased towards texture; increasing shape bias improves accuracy and robust- ness," " arXiv preprint arXiv: 1811111 2018.
[7] S. Sagawa, P. W. Koh, T. B. Hashimoto, and P. Liang. "Distributionally robust neural net- works for group shifts: On the importance of regularization for worst-case generalization, in International Conference on Learning Representations, 2020.
[8] E. Tartaglione, C. A. Barbano, and M. Grangetto, "End: Entangling and disentangling deep representations for bias correction." in Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 13508- -3507,1 June 2021.
[9] S. Sagawa, A. Raghunathan, P. W. Koh, and Liang, "An investigation of why overparameter- ization exacerbates spurious correlations," in International Conference on Machine Learning, pp. 8346 8356, PMLR, 2020.
[10] T. Brennan, W. Dieterich, and B. Ehret, "Evaluating the predictive validity of the compas risk and needs assessment system," Criminal Justice and behavior, vol. 36, no. pp. 21 40, 2009.
[11] J. F. Mahoney and J. M. Mohen, "Method and system for loan origination and underwriting." Oct. 23 2007. US Patent 7,287,008.
[12] M. Bogen and A. Rieke, "Help wanted: An examination of hiring algorithms, equity, and bias," 2018.
[13] D. Pessach and E. Shmueli, "A review on fairness in machine learning." ACM Computing Surveys (CSUR), vol. 55, no. 3, pp. 1-44, 2022.
[14] O. Parraga, M. D. More, C. M. Oliveira, N. Gavenski, S. Kupssinskü, A. Medronha, L. V. Moura, G. S. Simões, and R. C. Barros, "Debiasing methods for fairer neural models in vision and language research: A survey," arXiv preprint arXiv:2211.05617 2022.
[15] Y. Li and N. Vasconcelos, "Repair: Removing representation bias by dataset resampling." in Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pp. 9572-9581, 2019.
19

---

[16] M. Tanjim, R. Sinha. K. K. Singh. S. Mahadevan, D. Arbour, M. Sinha, G. W. Cottrell, et al., "Generating and controlling diversity in image search." " in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 41 411-419, 2022.
[17] F. Calmon, D. Wei, B. Vinzamuri, K. Natesan Ramamurthy, and K. R. Varshney, "Optimized pre-processing for discrimination prevention," " Advances in neural information processing systems, vol. 30, 2017.
[18] M. Feldman, S. A. Friedler, J. Moeller, C. Scheidegger, and S. Venkatasubramanian, "Certify- ing and removing disparate impact," in proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pp. 259-268, 2015.
[19] F. Kamiran and T. Calders, "Data preprocessing techniques for classification without discrimi- nation," Knowledge and information systems, vol. 33, no. 1. pp. 1-33, 2012.
[20] M. B. Zafar, I. Valera, M. Gomez Rodriguez, and K. P. Gummadi, "Faimess beyond disparate treatment & disparate impact: Learning classification without disparate mistreatment," in Proceedings of the 26th international conference on world wide web, pp. 1171-1180, 2017.
[21] E. Adeli, Q Zhao, A. Pfefferbaum, E. V. Sullivan, L. Fei-Fei, J. C. Niebles, and K. M. Pohl, "Representation learning with statistical independence to mitigate bias," in Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pp. 2511-25221 2021.
[22] P. Dhar, J. Gleason, A. Roy, C. D. Castillo, and R. Chellappa, "Pass: protected attribute suppression system for mitigating bias in face recognition." " in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15087-15096, 2021.
[23] M. Alvi, A. Zisserman, and C. Nellåker, "Turning blind eye: Explicit removal of biases and variation from deep neural network embeddings." " in Proceedings of the European Conference Computer Vision (ECCV) Workshops, pp. 0-0, 2018.
[24] H. Bahng, S. Chun, S. Yun, J. Choo, and S. J. Oh, "Learning de-biased representations with biased representations," " in International Conference on Machine Learning, pp. 528-539, PMLR, 2020.
[25] J. Lee, E. Kim, J. Lee, Lee, and J. Choo, "Learning debiased representation via disentangled feature augmentation," Advances in Neural Information Processing Systems, vol. 34, pp. 25123- 25133, 2021.
[26] T. Wang. J. Zhao, M. Yatskar, K.-W. Chang, and V. Ordonez, "Balanced datasets are not enough: Estimating and mitigating gender bias in deep image representations." in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5310-5319, 2019.
[27] M. Hardt, E. Price, and N. Srebro, "Equality of opportunity in supervised learning," Advances in neural information processing systems, vol. 29, 2016.
[28] A. K. Menon and R. C. Williamson, "The cost of fairness in binary classification," in Confer- ence on Fairness, accountability and transparency, pp. 107-118, PMLR, 2018.
[29] S. Corbett-Davies, F. Pierson, A. Feller, S. Goel, and A. Huq, "Algorithmic decision making and the cost of fairness," " in Proceedings of the 23rd aem sigkdd international conference on knowledge discovery and data mining, pp. 797-806, 2017
[30] P. Saleiro, B. Kuester, L. Hinkson, J. London, A. Stevens, A. Anisfeld, K. T. Rodolfa, and R. Ghani, "Aequitas: A bias and fairesss audit toolkit," arXiv preprint 1rXiv:1811.05500 2018.
[31] M. J. Kusner, J. Loftus, C. Russell, and R. Silva, "Counterfactual fairness," Advances in neural information processing systems, vol. 30, 2017.
[32] P. W. Koh and P. Liang, "Understanding black-box predictions via influence functions," in International conference on machine learning, pp. 1885-1894, PMLR, 2017.
[33] Z. Cao, J. Wang, S. Si, 7. Huang, and J. Xiao, "Machine unlearning method based on projection residual," arXiv preprint 22Xi:2209..12766 2022.
20

---

[34] C. Dwork, M. Hardt, T. Pitassi, O. Reingold, and R. Zemel, "Fairness through awareness." in Proceedings of the 3rd innovations in theoretical computer science conference, pp. 214-226, 2012.
[35] M. Hardt, E. Price, and N. Srebro, "Equality of opportunity in supervised learning," Advances in neural information processing systems, vol. 29, 2016.
[36] S. Jung, S. Chun, and T. Moon, "Learning fair classifiers with partially annotated group labels," " in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10348-10357, 2022.
[37] S. Verma and Rubin, "Fairess definitions explained," in Proceedings of the international workshop on software fairness, pp. 1-7, 2018.
[38] M. Kearns, S. Neel, A. Roth, and Z. S. Wu, "Preventing faimess gerrymandering: Auditing and learning for subgroup fairness," in International conference on machine learning, pp. 2564- 2572, PMLR, 2018.
[39] C. Dwork, M. Hardt, T. Pitassi, O. Reingold, and R. Zemel, "Fairness through awareness," in Proceedings of the 3rd innovations in theoretical computer science conference, pp. 214-226, 2012.
[40] M. Joseph, M. Kearns, J. Morgenstern, S. Neel, and A. Roth, "Rawlsian fairness for machine learning," arXiv preprint arXiv: 161000955, vol. 1, no. 2. p. 19, 2016.
[41] C. Louizos, K. Swersky, Li, M. Welling, and R. Zemel, "The variational fair autoencoder," arXiv preprint arXiv: 15::::551100000 1.00830, 2015.
[42] W. Fleisher, "What's fair about individual fairness?"" in Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 480-490, 2021.
[43] Binns, "On the apparent conflict between individual and group fairness," in Proceedings of the 2020 conference on fairness, accountability, and transparency, pp. 514-524, 2020.
[44] S. Chiappa, "Path-specific counterfactual fairness," in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 7801 -7088, 2019.
[45] S. Garg, V. Perot, N. Limtiaco, A. Taly, E. H. Chi, and A. Beutel, "Counterfactual fairness in text classification through robustness," in Proceedings of the 2019 AAAI/ACM Conference AI, Ethics, and Society, pp. 219-226, 2019.
[46] Y. Wu, Zhang, and X. Wu, "Counterfactual fairness: Unidentification, bound and algorithm." in Proceedings of the Twenty-Eighth International Joint Conference Artificial Intelligence, 2019.
[47] N. Kilbertus, P. J. Ball, M. Kusner, Weller, and R. Silva, "The sensitivity of counterfactual fairness to unmeasured confounding," in Uncertainty in artificial intelligence, pp. 616-626, PMLR, 2020.
[48] F. Kamiran, A. Karim, and X. Zhang, "Decision theory for discrimination-awarr classification,"" in 2012 IEEE 12th international conference on data mining, pp. IEFF, 2012.
[49] C. Dwork, N. Immorlica, A. T. Kalai, and M. Leiserson, "Decoupled classifiers for group-fair and efficient machine learning," Conference on fairness, accountability and transparency, pp. 119-133, PMLR, 2018
[50] N. Kallus, X. Mao, and A. Zhou, "Assessing algorithmic fairness with unobserved protected class using data combination," Management Science, vol. 68, no. 3. pp. 1959-1981, 2022.
[51] T. Baumhauer, Schöttle, and M. Zeppelzauer, "Machine unlearning: Linear filtration for logit-based classifiers," Machine Learning, vol. 111, no. 9, pp. 3203-3226, 2022.
[52] Q. P. Nguyen, R. Oikawa, D. M. Divakaran, M. C. Chan, and B. K. H. Low, "Markov chain monte carlo-based machine unlearning: Unlearning what needs be forgotten," arXiv preprint arXiv:2202. 13585, 2022.
21

---

[53] A. Tahiliani, V. Hassija, V. Chamola, and M. Guizani, "Machine unlearning: Its need and implementation strategies," " in 2021 Thirteenth International Conference on Contemporary Computing (IC3-2021), pp. 241-246, 2021.
[54] M. Magdziarezyk, "Right to be forgotten in light of regulation (eu) 2016/679 of the european parliament and of the council of 27 april 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing directive 95/46/ec," in 6th International Multidisciplinary Scientific Conference on Social Sciences and Art Sgem 20/9, pp. 177-184, 2019.
[55] L. Bourtoule, V. Chandrasekaran, C. Sraqueete-Choo H. Jia, A. Travers, B. Zhang, D. Lie, and N. Papernot, "Machine unlearning," in 2021 IEEE Symposium on Security and Privacy (SP), pp. 141-159, IEEE, 2021.
[56] J. Brophy and D. Lowd, "Machine unlearning for random forests," in International Conference Machine Learning, pp. 1092- 104, PMLR, 2021.
[57] A. Golatkar, A. Achille, and S. Soatto, "Eternal sunshine of the spotless net: Selective forgetting in deep networks," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9304-9312, 2020.
[58] A. Mahadevan and M. Mathioudakis, "Certifiable machine unlearning for linear models," arXiv preprint arXiv:2106. 15093, 2021.
[59] Y. Wu, E. Dobriban, and S. Davidson, "Deltagrad: Rapid retraining of machine learning models," in International Conference on Machine Learning, pp. 10355-10366, PMLR, 2020.
[60] S. Neel, A. Roth, and S. Sharifi-Malvajerdd "Descent-to-delete: tradient-based methods for machine unlearning." Algorithmic Learning Theory, pp. 931-962, PMLR, 2021.
[61] Z. Cao, J. Wang, S. Si, Z. Huang, and J. Xiao, "Machine unlearning method based on projection residual," arXiv preprint Sttiv:2209.1527700 2022.
[62] Z. Izzo, M. A. Smart, K. Chaudhuri, and J. Zou, "Approximate data deletion from machine learning models," in International Conference on Artificial Intelligence and Statistics, pp. 2008- 2016, PMLR, 2021.
[63] Guo, T. Goldstein, A. Hannun, and L. Van Der Maaten, "Certified data removal from machine learning models," arXiv preprint arXiv: 1911000910 2019.
[64] R. D. Cook and S. Weisherg, Residuals and influence in regression. New York: Chapman and Hall, 1982.
[65] X. Han, B. C. Wallace, and Y. Tsvetkov, "Explaining black box predictions and unveiling data artifacts through influence functions," arXiv preprint 2:Xi::2005.06666, 2020.
[66] A. Peste, D. Alistarh, and C. H. Lampert, "Ssse: Efficiently erasing samples from trained machine learning models," arXiv preprint arXiv:2107.03860, 2021.
[67] B. A. Pearlmutter, "Fast exact multiplication by the hessian," " Neural computation, vol. 6, no. 1, pp. 147-160, 1994.
[68] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, "Gradient-basee learning applied to document recognition," " IEEE, vol. 86, no. 11, pp. 2278 -7244, 1998.
[69] Y. Li and N. Vasconcelos, "Repair: Removing representation bias by dataset resampling." in Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 9572- 9581, 2019.
[70] H. Bahng, S. Chun, S. Yun, J. Choo, and S. J. Oh, "Learning de-biased representations with biased representations." International Conference on Machine Learning, 2020.
[71] Z. Liu, P. Luo, X. Wang, and X. Tang, "Deep learning face attributes in the wild," in IEEE International Conference on Computer Vision, 2015.
22

---

[72] A. Frank, A. Asuncion, et al., "Uci machine learning repository, 2010," URL http://archive ics. uci. edu/ml, vol. 15, p. 22, 2011.
[73] Zemel, Y. Wu, K. Swersky, T. Pitassi, and C. Dwork, "Learning fair representations"" in International conference machine learning, pp. 323-333, PMLR, 2013.
[74] T. Kamishima, S. Akaho, and J. Sakuma, "Fairness-aware learning through regularization approach." in 2011 IEEE 11th International Conference on Data Mining Workshops, pp. 643- 650, IEEE, 2011.
[75] R. K. Bellamy, K. Dey, M. Hind, S. C. Hoffman, S. Houde, K. Kannan, P. Lohia, J. Martino, S. Mchta, A. Mojsilović, et al., "Ai fairness 360: An extensible toolkit for detecting and mitigating algorithmic bias," IBM Journal of Research and Development, vol. 63, no. 4/5, pp. 1,1,
[76] M. Nadeem, A. Bethke, and S. Reddy, "Stereoset: Measuring stereotypical bias in pretrained language models," arXiv preprint arXiv:2004.0944 09456, 2020.
[77] P. Li and H. Liu, "Achieving fairness no utility cost via data reweighing with influence," in International Conference on Machine Leurning, pp. 12917-12930, PMLR, 2022.
[78] J. Nam, H. Cha, S. Ahn, J. Lee, and J. Shin, "Learning from failure: Training debiased classifier from biased classifier," in Advances in Neural Information Processing Systems, 2020.
[79] H. Bahng. Chun, S. Yun. Choo, and S. J. Oh, "Learning de-biased representations with biased representations," in International Conference on Machine Learning (ICML), 2020.
[80] M. Yurochkin and Y. Sun, "Sensei: Sensitive set invariance for enforcing individual faimess,"" arXiv preprint arXiv:2006.14435 2020.
[81] M. Yurochkin, A. Bower, and Sun, "Training individually fair ml models with sensitive subspace robustness," arXiv preprint (rXXiv:907.00000 2019.
[82] F. Kamiran, A. Karim, and X. Zhang, "Decision theory for discrimination-aware classification,"" in 2012 IEEE 12th International Conference on Data Mining, pp. 924-929, 2012.
[83] F. Petersen, D. Mukherjee, Sun, and M. Yurochkin, "Post-processing for individual fairness," Advances in Neural Information Processing Systems, vol. 34, pp. 25944-25955. 2021.
[84] R. Zmigrod, S. J. Mielke, H. Wallach, and R. Cotterell, "Counterfactual data augmenta- tion for mitigating gender stereotypes in languages with rich morphology," arXiv preprint arXiv:1906 1et.11996..77,,
[85] K. Webster, X. Wang, I. Tenney, A. Beutel, E. Pitler, E. Pavlick, J. Chen, E. Chi, and S. Petrov, "Measuring and reducing gendered correlations in pre-trained models," arXiv preprint arXiv:2010.06032, 2020.
[86] S. Ravfogel, Y. Elazar, H. Gonen, M. Twiton, and Y. Goldberg, "Null it out: Guarding protected attributes by iterative nullspace projection," arXiv preprint arXiv:2004.07667, 2020.
[87] T. Schick, S. Udupa, and H. Schütze, "Self-diagnosis and self-debiasing: A proposal for reduc- ing corpus-based bias in nip." Transactions of the Association for Computational Linguistics, vol. 9, pp. 1428-1424, 2021.
[88] P. P. Liang, I. M. Li, E. Zheng, Y. C. Lim, R. Salakhutdinov, and L.-P. Morency, "Towards debiasing sentence representations," arXiv preprint arXiv:2007.08100, 2020.
[89] N. Nangia, C. Vania, R. Bhalerao, and R. Bowman, "Crows-pairs: challenge dataset for measuring social biases in masked language models," arXiv preprint arXiv:2010.00133, 2020.
[90] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," arXiv preprint X5
2015.
[91] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, "Bert: Pre-training of deep bidirectional transformers for language understanding." arXiv preprint arX1v:1810.04805, 2018.
23

---

[92] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al., "Language models are unsupervised multitask learners," OpenAl blog, vol. no. 8. p. 9. 2019.
[93] S. Seo, J.-Y. attributes," in Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 16742-1660 16751, 2022.
Lee, and B. Han, "Unsupervised learning of debiased representations with pseudo-
[94] R. Zmigrod, S. J. Mielke, H. Wallach, and R. Cotterell, "Counterfactual data augmenta- tion for mitigating gender stereotypes in languages with rich morphology." arXiv preprint 2019.
arXiv:1906.
[95] K. Webster, X. Wang, I. Tenney, A. Beutel, E. Pitler, E. Pavlick, J. Chen, E. Chi, and S. Petrov, "Measuring and reducing gendered correlations in pre-trained models," arXiv preprint arXiv:2010.06032, 2020.
[96] Y. Goyal, Z. Wu, J. Ernst, D. Batra, D. Parikh, and S. Lee, "Counterfactual visual explanations." " in International Conference on Machine Learning, pp. 2376-2388 PMLR, 2019.
[97] S. Dash, V. N. Balasubramanian, and A. Sharma, "Evaluating and mitigating bias in image classifiers: A causal perspective using counterfactuals," in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 115-924, 2022.
[98] H. Kim, S. Shin, J. Jang, K. Song, W. Joo, W. Kang, and I.-C. Moon, "Counterfactual fairness with disentangled causal effect variational autoencoder," in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 8128-8136, 2021.
[99] J. Cheong. S. Kalkan, and H. Gunes, "Counterfactual faimess for facial expression recognition," " in European Conference on Computer Vision, pp. 245-261, Springer, 2022.
[100] S. Basu, P. Pope, and Feizi, "Influence functions in deep learning are fragile," arXiv preprint arXiv:2006. 14651, 2020.
[101] J. Bae, N. Ng. A. Lo, M. Ghassemi, and R. B. Grosse, "If influence functions the answer, then what is the question?," Advances in Neural Information Processing Systems, vol. 35, pp. 77953-17961 2022.
[102] G. Pruthi, F. Liu, S. Kale, and M. Sundararajan, "Estimating training data influence by tracing gradient descent," Advances in Neural Information Processing Systems, vol. 33, pp. 19920- 19930, 2020.
[103] C.-K. Yeh, J. Kim, I. E.-H. Yen, and P. K. Ravikumar, "Representer point selection for explaining deep neural networks," Advances in neural information processing systems, vol. 31, 2018.
[104] R. D. Cook and S. Weisberg. "Characterizations of an empirical influence function for detecting influential cases in regression." Technometries, vol. 22, no. 4. pp. 495-508, 1980.
[105] R. D. Cook and S. Weisberg, Residuals and influence in regression. New York: Chapman and Hall, 1982.
[106] B. A. Pearlmutter, "Fast exact multiplication by the hessian," " Neural computation, vol. 6, no. 1, pp. 147-160, 1994.
[107] N. Agarwal, B. Bullins, and Hazan, "Second-order stochastic optimization for machine learning in linear time," The Journal of Machine Learning Research, vol. 18, no. 1, pp. 4148- 4187, 2017.
[108] J. Pearl et al., "Models, reasoning and inference," " Cambridge, UK: CambridgeUmivveriiyyrresss vol. 19, no. 2, 2000.
[109] S. R. Pfohl, T. Duan, D. Y. Ding, and N. H. Shah, "Counterfactual reasoning for fair clinical risk prediction," in Machine Learning for Healthcare Conference, pp. 325-358, PMLR, 2019.
24


Reviews:
[{"review_id": "lCHn2fwb0f", "reviewer": "Reviewer_AWfs", "summary": "This paper presents a novel approach to addressing bias in models by utilizing a small external dataset for de-biasing. The proposed method effectively identifies and mitigates inherent biases by analyzing the disparities between original images and counterfactual images. Initially, the method determines the specific type of bias present in the model by computing these differences. Subsequently, it identifies samples with the highest influence scores in terms of counterfactual bias when a particular bias type is detected. To mitigate this bias, the method updates the model by unlearning these influential samples, using the difference between the gradients of the original and counterfactual samples. The method is evaluated using one synthetic dataset and two real-world datasets. The results demonstrate that the proposed method achieves state-of-the-art performance in mitigating counterfactual bias, while also maintaining low computational costs.", "questions": "* Why is the worst-case accuracy better than the baselines in Table 2? Why does unlearning by Co. outperform unlearning by De. in terms of De? \n\n* Why does unlearning by De. fail to better demographic parity bias in Table 4?\n\n* In Table 7, unlearning more layers of neural networks does not lead to better performance. Why?", "limitations": "The author has addressed their limitations.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "* The intuition of the proposed method seems clear and intuitive. Also, the proposed method is novel and well-designed two recent techniques in other domains, influence function and unlearning.\n\n* The proposed method can mitigate the bias of a model efficiently. It only needs a small amount of group-labeled data points. Furthermore, the method requires much less training time than the existing de-biasing methods based on re-training.\n\n* The method achieves good performance on CelebA and Colored MNIST.", "weaknesses": "* Lack of some baselines. Given that the objective of this method is to achieve counterfactual fairness or individual fairness, it is important for the authors to compare their approach with existing methods [1,2] specifically designed for individual fairness. This would ensure a fair and comprehensive evaluation of their method's performance in terms of counterfactual bias. It should be noted that Group DRO and LfF, the authors adopted as baselines, were originally designed for group robustness, rather than individual-level fairness.\n\n* Challenges in obtaining counterfactual samples. One drawback of the proposed method is that it requires counterfactual samples for each instance in the external dataset. However, acquiring such counterfactual samples for image datasets is not a trivial task. For instance, assuming that images are annotated with 40 attributes, as done for the CelebA dataset, may not always be feasible.\n* There should be a more detailed explanation of why the experiment results turned out the way they were presented. Why does your method that archives the counterfactual bias also mitigate the worst-case group accuracy or any other group fairness metrics?\n* Improving the writing clarity. \n  - It is suggested to provide clearer distinctions between the problems of model bias and fairness, emphasizing that while there is overlap between the two, they are not synonymous. \n  - Additionally, the definition of counterfactual fairness in Equation (1) may be incorrect, compared to the definition introduced in [31]. \n  - Some typos in equations. In lines 158 and 183, is it z_i instead of z_k?\n\n[1] Yurochkin and Sun. SENSEI: Sensitive set invariance for enforcing individual fairness, ICLR, 2021.\n[2] Yurochkin et al. Training individually fair ML models with sensitive subspace robustness, ICLR, 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thanks the reviewer for the constructive comments. Below are point-to-point responses (W denotes Weakness). \n\n>W1 *baselines*\n\nWe did experiment with the two individual fairness baselines.  The baselines can achieve good individual fairness while maintaining stable accuracy. However, they demand the entire annotated dataset for retraining, resulting in a much higher debiasing cost.\n\n|Attribute| Method|Acc(%) ↑|Bias ↓|Time(s)|\n| --- | --- | ---|---|---|\n|gender| Vanilla | 85.40 | 0.0195 |-|\n| | SenSR | 84.09 | 0.0049 | 571 |\n| | SenSeI | 83.91 | 0.0016 | 692 |\n| | Ours | 81.89 | 0.0005 | 2.49 |\n|race|Vanilla| 84.57 | 0.0089 |-|\n|  |SenSR| 84.09 | 0.0036 | 571 |\n|  |SenSeI| 83.91 | 0.0015 | 692 |\n|  |Ours|83.80| 0.0013 | 2.54 |\n\n>W2 *dataset generation*\n\nWe agree that most counterfactual fairness auditing work require generating pairs with labeled sensitive attributes [1, 2, 3], while many datasets do not have enough labels.\n\nOne possible solution is to directly use other fairness-auditing datasets, instead of building our own one if labels are unavailable. Following your advice, we did experiments with **large language models (LLMs)**, where we use the Crows-Pairs [4] , which measures the social bias in LMs against protected attributes, as “external” datasets. **Results show that our FMD can outperform or achieve comparable performance with baselines on large models (see Response to Reviewer 2Mrc W2).**\n\nIn addition, we compute the time cost to construct external datasets. For **Colored MNIST**, it takes **4.3s** to generate the external dataset (5000 samples) on an AMD CPU. Similarly, it takes **0.51s (200 samples)** for \n **Adult** and **11.2s (200 samples)** for **CelebA**. Counting such cost in, the time cost of baselines is still more than 15 times of ours.\n\nResearch on fairness with few/no attribute labels is still in the infant stage [5], and we will further explore it.\n\n>W3 *Worst-group accuracy*\n\nWorst-group accuracy reports the lowest accuracy among all groups. We computed the group statistics and accuracy on CelebA in the table below as an example. Supervised training achieves **94.90%** accuracy on the test set. However, for the **{blonde, male}** group, the accuracy is only **47.36%**. This indicates that the model captures the unintended decision rule between hair color and gender. \n\nOur method utilizes counterfactual samples to unlearn the unintended decision rule that the model has learned. The **38% improvement** in worst-group accuracy demonstrates its effectiveness. Moreover, the **smaller accuracy disparity** among different groups indicates that the predictions are more fair.\n\nThe reviewer's question may also revolve around why individual fairness metrics (Counterfactual) could lead to better group fairness (worst-group accuracy). Basically, our method reduces the mispredictions for each **{blonde, male}** individual, thereby leading to better group accuracy. **Additionally, as pointed out in [1], the individual and group fairness measures donot necessarily reflect different normative principles.**\n\n| |# of samples| |Acc (%)| |\n|---|---|---|---|---|\n|Group|Train|Test|Vanilla|Ours|\n|Blond=0, Male=0|71629|9767|99.32|97.80|\n|Blond=0, Male=1|66874|7535|89.77|89.98|\n|Blond=1, Male=0|22880|2480|94.54|94.01|\n|Blond=1, Male=1|1387|180|47.36|**87.15**|\n|Total|162770|19962|94.90|93.41|\n\n>W4 *clarity*\n\na) Fairness is a high-level category, indicating the absence of any prejudice or favoritism toward an individual or group [7]. Here we focus on the bias (unfairness) learned by ML models. We will clarify the definitions.\n\nb) Sorry for the confusion. It should be $P(\\hat{Y}_{A←a} = y | X = x, A = a) = P(\\hat{Y}\\_{A←\\bar{a}} =y | X = x, A = a)$, where $y = f\\_{\\hat{\\theta}}(X,A)$, to imply the process of attribute changing. Note that we assume that $X$ are non-descendants of $A$ so that $X\\_{A←a} = X\\_{A←\\bar{a}}$. \n\nc) We fixed the typos.\n\n>Q1\n\n See W3. \n\n>Q2 *De. & Co.*\n\nDifferent from counterfactual bias, demographic parity bias measures the difference in predictions (of a model) on different protected demographic **groups**, e.g., gender or race. One reason is that when scaling to a group, influence estimation may not be accurate, so that the selected harmful samples may not be effective. To validate it, we select top 1500 samples and count the number of bias-aligned samples (harmful). The number of De. is **1153** and of Co. is **1465**, which indicating Co. is **more effective on selecting harmful samples**.\n\n>Q3 *multi-layer estimation*\n\nPrevious work investigates the estimation accuracy of the influence function on both multi-layer and single-layer setups [8]. It performs a case study on the MNIST.  For each test point, they select 100 training samples and compute the ground-truth influence by model re-training. **Results show that estimations are more accurate for shallow networks.**\n\nOur results in Tab 7 also validate this point. When applying FMD to a three-layer neural net, the performance on either accuracy or bias becomes worse. This could potentially be attributed to the inaccurate estimation of influence function on multi-layer neural nets. In our experiments, we adhere to the set-up in [9], where the influence function is only applied to the last layer of deep models, which proves to be effective.\n\n[1] Explaining machine learning classifiers through diverse counterfactual explanations\n\n[2] Auditing fairness under unawareness through counterfactual reasoning\n\n[3] Counterfactual explanations for robustness, transparency, interpretability, and fairness of artificial intelligence models\n\n[4] CrowS-pairs: A challenge dataset for measuring social biases\n\n[5] Unsupervised learning of debiased representations with pseudo-attributes\n\n[6] On the apparent conflict between individual and group fairness\n\n[7] A survey on bias and fairness in machine learning\n\n[8] Influence functions in deep learning are fragile\n\n[9] Understanding black-box predictions via influence functions"}, {"review_id": "cM18VVvSjA", "reviewer": "Reviewer_2Mrc", "summary": "This paper presents a novel method for model debiasing, based on the techniques of counterfactual biases and machine unlearning. The method designs an all-inclusive debiasing pipeline: bias identification, biased-effect evaluation, and bias removal. The proposed method can get rid of time-consuming human labeling or re-training, which is appealing to practical applications. Experimental results and ablation analysis suggest that the proposed method is effective and more efficient on multiple datasets for fairness evaluation. Overall, I appreciate the idea of using machine unlearning and counterfactual samples for efficient model debiasing, though some concerns still need to be addressed for improved quality.", "questions": "See weakness. \nMinor issues: \n1. Writing need to be improved, e.g., typos and grammar errors: line 40 involves (involve), line 35 “or”, etc. \n2. Paper title and manuscript title are not the same. Title should be \"Fast model debiasing with machine unlearning\"?\n3. What are the definitions of  Demographic parity bias (De.) [34], and Equal opportunity bias, and how they relate to the counterfactual bias? The Eq 10 and 11 need to be further clarified.", "limitations": "Authors addressed the limitations.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Strengths:\n1. Model debiasing and fairness are critical for the deployment of large AI models in practice. The proposed method can identify and remove the bias in a cost-effective manner, which is more suitable for practical applications than previous retraining methods. \n2. The idea is novel and interesting. Employing counterfactual samples to identify biases seems quite helpful. Leveraging the machine unlearning techniques with counterfactually constructed datasets for bias removal shows a new promising direction for efficient model debiasing. \n3. The sanity check is clear, experiments in the main paper and appendix are quite comprehensive, and the performance in terms of accuracy, fairness and efficiency are promising.", "weaknesses": "1. The post-processing baselines are included on the Colored MNIST. I suggest including them for other datasets, i.e., CelebA and Adult. \n2. More discussions on how the proposed method can be generalized for large-scale networks or tasks. Existing fairness studies are mainly conducted on small datasets like C-MNIST and networks such as MLP/ResNet. It would be great if the authors can investigate how the proposed unlearning method can be extended to large-scale datasets or models, for examples, is it possible to extend this idea to  large-scale language models. Even some preliminary results or discussions would be highly appreciated. \n3. Besides quantitative results in Table 1-3, more straightforward visualizations of experimental results on how the proposed method can give rise to better fairness (less bias) are highly encouraged  to better verify the effectiveness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">Weakness. 1. *post-processing*\n\nThanks for your advice. We added the performance of post-processing on CelebA for reference. However, these post-processing methods aim to achieve group fairness by simply altering the predictions of a few individuals and cannot achieve comparable performance.\n\n| Target Attr | Bias Attr | Method | Avg. Acc | Bias |\n| --- | --- | --- | --- | --- |\n| Blonde hair | male | Vanilla | 94.90 | 0.4211 |\n|  |  | DRO | 92.90 | 0.3206 |\n|  |  | LfF | 93.52 | 0.2557 |\n|  |  | LDR | 86.67 | 0.3126 |\n|  |  | EqOdd | 91.03 | 0.4095 |\n|  |  | CEqOdd | 91.84 | 0.4271 |\n|  |  | Reject | 90.87 | 0.3290 |\n|  |  | Ours | 93.41 | 0.0717 |\n| Attractive | male | Vanilla | 77.42 | 0.3695 |\n|  |  | DRO | 78.35 | 0.3004 |\n|  |  | LfF | 77.24 | 0.2815 |\n|  |  | LDR | 81.70 | 0.2986 |\n|  |  | EqOdd | 77.03 | 0.3689 |\n|  |  | CEqOdd | 76.79 | 0.3718 |\n|  |  | Reject | 76.09 | 0.3471 |\n|  |  | Ours | 80.99 | 0.1273 |\n\n>Weakness. 2. *scale to large model*\n\nThanks for your comments. We further extended our method to LLM (large language model) debiasing scenario. We make use of pre-trained BERT [1] and GPT-2 [2], provided by Huggingface [3]. We use StereoSet [4] as our test set. StereoSet is a large-scale natural dataset to measure stereotypical biases in gender, profession, race, and religion. It reports two metrics: **Language Modeling Score (LMS)** measures the percentage of instances in which a language model prefers the meaningful over meaningless association. **The LMS of an ideal language model will be 100 (the higher the better)**. **Stereotype Score (SS)** measures the percentage of examples in which a model prefers a stereotypical association over an anti-stereotypical association. **The SS of an ideal language model will be 50 (the closer to 50 the better)**.\n\nWe use Crows-Pairs [5] as our external dataset. Each sample in Crows-Pairs consists of two sentences: one that is more stereotyping and another that is less stereotyping, which can be utilized as counterfactual pairs. Following the setting in the main paper, we freeze all but the top layer in the trained large model. Results are presented in the following table. We compare our method with five commonly used LLM debiasing baselines: Counterfactual Data Augmentation (CDA) [6], Dropout [7], Iterative Null-space Projection (INLP) [8], Self-debias [9], and SentenceDebias [10]. \n\nResults on BERT are presented in the following table (**Results on GPT-2 are presented in Tab.3 in the PDF due to the space constraints**), and show that our method can outperform or achieve comparable performance with baseline methods. **As for BERT, our method reaches best (denoted by “bold”) or second best (denoted by “underline”) performance in 5 of 6 metrics.** \n\n| backbone | attribute | method |SS|LMS|\n| --- | --- | --- | --- | --- |\n| BERT | gender | Vanilla|60.28|84.17|\n|  |  | CDA | 59.61 | 83.08 |\n|  |  | Dropout  | 60.66 | 83.04 |\n|  |  | INLP | **57.25** | 80.63 |\n|  |  | Self-debias | 59.34 | 84.09 |\n|  |  | SentenceDebias | 59.37 | 84.20 |\n|  |  | Ours | $\\underline{57.77}$ | **85.45** |\n| BERT | race | Vanilla| 57.03 | 84.17 |\n|  |  | CDA | 56.73 | 83.41 |\n|  |  | Dropout| 57.07 | 83.04 |\n|  |  | INLP | 57.29 | 83.12 |\n|  |  | Self-debias | **54.30** | **84.24**|\n|  |  | SentenceDebias | 57.78 | 83.95 |\n|  |  | Ours | 57.24 | $\\underline{84.19}$ |\n| BERT | religion | Vanilla| 59.70 | 84.17 |\n|  |  | CDA | 58.37 | 83.24 |\n|  |  | Dropout  | 59.13 | 83.04 |\n|  |  | INLP | 60.31 | 83.36 |\n|  |  | Self-debias | **57.26** | 84.23 |\n|  |  | SentenceDebias | 58.73 | 84.26 |\n|  |  | Ours | $\\underline{57.85}$ | **84.90** |\n\n\n\n>Weakness. 3. *visualization*\n\nThanks for your advice. We have included presentative images in Fig. 1 in the PDF.\n\n\n>Q1. *typo*\n\nThanks for your advice. We have fixed that in our manuscript.\n\n>Q2. *title*\n\nWe are really sorry for the mistake. The title should be \"Fast model debiasing with machine unlearning\". We will modify the title.\n\n>Q3. *related works*\n\nThank for your comments. We will incorporate a more detailed explanation into the related works section. The definitions are as follows: \n\nDemographic Parity [11]:  A predictor $Y$ satisfies demographic parity  if\n$P(Y |A = 0) = P(Y|A = 1)$, where $A$ is the sensitive attribute. The likelihood of a positive outcome\nshould be the same regardless of whether the person is in the protected (e.g., female) group.\n\nEqual Opportunity [12]:  “A binary predictor $Y$ satisfies equal opportunity with respect \nto $A$ and $Y$ if $P(Y=1|A=0,Y=1) = P( Y=1|A=1,Y=1)$” . This means that the probability of a person in a positive class being assigned to a positive outcome should be equal for both protected and unprotected (female and male) group members.\n\nThe counterfactual fairness definition is based on the intuition that “a\ndecision is fair towards an individual if it is the same in both the actual world and a counterfactual world where the individual belonged to a different **demographic group**.” It can be viewed as individual-level Demographic Parity [13].\n\n[1] Bert: Pre-training of deep bidirectional transformers for language understanding.\n\n[2] Language models are unsupervised multitask learners.\n\n[3] Huggingface's transformers: State-of-the-art natural language processing.\n\n[4] StereoSet: Measuring stereotypical bias in pretrained language models.\n\n[5] CrowS-pairs: A challenge dataset for measuring social biases in masked language models.\n\n[6] Counterfactual data augmentation for mitigating gender stereotypes in languages with rich morphology.\n\n[7] Measuring and reducing gendered correlations in pre-trained models.\n\n[8] Null it out: Guarding protected attributes by iterative nullspace projection.\n\n[9] Self-diagnosis and self-debiasing: A proposal for reducing corpus-based bias in nlp.\n\n[10] Towards debiasing sentence representations.\n\n[11] Fairness through awareness.\n\n[12] Equality of opportunity in supervised learning.\n\n[13] Counterfactual Fairness Is Basically Demographic Parity."}, {"review_id": "TKdvYjqCOw", "reviewer": "Reviewer_NRrd", "summary": "In this paper, the authors propose Fast Model Debiasing (FMD), which targets solving biased prediction problems. The proposed FMD can effectively and efficiently remove model bias by leveraging only a small external dataset by altering the attribute, then evaluating the biased-effect of each sample, and finally using the most harmful samples to unlearn the model bias with the Newton update step. Experiments on Colored MNIST, CelebA, and Adult datasets demonstrate the effectiveness of the proposed method.", "questions": "N/A", "limitations": "N/A", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The motivation and the idea proposed in this paper are reasonable. The unlearning progress is efficient since the model parameters only need to be updated in several steps. The execution time is greatly shorter than the compared methods. The experimental results are sufficient to support the effectiveness of the proposed method.", "weaknesses": "1. Some writing issues. \"Fast Model Debias\" in the title, but \"fast model debiasing\" in the abstract. The full name of \"FMD\" is not explained when it first appeared in Sec 1. \n2. In Eq 3, reduant \")\". \n3. Some usage of notation is not standard. For example, the function \"B\" accepts three kinds of input. \n4. In line 231, the notation $C_i$ should be $c_i$.\n5. The number of K when selecting harmful samples, and the bias threshold seems not explained in the experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Weakness. 1. *Some writing issues. \"Fast Model Debias\" in the title, but \"fast model debiasing\" in the abstract. The full name of \"FMD\" is not explained when it first appeared in Sec 1.*\n> \n\nWe are really sorry for the mistake. The title should be \"Fast model debiasing with machine unlearning\". We will modify the title.\n\n> Weakness. 2. *In Eq 3, reduant \")\".*\n> \n\nThanks for your advice. We have fixed that in our manuscript.\n\n> Weakness. 3. *Some usage of notation is not standard. For example, the function \"B\" accepts three kinds of input.*\n> \n\nThanks for your advice. We have fixed that in our manuscript.\n\n> Weakness. 4. In line 231, the notation $C_i$ should be $c_i$.\n> \n\nThanks for your advice. We have fixed that in our manuscript.\n\n> Weakness. 5. *The number of K when selecting harmful samples, and the bias threshold seems not explained in the experiments.*\n> \n\nThanks for your comments.  In our experiment, we select the number of samples **k=5000** for Colored MNIST, and **k=200** for both Adult and CelebA. \n\nWe present experiments of debiasing using Eq. 9 with different values of K (on Colored MNIST with a bias ratio of 0.99) as an example. It can be observed that accuracy and bias improve gradually and tend to be saturated as the number of samples increases, while the time for unlearning (debiasing) continually increases. Hence, in experiments, **we select the number of samples based on both the performance and time cost.**\n\n| Method  | # of samples (K) | Acc.(%) ↑ | Time(s) | Bias ↓ |\n| --- | --- | --- | --- | --- |\n| Vanilla | N/A | 51.34 | - | 0.4931 |\n| Ours | 500 | 73.67 | 3.750s | 0.2757 |\n|  | 1000 | 75.70 | 7.396s | 0.2648 |\n|  | 2000 | 77.36 | 14.60s | 0.2451 |\n|  | 3000 | 78.98 | 25.37s | 0.2232 |\n|  | 5000 | 80.04 | 48.37s | 0.2042 |\n|  | 6000 | 80.06 | 58.24s | 0.2044 |\n|  | 8000 | 80.31 | 84.90s | 0.2044 |\n|  | 10000 | 80.42 | 105.25s | 0.2035 |\n\n**The bias threshold** varies across tasks and could be selected according to practical requirements. In our experiment, we perform the debiasing phase on all tasks to showcase our proposed method (**assuming a threshold of 0**). However, in practical applications, the debiasing phase may not always be necessary. For instance, if we set the threshold to 0.3, in our experiment on Colored MNIST with a bias ratio of 0.95, the bias of the vanilla model (0.2589) is already compliant."}, {"review_id": "AHXVL8WIY5", "reviewer": "Reviewer_hEy4", "summary": "The paper addresses the issue of biases in trained models, which are often caused by imbalances in the dataset used for training. The conventional approach to mitigating biases involves finetuning the model, but this incurs additional training costs. The paper proposes an alternative method called Fast Model Debiasing (FMD) that does not require finetuning.\n\nThe FMD method consists of three steps: identifying the bias attributes, identifying the top samples that contribute to the bias, and removing the bias. The key contribution of the paper is the use of Influence Functions (IF) and its extension to identify and eliminate biases. This approach is interesting and appears to be reasonable.\n\n", "questions": "In the approach, several layers in the model are updated based on certain rules to remove the bias. To further extend this work, it would be interesting to explore which modules are sensitive to bias and contribute to the debiasing process. \nFor example, when considering model adaptation during the inference stage, we can optimize the Batch Normalization (BN) module by re-estimating the mean and variance based on the fresh samples from new scenarios in just a few steps. HEre, BN is the sensitive for model adaptation. \nBack to the debias, which module is the key?\n\nThere are some other questions:\n1. Select K top harmful samples. what is the suggested value for K? \n2. Such harmful samples highly depends on the testing samples. what is such testing samples from? \n-if testing samples is from one subset of the training/validation set, how does the proposed not involve new bias after removing the influence of K top harmful samples? \n3. Not find the bassline accuracy of the model without the debias, and how to guarantee that the updated parameters would not impact the model accuracy? \n4. Have you tried other complex or big model?\n\n", "limitations": "1. In the proposed FMD,  based on the Newton update, the model is updated. The model accuracy generally would be impacted. But, most of the given experiments show that both of accuracy and debias performance are best. Such results might not be convinced. All of the experiments have been conducted using small or simple models on easy tasks. It is important to verify whether such methods can be applied to other typical or complicated tasks/datasets, such as ImageNet/COCO or segmentation tasks. Please enable more experiments or cite more other reference.\n* Firstly, add one line or column to show the baseline performance, e.g., the accuracy without the debias. \n* Secondly, extend to more scenarios, e.g., imageNet dataset.\n* Thirdly, maybe one flow figure is helpful to understand the overall FMD procedure.\n* Model debias is partially related to other topics, e.g., OOD, model, domain adaptation or others, where some debias method are also introduced. if time and space are ok, survey and add more reference. \n\n2. Implementing such a scheme requires building a specific dataset (even if it is small) and updating the model layers, which can be a time-consuming process. It might not be easier than performing a quick finetuning for debiasing. Therefore, it would be beneficial to compare the complexity between the proposed approach and quick finetuning, as well as evaluate their performance.\n* In general, quick finetuning does not necessarily require a larger dataset. A small specific dataset with data augmentation might be sufficient for finetuning to address bias.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The main contribution of the proposed method is to leverage the IF (Implicit Fairness) technique and its extension to identify and remove bias in models. This approach offers a practical solution for making models debiased.\n\nCompared to typical fine-tuning solutions, this method avoids the need for additional training effort, making it a more efficient procedure. By directly removing bias through the IF technique, the model can be made fairer without requiring extensive retraining.\n\nFurthermore, this method can be implemented as a separate API, allowing for easy integration into existing systems. This API can be used to remove bias from models, ensuring fairness in various applications.", "weaknesses": "Here, the general idea is simple: to identify the samples, and further enhance the model to debias.\nMore experiments should be added to verify this.\nrefer to the following: limitation parts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the constructive comment！\n\n>Q1. *which module is the key:* \n\nIn our method, we leverage the influence functions [1] to identify training samples that lead to biases,  which are not originally designated to identify specific machine learning modules that lead to bias. The debiasing process in our FMD is through the updating of the top 1-3 MLP layers of the biased model with the counterfactually constructed external datasets. Instead of telling which module is the key, our method mainly focus on identifying which data samples are the key (i.e., leading to bias or can be used for debiasing) and then update the top layers accordingly. \n\nFollowing your suggestion and to our knowledge, there is another line of work in the field of model editing [2, 3] which could be helpful to identify the key modules for debiasing. For instance, the Locate-Then-Edit [4, 5] method initially identifies parameters corresponding to specific knowledge and then modifies them through direct updates to the target parameters.\n\nAgain many thanks for this valuable comment, which present a new promising direction for model debiasing. In our future work with large models (LMs), we will delve deeper into investigating the impact of different modules within LMs for fairness.\n\n>Q2. *value for K:* \n\nIn our experiment, we select the number of samples **k=5000** for Colored MNIST, and **k=200** for both Adult and CelebA. \n\nWe present experiments of debiasing using Eq. 9 with different values of K (on Colored MNIST with a bias ratio of 0.99) as an example. It can be observed that accuracy and bias improve gradually and tend to be saturated as the number of samples increases, while the time for unlearning (debiasing) continually increases. Hence, in experiments, we select the number of samples based on both the performance and time cost. \n\n| Method | # of samples (K) | Acc.(%) ↑ | Bias ↓ | Time(s) |\n| --- | --- | --- | --- | --- |\n| Vanilla | N/A | 51.34 | 0.4931 | - |\n| Ours | 500 | 73.67 | 0.2757 | 3.750s |\n|  | 1000 | 75.70 | 0.2648 | 7.396s |\n|  | 2000 | 77.36 | 0.2451 | 14.60s |\n|  | 3000 | 78.98 | 0.2232 | 25.37s |\n|  | 5000 | 80.04 | 0.2042 | 48.37s |\n|  | 6000 | 80.06 | 0.2044 | 58.24s |\n|  | 8000 | 80.31 | 0.2044 | 84.90s |\n|  | 10000 | 80.42 | 0.2035 | 105.25s |\n\n>Q3. *testing samples:* \n\nIn our experiments with external datasets, the testing samples (the samples for debiasing) **have no overlap** with the train/val datasets. We agree with the reviewer that the debiasing process which might inadvertently introduce new biases. While based on our problem setting, these “potential” new biases would not affect our main objective to remove some specific bias . Some recent research also study the compatibility of different fairness criteria [6, 7, 8], and we will include the discussion of this issue in the revised version. \n\n>Q4. *baseline accuracy:* \n\nSorry for the confusion, we use **\"Vanilla\"** to denote the model without any debiasing method, which is supervised trained with cross-entropy loss. For example, in our experiment on Adult (in Tab. 3), where training and test data are i.i.d., Vanilla achieves the best accuracy cause it fits the training set quite well. Compared to baseline methods, our proposed method achieved the best fairness score with second-best accuracies (just lower than Vanilla), indicating that our method ensures stable accuracy by only requiring rapid updates of a small set of parameters. In contrast, other methods sacrifice more accuracy to achieve certain levels of fairness. \n\n>Q5. *big model:* \n\nSee Response to Reviewer 2Mrc Weakness 2.\n\n>Limitation. 1. *flow figure & reference:* \n\nWe have included a flow figure in the PDF document. Model debiasing has close connections with out-of-distribution (OOD) detection and domain adaptation [9, 10], and we will discuss these relationships in our paper.\n\n>Limitation. 2. *time-consuming & quick finetuning:*\n\n**Complexity:** In our experiments, as for **Colored MNIST**, it takes about **4.3s** to generate the whole external dataset (5000 samples) on a single AMD CPU. For **Adult**, flipping sensitive attributes for the external dataset takes about **0.51s (200 samples)**. For **CelebA**, constructing counterfactual pairs takes about **11.2s (200 samples)**. \n\n**Experiment on finetuning:** We conducted experiments on Colored MNIST by fine-tuning the model on an **unbiased** dataset (i.e. bias ratio = 0) with 5000 (and 500) samples. Results are reported in the following table. It can be observed that our method outperforms Fine-tuning on bias ratios of 0.95 and 0.99. \n\n| Bias Ratio  | Method | Acc.(%) ↑ | Bias ↓ | Time(s) | # Samp. |\n| --- | --- | --- | --- | --- | --- |\n| 0.95 | Vanilla | 77.63 | 0.2589 | - | -|\n|  | LDR | 90.42 | 0.2334 | 19min40s | 60000 |\n|  | LfF | 85.55 | 0.1264 | 12min04s | 60000 |\n|  | Rebias | 89.63 | 0.1205 | 28min34s | 60000 |\n|  | Ours | 89.26 | 0.1189 | 56.44s | 5000 |\n|  | Fine-Tuning | 83.53 | 0.1656 | 199.57s | 5000 |\n|  | Fine-Tuning | 79.02 | 0.2176 | 116.17s | 500 |\n| 0.99 | Vanilla | 51.34 | 0.4931 |-|-|\n|  | LDR | 76.48 | 0.2511 | 22min10s | 60000 |\n|  | LfF | 64.71 | 0.2366 | 12min06s | 60000 |\n|  | Rebias | 80.41 | 0.2302 | 27min38s | 60000 |\n|  | Ours | 80.04 | 0.2042 | 48.37s | 5000 |\n|  | Fine-Tuning | 79.12 | 0.1971 | 198.37s | 5000 |\n|  | Fine-Tuning | 75.83 | 0.2105 | 118.03s | 500 |\n\n\n\n[1] Understanding black-box predictions via influence functions.\n\n[2] Fast model editing at scale. \n\n[3] Memory-based model editing at scale. \n\n[4] Knowledge neurons in pretrained transformers. \n\n[5] Locating and editing factual associations in GPT. \n\n[6] On the apparent conflict between individual and group fairness. \n\n[7] Fairness in criminal justice risk assessments: The state of the art. \n\n[8] Counterfactual fairness.\n\n[9] Reducing gender bias in neural machine translation as a domain adaptation problem.\n\n[10] Graphde: A generative framework for debiased learning and out-of-distribution detection on graphs."}, {"review_id": "QRVFtJ9zPn", "reviewer": "Reviewer_hkdb", "summary": "This paper proposes an approach to removing biases from trained models without necessitating full retraining and without necessitating access to a large dataset that has annotations w.r.t biases. The proposed method consists of 3 steps: firstly, identifying which attributes are ‘biased’ (according to a counterfactual definition of bias: the more the predictions differ between the factual and counterfactual examples, the more biased the model is). Second, identify which examples are those contributing to biased behavior, using influence functions. Finally, once the (top K) harmful samples are identified, they run an unlearning procedure (via a Newton step) to remove the influence of those examples. \n", "questions": "- How did the authors pick the value of K (number of harmful samples to unlearn)? Is this different for each dataset / setting?\n\n- When computing the time (for quantifying the efficiency of the method), which steps are taken into account for the proposed method? Are all steps 1-3 counted towards this? If not, why not? Is this done fairly w.r.t how it’s counted for other methods?\n\n- Follow-up: what is the computational complexity of running the proposed method, in all 3 steps? Computing Hessians is expensive, and it sounds like the proposed method is to compute the influence of each example, in order to then rank them and find the top K.\n\n- How well does the method scale to deeper neural networks? All architectures used in the experiments are very small.\n\n- When using an external dataset for “unlearning”, how similar does that dataset need to be to the training set, in order for that to be beneficial? What if the external dataset is distributionally-shifted in different ways?\n\n- Could some of the baselines considered also be (easily) modified to use external datasets and / or fewer samples?\n", "limitations": "- The proposed method requires datasets with access to attributes, so that counterfactuals can be created by e.g. changing the value of one attribute while keeping the values of other attributes fixed. But in practice, we typically don’t have attribute information for datasets of interest.\n", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- This paper tackles an important problem and provides useful motivation and intuition for understanding the issues with biases in neural networks\n\n- The proposed approach makes fewer limiting assumptions than some of the previous methods, and the authors show that empirically, it outperforms certain baselines on certain metrics on some (simple) datasets / models\n\n- The paper is well-structured and mostly clear, with some exceptions (see weaknesses section below) \n", "weaknesses": "A) Clarity can be significantly improved in some sections. Some particular examples:\n\na) Some imprecise statements, e.g. what does it mean to be “closely tied” (line 136)\n\nb) In line 158, what is z_k? Should that have been z_i? (i is the variable that the sum is over, but it’s not used, unless I’m missing it?). Also, need to define n in that equation.\n\nc) In Equation 3 (which defines the counterfactual bias), how do we choose the value to modify a_i with? Does the equation need some expectation over which alteration to make to the affected attribute?\n\nd) in the “Biased-effect evaluation section”, the authors say that they investigate how the bias is learned. More precisely, though, what they do is estimate the influence of specific training examples, so a more appropriate description is to identify which examples lead to learning the bias, not *how* the bias is learned.\n\ne) What does “up” mean / stand for, in e.g. Equation 6.\n\nf) Generally, the Bias-effect evaluation paragraph is hard to understand without a background in influence functions. It would be great to give more intuition around equations.\n\ng) In the experiments section, the authors don’t describe the baselines they use, making it harder for someone unfamiliar with this literature to assess their appropriateness and the value of the associated comparisons\n\n\nB) [Terminology] Unlearning refers to removing the influence of a subset of training examples from a trained model. Therefore, doing “unlearning” on external samples (e.g. from a test set) doesn’t make sense according to that definition. Would be useful to clarify this point and perhaps use a different term instead. Debiasing sounds like a more appropriate word to me rather than unlearning (though inspiration may be drawn from unlearning methods). \n\nC) Issues with the experimental setup: \n\na) there are no confidence intervals in the tables, making it hard to assess whether the proposed method’s improvement over the considered baselines in each case is statistically significant. Sometimes the results are quite similar.\n\nb) In Table 1, why is DRO excluded from comparisons? In Table 3, why don’t the authors also show the Unb. and Wor. columns, as in Table 2? Those seem like a more direct and appropriate way to assess the degree of biasing in a way that actually matters / affects the model’s behavior, rather than only looking at the Bias column.\n\nc) Overall, the experiments are conducted on simple datasets with relatively small neural networks, and it’s unclear how this affects the conclusions. For instance, it’s surprising that unlearning only the top layer is successful, as in general this is not a good unlearning method (examples typically influence the trained model throughout layers, in a highly non-convex manner)  \n\nd) It would also be nice to compare against the oracle for unlearning, to get an upper bound. Specifically, once the harmful examples are identified, the oracle for unlearning would simply remove them from the training set and retrain the model without them (this of course only applies to the case where unlearning is done using training examples, as is the setup in unlearning). How well does the proposed approach debias, compared to this oracle for the unlearning step? \n\ne) Looking at Table 5, counterintuitively, Eq. 9 (“unlearning” using external samples) leads to smaller bias than Eq. 7 (unlearning using training samples). This seems to contradict the narrative that the former should do worse than the latter but is adopted instead due to not wanting to assume access to the training set? Do the authors have an explanation for this? Or is it the case that all these results are statistically indistinguishable from each other? (again, there are no confidence intervals in the tables, making it hard to tell.)\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for the detailed comments! Please find our replies to your questions below (**Q denotes Question, W denotes Weakness and R denotes Reviewer**). **Due to the space limitation, answers to Weakness continues in the global response.** Sorry for the inconvenience. **Please note that results for WC-abd and Q2 are included in Tab.1 in the PDF document.** \n\n>Q1. *value of K* : \n\nSee response to Reviewer NRrd Weakness 3.\n\n>Q2. *time counting*: \n\nThe reported time of our method is counted on step 3 (debiasing phase). For fair comparison, we also reported the debiasing time of the baseline methods. As an additional reference, we included the total process time (in Tab.1 in the PDF document). The results demonstrate that our method remains efficient in terms of time.\n\n>Q3. *computational complexity*: \n\nThanks for the comments. As for bias-effect evaluation, with $n$ training points and $\\theta \\in R^d$, directly computing Eq. 5 requires $O(nd^2 + nd^3)$ operations. In our experiment, we only activate the last layer so that d is small. However, when the number of training samples is very large, performing Eq. 5 is expensive. As for the debiasing phase, it requires $O(nd^2 + kd^2)$ operations, where k is the number of samples to unlearn. Note that if hessian is calculated in the bias-effect evaluation phase, it can be directly used in the debiasing phase. Hence, the overall computational complexity using Eq. 7 and Eq. 8 is $O(nd^2 + kd^2 + nd^3)$.\n\nHowever, in our proposed alternative debiasing method, we only utilize an external counterfactual dataset with a small number of k. Hence, we can omit the $O(nd^3)$ operations to compute influences and rank the training samples. **Hence, the overall computational complexity using Eq. 9 (Ours) is** $O(nd^2 + kd^2)$. \n\nExperimental comparison results can be referred to Table 5. Debiasing with Eq. 8 takes about 500x more time than Eq. 9.\n\n>Q4. *large model*: \n\nSee response to Reviewer 2Mrc Weakness 2.\n\n>Q5. *distribution shift*: \n\nIn our experiment, we randomly split the external dataset from the whole dataset so that it is i.i.d with the training set.\n\nWe further experiment on unbalanced external datasets (**long-tailed MNIST**) with imbalanced factors of 5, 10, and 20, **i.e., distribution-shift**. The results demonstrated that an **imbalanced external dataset can lead to a decline in performance**. Thank you for providing the inspiration, and we agree that it is an interesting topic. How to prevent the influence of distribution shifts is not included in our problem setting, however, will be discussed in our future works.\n\nWorth mentioning is that in our experiments on large models (tested on StereoSet), we also use a **distributionally-shifted dataset (Crows-Pairs)** for debiasing. The debiasing results are promising, but it is possible that some performance limitations exist due to the differences in distributions between datasets.\n\n| Bias Ratio | Method | Imbalanced Factor | Acc.(%) ↑ | Unb. | Wor. | Bias ↓ |\n| --- | ---|---|--- | --- | --- | --- |\n| 0.99 | Vanilla | - | 51.34±3.37 | 73.06±3.17 | 46.41±7.28 | 0.4931 |\n|  | LDR | - | 76.48±1.45 | 87.18±1.43 | 74.93±3.49 | 0.2511 |\n|  | LfF | - | 64.71±2.17 | 80.42±1.94 | 61.83±4.29 | 0.2366 |\n|  | Rebias | - | 80.41±1.03 | 87.85±0.95 | 76.19±1.98 | 0.2302 |\n|  | DRO | - | 71.33±1.57 | 83.82±0.87 | 68.24±2.27 | 0.2563 |\n|  | Ours | **1** | 80.04±1.42 | 87.69±1.03 | 76.31±2.03 | 0.2042 |\n|  | Ours | **5** | 78.28±1.37 | 83.08±1.31 | 67.07±2.44 | 0.235 |\n|  | Ours | **10** |76.90±1.64 | 82.33±1.49 | 65.54±2.41 | 0.2514 |\n|  | Ours | **20** | 76.16±1.61 | 80.84±1.27 | 62.48±2.17 | 0.2581 |\n|  | Oracle | - | 82.74±1.09 | 89.08±0.98 | 79.03±1.84 | 0.1683 |\n\n>Q6. *baseline*: \n\nFor additional reference, we modify our baselines to be trained on the same number of samples as our method (Experiment conducted on Colored MNIST). The results indicate that reducing the data volume consistently leads to a decline in the performance of the baseline method.\n\n| Bias Ratio | Method | Acc.(%) ↑ | Bias ↓ | Time(s) | # Samp. |\n|---|---|---|---| --- | --- |\n| 0.99 | Vanilla| 34.62±3.17 | 0.5731 | - | - |\n|  |LDR| 76.48±1.45 | 0.2511 | 1,330 | 50000 |\n|  |LfF|64.71±2.17 | 0.2366 | 726 | 50000 |\n|  |Rebias| 80.41±1.03 | 0.2302 | 1,658 | 50000 |\n|  |DRO|71.33±1.57 | 0.2563 | 4,362 | 50000 |\n|  |LDR| 65.62±1.61 | 0.3123 | 317 | **5000**|\n|  |LfF |59.38±2.11 | 0.2983 | 196 | **5000** |\n|  |Rebias | 73.24±1.47 | 0.2581 | 408 | **5000** |\n|  |DRO | 65.24±3.28 | 0.2674 | 537 | **5000** |\n|  |Ours | 80.04±1.42 | 0.2042 | 59 (+33) | 5000 |\n\n>Q7. *dataset generating*: \n\nSee response to **Reviewer AWfs Weakness2**.\n\n> WA-a: *clarity*\n\nIn line 136, we intended to convey \"highly correlated\" as \"closely tied\". We will clarify it in the revision.\n\n> WA-b: *clarity*\n\nIt should indeed be z_i in line 158. n is the number of training samples. We will further proofread our manuscript.\n\n> WA-c: *value of a_i*\n\nAccording to the definition of counterfactual fairness, the prediction should not change for all attainable a_i. **We randomly choose a_i from all attainable values (the set of values for a_i in the training set).** For example, in our experiment on Colored MNIST, with ten distinct colors included in the dataset, the alteration set is denoted as {color1, color2, ..., color10}. When modifying $a_i$, we assign a random color from the alteration set to $a_i$. The equation needs expectation over alterations.\n\n> WA-d: *clarity*\n\nIn this section, our goal is to quantify how each training point $z$ in the training set $D_{tr}$ contributes to $B$. We agree with the reviewer that it should be “*which examples lead to learning the bias*”. We will clarify it in the revision.\n\n> WA-e: *\"up\"*\n\nThe word \"up\" is the abbreviation of \"upweight\". Here we follow the subscript \"up\" used in [1]. We will further clarify this equation in our manuscript.\n\n **Due to the space limitation, answers to Weakness continues in the global response.**\n"}]

Author rebuttal:
Dear reviewers,

We greatly appreciate your valuable comments and the time and dedication you invested in reading, comprehending, and assessing our paper. We thoroughly reviewed all the questions, weaknesses raised by the reviewers and addressed them accordingly. Several questions have attracted the attention of multiple reviewers, such as **scale to large model** (R. 2Mrc W2, hEy4 and hkdb) , **number of K** (R. NRrd W5, hEy4 and hkdb) and **dataset generation** (R. AWfs W2&hkdb). **Due to space limits, we only placed the response to each first corresponding reviewer. We apologize for the inconvenience caused to other reviewers.** We have provided comprehensive responses to each review individually, and we are confident that our responses adequately address all the concerns raised by the reviewers. We are in the process of updating and uploading both the paper and supplementary materials to incorporate our responses. Should further details, explanations, or clarifications be required, we are more than willing to provide them.

Thank you once again for your thoughtful feedback.

**Due to the space limitation, we continue the response to R. hkdb below. Sorry for the bothering to other reviewers.**

**Response to R. hkdb:**

> WA-f. *influence function*: 

We will incorporate this explanation into the article. Our Bias-effect evaluation is based on the influence functions [1]:
$I\_{up, params}(z)  = \frac{d\hat\theta\_{\epsilon, z}}{d\epsilon}\Bigr|\_{\substack{\epsilon = 0}}.$
It gives an efficient approximation to compute the parameter change if $z$ were upweighted (by some small $\epsilon$).

In our paper, our goal is to quantify how removing each training point $z$ contributes to bias. We apply the chain rule:
$I_{up, bias}(z, B(\hat{\theta})) = \frac{dB(\hat\theta_{\epsilon, z})}{d\hat\theta_{\epsilon, z}} \frac{d\hat\theta_{\epsilon, z}}{d\epsilon}\Bigr|_{\substack{\epsilon = 0}}.$
Intuitively, this equation can be understood in two parts: the latter part calculates the impact of removing $z$ on the parameters. The former part corresponds to the derivative of bias with respect to parameters, assessing how changes in parameters affect the bias. Hence, this equation quantifies the influence of removing $z$ on the bias.

> WA-g. *baselines*: 

We compare our method with four in-processing debiasing baselines. We further include two individual-fairness baselines and three post-processing debiasing baselines following the suggestion of R. AWfs and 2Mrc. LDR [2] proposes a feature augmentation approach via disentangled representation for debiasing. Rebias [3] proposes to train a de-biased representation by encouraging it to be different from a set of biased representations. LfF [4] proposes to firstly train a biased network, then debias the second network by focusing on samples that go against the biased network. Distributionally robust optimization (DRO) [5] proposes to improve the worst-case generalization performance to perform unbiased optimization. We will include these details in the supplementary section.

> WB. *terminology*:

This is an insightful perspective. In our proposed method (Eq. (9)), what we actually eliminate is the influence brought by the bias attribute, rather than the impact of the sample itself. Therefore, "debiasing" would be more appropriate. We will replace "unlearn" with "debias" in our method description.

> WC-a. *confidence intervals*:

We have added confidence intervals in Tab.1 in the PDF.

> WC-b. *DRO & Unb*:

Group Distributionally robust optimization (DRO) [5] has been proposed to improve the worst-case generalization performance. In [5], experiments are all performed on binary classification tasks with binary attributes (i.e. 4 sub-groups).
When extending DRO to Colored MNIST, the number of groups to form is $10*10$, which will not be efficient to optimize. We added the performance of DRO on Colored MNIST (in Tab.1 in the PDF). The results indicate that DRO is not a robust baseline. 

We also add the results on Colored MNIST for **Wor.** (testing on bias-conflicting samples) and **Unb.** (testing on 50% bias-aligned and 50% bias-conflicting samples).

> WC-c. *scale to large model*:

See response to R. 2Mrc W2 due to the space limit.

> WC-d. *oracle*:

Thank you for your suggestion. We have included the "Oracle" results in Tab.1 in the PDF. Following your advice, we identified harmful samples using influence functions, excluded an equal number of samples from the training set, and then retrained the model. It can be observed that "Oracle" is better than our method. This outcome might be attributed to errors in influence estimation. However, it's important to note that computing influence on the training set and subsequent retraining is time-consuming (27 times of ours).

> WC-e. *debiasing strategies*:

In Tab.5, we compare Eq. 7 (unlearning training samples), Eq. 8 (unlearning training samples with counterfactual pair), and Eq. 9 (unlearning external counterfactual pairs). Comparing Eq. 7 and Eq. 8, we find that under the same other settings, introducing counterfactual samples to remove bias proves to be a more efficient approach. Furthermore, Eq. 9 serves as an alternative solution to Eq. 8 when training data is unavailable. Empirical evidence indicates that the performance of Eq. 9 is slightly lower than that of Eq. 8.

Directly comparing Eq.7 and Eq.9 could encounter issues due to non-uniform variables. However, intuitively, the observation that Eq.7 performs worse than Eq.9 suggests that debiasing by counterfactual samples is more efficient than directly unlearning harmful training data.


[1] Understanding black-box predictions via influence functions.

[2] Learning debiased representation via disentangled feature augmentation.

[3] Learning de-biased representations with biased representations.

[4] Learning from failure: De-biasing classifier from biased classifier.

[5] Distributionally robust neural networks for group shifts.

Meta-review: This paper presents a novel approach for model debiasing based on machine unlearning and counterfactual fairness. The method can identify and remove the bias with a small external dataset, without requirement for model retraining or time-consuming finetuning. Empirical results show that the proposed FMD can achieve superior debiasing performance across multiple datasets, network architectures and tasks. The paper is well motivated, and provides an extensive set of experiments to demonstrate the efficiency and effectiveness of the proposed method. Thus, this work is of value to the NeurIPS community, and all reviewers agree to accept this paper.
Decision: Accept (poster)