SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ruizhe Chen $^{1}$ , Jianfei Yang $^{2}$ , Huimin Xiong $^{1}$ , Jianhong Bai $^{1}$ , Tianxiang Hu $^{1}$ , Jin Hao $^{3}$ , Yang Feng $^{4}$ , Joey Tianyi Zhou $^{5}$ , Jian Wu $^{1}$ , Zuozhu Liu $^{1*}$
<sup>1</sup> Zhejiang University <sup>2</sup> Nanyang Technological University
<sup>3</sup> Stanford University <sup>4</sup> Angelalign Technology Inc. <sup>5</sup> Centre for Frontier AI Research ruizhec.21@intl.zju.edu.cn
Recent discoveries have revealed that deep neural networks might behave in a biased manner in many real-world scenarios. For instance, deep networks trained on a large-scale face recognition dataset CelebA tend to predict blonde hair for females and black hair for males. Such biases not only jeopardize the robustness of models but also perpetuate and amplify social biases, which is especially concerning for automated decision-making processes in healthcare, recruitment, etc., as they could exacerbate unfair economic and social inequalities among different groups. Existing debiasing methods suffer from high costs in bias labeling or model re-training, while also exhibiting a deficiency in terms of elucidating the origins of biases within the model. To this respect, we propose a fast model debiasing framework (FMD) which offers an efficient approach to identify, evaluate and remove biases inherent in trained models. The FMD identifies biased attributes through an explicit counterfactual concept and quantifies the influence of data samples with influence functions. Moreover, we design a machine unlearning-based strategy to efficiently and effectively remove the bias in a trained model with a small counterfactual dataset. Experiments on the Colored MNIST, CelebA, and Adult Income datasets along with experiments with large language models demonstrate that our method achieves superior or competing accuracies compared with state-of-the-art methods while attaining significantly fewer biases and requiring much less debiasing cost. Notably, our method requires only a small external dataset and updating a minimal amount of model parameters, without the requirement of access to training data that may be too large or unavailable in practice.
Biased predictions are not uncommon in well-trained deep neural networks [1-3]. Recent findings indicate that many deep neural networks exhibit biased behaviors and fail to generalize to unseen data [4, 5], e.g., convolutional neural networks (CNNs) might favor texture over shape for object classification [6]. For instance, well-trained networks on a large-scale dataset (e.g. CelebA) tend to predict a female person to be with blonde hair, and a male to be with black hair [7, 8]. This is because the number of <blonder hair, female> and <black hair, male> image pairs is significantly higher than others, although there is no causal relationship between hair color and gender [9]. In this case, the model does not learn the correct classification strategy based on human appearance, but rather shows a preference for specific individuals or groups based on irrelevant attributes (error correlations) [2]. Such error correlations not only affect the model's ability to make robust predictions but also perpetuate and exacerbate social bias, resulting in potential risks in many real-world scenarios, such as racism, underestimating minorities, or social disparities among groups in crime prediction [10], loan assessment [11], and recruitment [12] etc.
Efforts have been made to remove bias in models based on innate or acquired characteristics of individuals or groups. Existing debiasing mechanisms could be categorized into three types depending on when debiasing is conducted: pre-processing, in-processing, and post-processing [2, 13, 14]. Pre-processing debiasing methods usually modify the dataset for fair learning, which often involve reweighing samples [15, 16], modifying feature representations [17, 18], changing labels [19] etc. Another line of research accounts for fairness during training, i.e., in-processing [20-24], including feature-level data augmentation or adversarial training [25, 26] etc. However, the aforementioned methods require expensive costs for human labeling of misleading biases or computationally-intensive debiased model retraining, resulting in unsatisfactory scalability over modern large-scale datasets or models. Few research explore post-processing strategies to achieve fairness with minimal cost [27-29]. They ensure group fairness by alternating predictions of some selected samples, causing degraded accuracy or unfairness on individuals. Moreover, most methods assume that the biased attributes were known, while a generalized debiasing framework should be able to verify whether an attribute (e.g. shape, texture, and color in an image classification task) is biased or not as well [30].
Figure 1: Pipeline of our proposed FMD.
In this paper, we propose FMD, an all-inclusive framework for fast model debiasing. As illustrated in Fig. 1, the FMD comprises three distinct steps: bias identification, biased-effect evaluation, and bias removal. In contrast to pre- or in-processing debiasing methods, our approach eliminates the need for supervised retraining of the entire model or additional labeling of bias attributes. Notably, FMD leverages only a small external dataset, thereby obviating the requirement for access to extensive or unavailable training data in practical scenarios. Furthermore, achieving fair outputs through FMD necessitates updating only a minimal number of parameters, such as the top MLP layers of pre-trained deep networks. Compared to post-processing debiasing methods, FMD yields superior debiasing performance and consistently enhances fairness across diverse bias metrics with little costs.
The FMD operates through the following procedure. Given an attribute and a well-trained model, our first step is to ascertain whether and to what extent the model exhibits bias towards the attribute. To achieve this, we construct a dataset comprising factual samples along with their corresponding counterfactual samples [31], wherein the attribute in question can be varied. By observing how the model's predictions change with the attribute variations, we can effectively identify any bias present. In the biased-effect evaluation phase, we quantitatively assess the extent to which a biased training sample contributes to the model's biased predictions. This evaluation entails measuring how the biased training sample misleads the model and influences its predictions. To this end, we extend the theory of influence functions [32], employing it to estimate the impact of perturbing a biased attribute within the training data on the model's prediction bias measurement. Finally, we introduce an unlearning mechanism that involves performing a Newton step [33] on the learned model parameters to remove the learned biased correlation. We further design an alternative strategy to unlearn biases with the counterfactual external dataset, avoiding hard requirements on access to the training data which might be unavailable in practice. Our unlearning strategy effectively eliminates the estimated influence of the biased attribute, leading to a more fair and unbiased model. Experiments on multiple datasets show that our method can achieve accuracies on par with bias-tailored training methods with a much smaller counterfactually constructed dataset. The corresponding biases and computational costs are significantly reduced as well. Our main contributions are summarized as:
- We propose a counterfactual inference-based framework that can quantitatively measure the biased degree of a trained (black-box) deep network with respect to different data attributes with a novel influence function.
- We propose an unlearning-based debiasing method that effectively and efficiently removes model biases with a small counterfactual dataset, getting rid of expensive network re-training or bias labeling. Our approach inherently applies to in-processing debiasing.
- Extensive experiments and detailed analysis on multiple datasets demonstrate that our framework can obtain competing accuracies with significantly smaller biases and much fewer data and computational costs.
2.1 Group, Individual and Counterfactual Fairness
The pursuit of fairness in machine learning has led to the proposal of fairness-specific metrics. These metrics have been mainly categorized into two types: metrics for group fairness that require similar average outputs of different demographic groups [34-38]; and metrics for individual fairness that necessitate similarity in the probability distributions of individuals that are similar in respect to a specific task, regardless of their demographic group [39-42]. Generally, statistical parity among protected groups in each class (group fairness) could be intuitively unfair at the individual level [43]. Moreover, existing fairness metrics put a heavy emphasis on model predictions, while underestimating the significance of sensitive attributes for decision-making and are insufficient to explain the cause of unfairness in the task [31, 44]. Recently, [31] introduces counterfactual fairness, a causal approach to address individual fairness, which enforces that the distribution of potential predictions for an individual should remain consistent when the individual's protected attributes had been different in a causal sense. In contrast to existing individual bias metrics, counterfactual fairness can explicitly model the causality between biased attributes and unfair predictions, which provides explainability for different biases that may arise towards individuals based on sensitive attributes [45-47].
2.2 Bias Mitigation
Proposed debiasing mechanisms are typically categorized into three types[2, 13, 14]: pre-processing, in-processing, and post-processing. Pre- and in-processing algorithms account for fairness before and during the training process, where typical techniques entail dataset modification [15-19] and feature manipulation [20-24, 26, 25]. Post-processing algorithms are performed after training, intending to achieve fairness without the need of modifying data or re-training the model. Current post-processing algorithms make more fair decisions by tweaking the output scores [48-50]. For instance, Hardt [27] achieves equal odds or equal opportunity by flipping certain decisions of the classifier according to their sub-groups. [29, 28] select different thresholds for each group, in a manner that maximizes accuracy and minimizes demographic parity. However, achieving group fairness by simply changing the predictions of several individuals is questionable, e.g., the process might be unfair to the selected individuals, leading to an unsatisfactory trade-off between accuracy and fairness.
2.3 Machine Unlearning
Machine unlearning [51-53] is a new paradigm to forget a specific data sample and remove its corresponding influence from a trained model, without the requirement to re-train the model from scratch. It fulfills a user's right to unlearn her private information, i.e., the right to be forgotten, in accordance with requests from the General Data Protection Regulation (GDPR) [54]. Existing unlearning approaches can be roughly categorized into two types: exact unlearning [55, 56] and approximate unlearning [57-60]. Data influence-based unlearning is a representative branch of approximate unlearning that utilizes influence functions [32] to approximate and remove the effect of a training sample on the model's parameters [61-63]. In this paper, we are inspired by the paradigm of machine unlearning and extend it to remove the model's bias from a deep network without retraining it from scratch.
3.1 Overview and Preliminaries
Problem Formulation. Consider a supervised prediction task with fairness considerations that maps input attributes  $\mathcal{A}$  (biased attribute) and  $\mathcal{X}$  (other attributes except  $\mathcal{A}$ ) to certain outputs  $\mathcal{Y}$  (labels). The training dataset  $D_{tr}$  can be represented as  $\{z_1, z_2, \ldots, z_n\}$  where each training point  $z_i = \{(a_i, x_i), y_i\} \in \mathcal{A} \times \mathcal{X} \times \mathcal{Y}$ . Let  $f_{\hat{\theta}}$  denote the trained model (predictor) with parameter  $\hat{\theta}$ . Let  $L(z_i, \theta)$  denote the loss on the training sample  $z_i$  w.r.t. parameter  $\theta$ . It is deemed biased if a biased attribute  $a$  is highly correlated but wrongly correlated to the prediction  $\hat{y} = f_{\hat{\theta}}(x, a)$ , e.g., a CNN is biased if it predicts hair color (black/blonde) with the biased attribute genders (male/female).
Motivation. In large part, existing works focused on measuring fairness with implicit quantitative values (e.g. accuracy). However, they do not provide explicit illustrations on whether the decision-making is based on sensitive/protected attributes. Furthermore, based on the bias identified, research on how such bias is learned from training samples is limited. Our proposed method bridges this gap with two components: identifying bias from different predictions with counterfactual samples and evaluating the biased-effect from training samples with a modified influence function. Furthermore, we propose a novel machine unlearning-based method to efficiently and effectively remove the biases.
Counterfactual Fairness. We identify the biases of trained models with the concept of counterfactual fairness [31, 46, 45] which better models the causality between biased attributes and unfair predictions. We detail the definition following [31]:
Definition 1 (Counterfactual fairness). A trained model  $f_{\hat{\theta}}$  is counterfactual fair on  $\mathcal{A}$  if for any  $a, \bar{a} \in \mathcal{A}$ ,
$$
P \left(\hat {Y} _ {A \leftarrow a} = y \mid X = x, A = a\right) = P \left(\hat {Y} _ {A \leftarrow \bar {a}} = y \mid X = x, A = a\right), \tag {1}
$$
for all  $x\in \mathcal{X}$  attainable by  $X$
Note that  $y = f_{\bar{\theta}}(X, A)$ , which implies the process of attribute changing. The definition suggests that, for any individual, changing  $a$ , i.e., from  $a$  to  $\bar{a}$ , while holding other attributes  $x$  unchanged should not change the distribution of  $\hat{Y}$  if  $a$  is a biased attribute.
Influence function. Influence functions, a standard technique from robust statistics, are recently extended to characterize the contribution of a given training sample to predictions in deep networks [32, 64, 65], e.g., identify whether a sample is helpful or harmful for model predictions. A popular implementation of influence functions is to approximate the effects by applying the perturbation  $z = (x,y) \mapsto z_{\delta} = (x + \delta, y)$  [32] that define the parameters resulting from moving  $\epsilon$  mass from  $z$  onto  $z_{\delta}$ :  $\hat{\theta}_{\epsilon, z_{\delta}, -z} = \operatorname*{argmin}_{\theta \in \Theta} \frac{1}{n} \sum_{i=1}^{n} L(z_i, \theta) + \epsilon L(z_{\delta}, \theta) - \epsilon L(z, \theta)$ . An approximated computation of the influence as in [32] can be defined as:
$$
\left. \frac {d \hat {\theta} _ {\epsilon , z _ {\delta} , - z}}{d \epsilon} \right| _ {\epsilon = 0} = - H _ {\hat {\theta}} ^ {- 1} \left(\nabla_ {\theta} L \left(z _ {\delta}, \hat {\theta}\right) - \nabla_ {\theta} L (z, \hat {\theta})\right). \tag {2}
$$
3.2 Bias Identification and Biased-Effect Evaluation
Counterfactual bias identification. We first identify the biases in a trained model with counterfactual concepts. Given a trained model  $f_{\hat{\theta}}$  and an attribute of interest  $\mathcal{A}$ , a primary question is whether  $f_{\hat{\theta}}$  is fair on  $\mathcal{A}$ . We employ an external dataset  $D_{ex}$  (can be constructed from the test set) to identify biases. To measure how prediction changes in accordance with the attribute, for each sample  $c_i = (x_i, a_i) \in D_{ex}$ , where  $a_i \in \mathcal{A}$ , we alter  $a_i$  while keeping  $x_i$  unchanged based on the requirements of counterfactual fairness. The generated counterfactual sample is denoted as  $\bar{c}_i = (x_i, \bar{a}_i)$ ,  $\bar{a}_i \in \mathcal{A}$ . We further define the counterfactual bias of the model  $f_{\hat{\theta}}$  on sample  $c_i$  as the difference in predictions:
$$
B \left(c _ {i}, \mathcal {A}, \hat {\theta}\right) = \left| P \left(\hat {Y} = f _ {\hat {\theta}} (X, A)\right) \mid X = x _ {i}, A = a _ {i})\right) - P \left(\hat {Y} = f _ {\hat {\theta}} (X, A) \mid X = x _ {i}, A = \bar {a} _ {i}\right) \Bigg |. \tag {3}
$$
The counterfactual bias on the whole dataset  $D_{ex}$  can be represented as the average of individual counterfactual biases:
$$
B \left(D _ {e x}, \mathcal {A}, \hat {\theta}\right) = \frac {1}{\left| D _ {e x} \right|} \sum_ {i} B \left(c _ {i}, \mathcal {A}, \hat {\theta}\right). \tag {4}
$$
The measured bias is a scalar normalized from 0 to 1. We set a bias threshold  $\delta$  that if the measured  $B(D_{ex},\mathcal{A},f_{\hat{\theta}})$  is larger than  $\delta$ , we regard  $f_{\hat{\theta}}$  to be biased on  $\mathcal{A}$ . Note that our method could also generalize to other individual bias metrics besides Eq. 3.
Biased-Effect Evaluation. Based on the identified counterfactual bias, we then investigate how the bias on  $\mathcal{A}$  is learned by the model from training samples. Considering  $B(\hat{\theta})$  measured on any  $\mathcal{A}$  with any  $D_{ex}$ , our goal is to quantify how each training point  $z_{k}$  in the training set  $D_{tr}$  contributes to  $B(\hat{\theta})$ . Let's denote the empirical risk minimizer as  $\hat{\theta} = \arg \min_{\theta} \frac{1}{n} \sum_{i=1}^{n} L(z_{i}, \theta)$ , and assume that the empirical risk is twice-differentiable and strictly convex in  $\theta$ . The influence function [64] provides an approximation on the updates to parameters if  $z_{k}$  were removed from  $D_{tr}$  with a small coefficient  $\epsilon$ . The new parameters can be obtained as  $\hat{\theta}_{\epsilon, z_{k}} = \arg \min_{\theta} \frac{1}{n} \sum_{i=1, i \neq k}^{n} L(z_{i}, \theta) + \epsilon L(z_{k}, \theta)$ . By doing so, the influence of removing  $z_{k}$  on the bias  $B(\hat{\theta})$  can be defined as:
$$
I _ {u p, b i a s} \left(z _ {k}, B (\hat {\theta})\right) = \frac {d B \left(\hat {\theta} _ {\epsilon , z _ {k}}\right)}{d \epsilon} \Big | _ {\epsilon = 0} = \frac {d B \left(\hat {\theta} _ {\epsilon , z _ {k}}\right)}{d \hat {\theta} _ {\epsilon , z _ {k}}} \frac {d \hat {\theta} _ {\epsilon , z _ {k}}}{d \epsilon} \Big | _ {\epsilon = 0} = - \nabla_ {\hat {\theta}} B (\hat {\theta}) H _ {\hat {\theta}} ^ {- 1} \nabla_ {\hat {\theta}} L \left(z _ {k}, \hat {\theta}\right), \tag {5}
$$
where  $H_{\hat{\theta}} \stackrel{\mathrm{def}}{=} \frac{1}{n} \sum_{i=1}^{n} \nabla_{\theta}^2 L(z_k, \hat{\theta})$  is the positive definite (PD) Hessian, and the closed form expression of  $\left. \frac{d\hat{\theta}_{\epsilon, z_k}}{d\epsilon} \right|_{\epsilon=0}$ , explaining the influence of  $z_k$  to model parameters, is provided by the influence function [32]. Note that "up" denotes "upweight". Refer to Appendix A for the derivation. Intuitively, this equation can be understood in two parts: the latter part calculates the impact of removing on the parameters. The former part corresponds to the derivative of bias with respect to parameters, assessing how changes in parameters affect the bias. Hence, this equation quantifies the influence of removing on the bias. Note that  $B(\hat{\theta})$  can be any bias measurement of interest. Taking  $B(D_{ex}, \mathcal{A}, \hat{\theta})$  defined in Eq. 4 as an example, the influence on counterfactual bias can be boiled down as:
$$
I _ {u p, b i a s} \left(z _ {k}, B \left(D _ {e x}, \mathcal {A}, \hat {\theta}\right)\right) = \frac {1}{\left| D _ {e x} \right|} \sum_ {c _ {i} \in D _ {e x}} \left(\nabla_ {\hat {\theta}} f _ {\hat {\theta}} \left(c _ {i}\right) - \nabla_ {\hat {\theta}} f _ {\hat {\theta}} \left(\bar {c} _ {i}\right)\right) H _ {\hat {\theta}} ^ {- 1} \nabla_ {\hat {\theta}} L \left(z _ {k}, \hat {\theta}\right), \tag {6}
$$
where  $I_{up, bias}(z_k, B)$  is a scalar that measures how each training sample contributes to  $B$ . If removing the point  $z_k$  increases the bias, we regard  $z_k$  as a helpful sample, or harmful otherwise. We provide an illustration of the helpful and harmful samples with a toy example in Section B.1.
3.3 Bias Removal via Machine Unlearning
After quantifying how biases are learned by the model from harmful samples, the next question is how to remove such biases. Here we propose a machine unlearning-based strategy to remove the biases caused by harmful samples. In particular, we exploit the powerful capability of machine unlearning paradigms for forgetting certain training samples [66, 62, 63, 61]. Specifically, for a bias measurement  $B(\hat{\theta})$ , we first rank the influence  $I_{up,bias}(z_k, B(\hat{\theta}))$  of every training sample  $z_k$  in  $D_{tr}$ , and then select the top-  $K$  harmful samples. Afterward, we unlearn, i.e., let the model forget, these samples by updating the model parameters  $\theta$  with a Newton update step as in [63]:
$$
\theta_ {n e w} = \hat {\theta} + \sum_ {k = 1} ^ {K} H _ {\hat {\theta}} ^ {- 1} \nabla_ {\hat {\theta}} L \left(z _ {k}, \hat {\theta}\right), \tag {7}
$$
where  $H_{\hat{\theta}}^{-1}\nabla_{\hat{\theta}}L(z_k,\hat{\theta}) = I_{up,params}(z_k)$  is explained as the influence of  $z_{k}$  on model parameter [32]. Note that  $I_{up,params}(z_k)$  share similar computation as in Eq. 6, while  $I_{up,params}(z_k)$  estimates the influence on model parameter and  $I_{up,bias}(z_k,B)$  focuses on influence on biases.
Our unlearning strategy is further refined following the observations from experiments in Section B.1. In particular, by ranking and visualizing the harmful and helpful samples on the biases (as shown in Fig. 5), we have observed that the harmful samples heavily lead to biased/incorrect correlations (i.e., bias-aligned) while the helpful samples behave oppositely (i.e., bias-conflicting). Hence, we propose a straightforward solution that further mitigates the influence of a harmful sample with a
bias-conflicting sample. Consequently, we update the parameters to unlearn the harmful samples by:
$$
\theta_ {n e w} = \hat {\theta} + \sum_ {k = 1} ^ {K} H _ {\hat {\theta}} ^ {- 1} \left(\nabla_ {\hat {\theta}} L \left(z _ {k}, \hat {\theta}\right) - \nabla_ {\hat {\theta}} L \left(\bar {z} _ {k}, \hat {\theta}\right)\right), \tag {8}
$$
where  $\bar{z}_k$  denotes the bias-conflicting sample of  $z_k$ . Following the explanation in influence theory [32], our unlearn mechanism removes the effect of perturbing a training point  $(\bar{a}, x, y)$  to  $(a, x, y)$ . In other words, we not only remove the influence caused by harmful sample  $z_k$ , but further ensure fairness with the corresponding counterfactual sample  $\bar{z}_k$ , see more details in Section B.1, 4.4 and Appendix.
Alternative Efficient Unlearn with Cheap External Datasets. In the above sections, the unlearning process is based on the assumption that we could access the original training sample  $z_{k}$  to identify and evaluate biases and then forget them. However, in practice, the training set might be too large or even unavailable in the unlearning phase. In response, we further propose to approximate the unlearning mechanism with a small external dataset. As the influence to be removed can be obtained from the change of the protected attribute, we can construct the same modification to the protected attribute on external samples. In particular, we employ the  $D_{ex}$  as in Section 3.2 to construct counterfactual pairs for unlearning, which redefines Eq. 21 as:
$$
\theta_ {n e w} = \hat {\theta} + \sum_ {i} H _ {\hat {\theta}} ^ {- 1} \left(\nabla_ {\hat {\theta}} L \left(c _ {i}, \hat {\theta}\right) - \nabla_ {\hat {\theta}} L \left(\bar {c} _ {i}, \hat {\theta}\right)\right). \tag {9}
$$
As  $D_{ex}$  can be easily obtained from an external dataset rather than the training set, the practical applicability of our method could be greatly enhanced, as demonstrated in the experiments.
3.4 Model Generalization