SlowGuess/NAACL_2022_OCR huggingface dataset:
Aili Shen\* Xudong Han\* Trevor Cohn\* Timothy Baldwin\* Lea Frermann\*
$\spadesuit$  School of Computing and Information Systems
{aili.shen,t.cohn,tbaldwin,lfrermann}@unimelb.edu.au,xudongh1@student.unimelb.edu.au
Real-world datasets often encode stereotypes and societal biases. Such biases can be implicitly captured by trained models, leading to biased predictions and exacerbating existing societal preconceptions. Existing debiasing methods, such as adversarial training and removing protected information from representations, have been shown to reduce bias. However, a disconnect between fairness criteria and training objectives makes it difficult to reason theoretically about the effectiveness of different techniques. In this work, we propose two novel training objectives which directly optimise for the widely-used criterion of equal opportunity, and show that they are effective in reducing bias while maintaining high performance over two classification tasks.
Modern neural machine learning has achieved great success across a range of classification tasks. However, when applied over real-world data, especially in high-stakes settings such as hiring processes and loan approvals, care must be taken to assess the fairness of models. This is because real-world datasets generally encode societal preconceptions and stereotypes, thereby leading to models trained on such datasets to amplify existing bias and make biased predictions (i.e., models perform unequally towards different subgroups of individuals). This kind of unfairness has been reported over various NLP tasks, such as part-of-speech tagging (Hovy and Søgaard, 2015; Li et al., 2018; Han et al., 2021b), sentiment analysis (Blodgett et al., 2016; Shen et al., 2021), and image activity recognition (Wang et al., 2019; Zhao et al., 2017).
Various methods have been proposed to mitigate bias, including adversarial training, and pre- and post-processing strategies. Adversarial training aims to make it difficult for a discriminator to
predict protected attribute values from learned representations (Han et al., 2021c; Elazar and Goldberg, 2018; Madras et al., 2018). Pre- and post-processing strategies vary greatly in approach, including transforming the original dataset to reduce protected attribute discrimination while retaining dataset utility (du Pin Calmon et al., 2017), iteratively removing protected attribute information from (fixed) learned representations (Ravfogel et al., 2020), or reducing bias amplification by injecting corpus-level constraints during inference (Zhao et al., 2017).
However, training strategies and optimisation objectives are generally disconnected from fairness metrics which directly measure the extent to which different groups are treated (in)equitably. This makes it difficult to understand the effectiveness of previous debiasing methods from a theoretical perspective. In this work, we propose to explicitly incorporate equal opportunity into our training objective, thereby achieving bias reduction. This paper makes the following contributions:
1. We are the first to propose a weighted training objective that directly implements fairness metrics.
2. Observing that model performance for different classes can vary greatly, we further propose a variant of our method, taking both bias reduction among protected attribute groups and bias reduction among different classes into consideration.
3. Experimental results over two tasks show that both proposed methods are effective at achieving fairer predictions, while maintaining performance.
Our code is available at: https://github.com/
AiliAili/Difference_Mean_Fair_Models.
*Equal contributors to this work.
Human Language Technologies, pages 4073 - 4084
2.1 Fairness Criteria
Various criteria have been proposed to capture different types of discrimination, such as group fairness (Hardt et al., 2016; Zafar et al., 2017a; Cho et al., 2020; Zhao et al., 2020), individual fairness (Sharifi-Malvajerdi et al., 2019; Yurochkin et al., 2020; Dwork et al., 2012), and causality-based fairness (Wu et al., 2019; Zhang and Bairenboim, 2018a,b). In this work, we focus on group fairness, whereby a model should perform equally across different demographic subgroups.
To quantify how predictions vary across different demographic subgroups, demographic parity (Feldman et al., 2015; Zafar et al., 2017b; Cho et al., 2020), equal opportunity (Hardt et al., 2016; Madras et al., 2018), and equalised odds (Cho et al., 2020; Hardt et al., 2016; Madras et al., 2018) are widely used to measure fairness. Demographic parity ensures that models achieve the same positive rate for each demographic subgroup, oblivious the ground-truth target label. Equal opportunity requires that a model achieves the same true positive rate (TPR) across different subgroups, considering only instances with a positive label. Equalised odds goes one step further in requiring not only the same TPR but also the same false positive rate (FPR) across groups.
Demographic parity, equal opportunity, and equalised odds only focus on the prediction outcome for one specific target label (i.e. a "positive" class) in a binary classification setting, but does not apply fairness directly to multi-class settings, when fairness for different subgroups across all classes is required. Equal opportunity can be generalised by extending the "positive" class to each target class, as we do in our work.
2.2 Debiasing Methods
A broad range of methods has been proposed to learn fair models. Based on where debiasing occurs, in terms of dataset processing, model training, and inference, we follow Cho et al. (2020) in categorising methods into: (1) pre-processing, (2) post-processing, and (3) in-processing.
Pre-processing methods manipulate the original dataset to mitigate discrimination (Wang et al., 2019; Xu et al., 2018; Feldman et al., 2015; du Pin Calmon et al., 2017; De-Arteaga et al., 2019). For example, du Pin Calmon et al. (2017) transform the original dataset to reduce discrimin-
ination while retaining dataset utility. Class imbalance methods used in bias reduction, such as dataset sampling (Kubat and Matwin, 1997; Wallace et al., 2011), instance reweighting (Cui et al., 2019; Li et al., 2020; Lin et al., 2017), and weighted max-margin (Cao et al., 2019), also belong to this category. For example, Lahoti et al. (2020), Subramanian et al. (2021b), and Han et al. (2021a) reweight instances by taking the (inverse of) joint distribution of the protected attribute classes and main task classes into consideration. Wang et al. (2019) and Han et al. (2021a) down-sample the majority protected attribute group within each target class, and train on the resulting balanced dataset.
Post-processing methods calibrate the prediction outcome or learned representations of models to achieve fair predictions (Hardt et al., 2016; Pleiss et al., 2017; Zhao et al., 2017; Ravfogel et al., 2020). For example, Zhao et al. (2017) enforce a corpus-level constraint during inference to reduce bias. Ravfogel et al. (2020) iteratively remove protected attribute information from representations generated by an fixed encoder, by iteratively training a discriminator over the projected attribute and projecting the representation into the discriminator's null space.
In-processing methods learn fair models during model training. One family of approaches is based on constrained optimisation, incorporating fairness measures as regularisation terms or constraints (Zafar et al., 2017b; Subramanian et al., 2021a; Donini et al., 2018; Narasimhan, 2018; Cho et al., 2020). For example, Zafar et al. (2017a) translate equalised odds into constraints on FPR and FNR across groups, and solve using constraint programming. Cho et al. (2020) adopt kernel density estimation to quantify demographic parity and equalised odds, but in a manner which is limited to low-dimensional data and binary classification tasks. Another line of work is to use adversarial training to obtain fair models, in jointly training an encoder and discriminator(s) over the encoded representations such that the discriminator(s) are ineffective at predicting the protected attribute values from learned representations (Han et al., 2021c; Elazar and Goldberg, 2018; Madras et al., 2018; Zhang et al., 2018; Agarwal et al., 2018; Roh et al., 2020). Elsewhere, Shen et al. (2021) use contrastive learning to learn fair models by simultaneously pushing instances belonging to the same target class closer and pulling instances belonging
to the same protected attribute class further apart.
The most relevant work to ours is FairBatch (Roh et al., 2021). It proposes to formulate the original task as a bi-level optimisation problem, where the inner optimiser is the standard training algorithm and the outer optimiser is responsible for adaptively adjusting the sampling probabilities of instances with a given target class and protected attribute value, based on the equal opportunity metric achieved by the intermediate inner model. That is, they adaptively adjust the instance resampling probability during training to reduce bias. However, different from FairBatch, whose resampling strategy is bound by the sampling probability [0, 1], our proposed method achieves bias reduction by reweighting instances during training, where the reweighting range is unbounded, leading to greater flexibility in trading off performance and fairness.
Suppose we have some data  $X\in \mathbb{R}^n$  , target labels  $Y\in C$  , and protected attribute values  $A = \{0,1\}$  where  $C$  is the number of target classes for a given task.
Equal opportunity A classifier is said to satisfy equal opportunity if its prediction is conditionally independent of the protected attribute  $A$  given the target label  $Y$ ,  $\{P(\hat{y} = y|Y = y, A = 0) = P(\hat{y} = y|Y = y, A = 1)\}$  for  $\forall y \in Y$ . Here,  $\hat{y}$  is a prediction outcome,  $y \in Y$  and  $a \in A$ . As mentioned above, we slightly modify the definition of equal opportunity by allowing  $y$  to be each candidate target class, accommodating multi-class settings. We explicitly address the fairness criterion across all target classes by promoting comparable true positive rates across protected classes.
3.2 Optimising Equal Opportunity
Instead of using a fairness proxy (Zafar et al., 2017b) or kernel density estimation to quantify fairness (Cho et al., 2020), we propose to optimise equal opportunity by directly minimising the absolute difference in loss between different subsets of instances belonging to the same target label but with different protected attribute classes,
$$
\mathcal {L} _ {\mathrm {e o}} ^ {\text {c l a s s}} = \mathcal {L} _ {c e} + \lambda \sum_ {y = \in C} \sum_ {a \in A} \left| \mathcal {L} _ {c e} ^ {y, a} - \mathcal {L} _ {c e} ^ {y} \right| \tag {1}
$$
Here,  $\mathcal{L}_{ce}$  denotes the average cross-entropy loss based on instances in the batch;  $\mathcal{L}_{ce}^{y,a}$  denotes the av
erage cross-entropy loss computed over instances with the target label  $y$  and the protected attribute label  $a$ ; and  $\mathcal{L}_{ce}^{y}$  denotes the average cross-entropy loss computed over all instances with target label  $y$ . Our proposed loss  $\mathcal{L}_{\mathrm{eo}}^{\mathrm{class}}$  is the weighted sum of the overall cross-entropy and the sum of the cross-entropy difference for each target label overall and that conditioned on the target label, thereby capturing both performance and fairness. This method is denoted as  $\mathsf{EO}_{\mathrm{CLA}}$ , as it captures class-wise equal opportunity.
3.3 Equal Opportunity across Classes
One drawback of  $\mathsf{EO}_{\mathrm{CLA}}$  is that it only focuses on optimising equal opportunity, ignoring the fact that the performance for different classes can vary greatly, especially when the dataset is skewed. To learn fair models not only towards demographic subgroups but also across target classes, we propose a variant of Equation 1, by introducing one additional constraint on top of equal opportunity to encourage the label-wise cross entropy loss terms to align. Formally:  $\mathcal{L}_{ce}^{y_1}\approx \mathcal{L}_{ce}^{y_2}$ , where  $y_{1}\neq y_{2}$ , and  $y_{1}\in Y$ ,  $y_{2}\in Y$ . This objective encourages equal opportunity not only for demographic subgroups but also across different target classes:
$$
\mathcal {L} _ {\mathrm {e o}} ^ {\text {g l o b a l}} = \mathcal {L} _ {c e} + \lambda \sum_ {y = \in C} \sum_ {a \in A} | \mathcal {L} _ {c e} ^ {y, a} - \mathcal {L} _ {c e} | \tag {2}
$$
This method is denoted as  $\mathsf{EO}_{\mathrm{GLB}}$ , short for  $\underline{\underline{\mathrm{global}}}$  equal opportunity.
3.4 Theory
In this section, we show how our training objective is related to equal opportunity in the binary classification and binary protected attribute setting. Note that our proof naturally extends to cases where the numbers of target classes and/or protected attribute values are greater than two as described in Equations 1 and 2.
Let  $m_{y,a}$  be the number of training instances with target label  $y$  and protected attribute  $a$  in a batch. For example,  $m_{1,0}$  denotes the number of instances with target label 1 and protected attribute 0 in the batch. Let  $\mathcal{L}^{y,a}$  be the average loss for instances with target label  $y$  and protected attribute  $a$ . For example,  $\mathcal{L}^{1,0}$  is the average loss for instances with target label 1 and protected attribute 0.
3.4.1 Cross-Entropy Loss
The vanilla cross-entropy loss is computed as:
$$
\frac {1}{N} \left(m _ {0, 0} \mathcal {L} ^ {0, 0} + m _ {0, 1} \mathcal {L} ^ {0, 1} + m _ {1, 0} \mathcal {L} ^ {1, 0} + m _ {1, 1} \mathcal {L} ^ {1, 1}\right) \tag {3}
$$
which is the average loss over different subsets of instances with a given target label and protected attribute class.
3.4.2 Difference Loss
The  $\mathsf{EO}_{\mathrm{CLA}}$  method defined in Equation 1 can be written as:
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {e 0}} ^ {\text {c l a s s}} = \mathcal {L} _ {c e} + \lambda \sum_ {y, a} \left| \mathcal {L} _ {c e} ^ {y, a} - \mathcal {L} _ {c e} ^ {y} \right| \\ = \sum_ {y, a} \left[ \frac {m _ {y , a}}{N} \mathcal {L} _ {c e} ^ {y, a} + \lambda \mathrm {s i g n} _ {y, a} \left(\mathcal {L} _ {c e} ^ {y, a} - \mathcal {L} _ {c e} ^ {y}\right) \right], \\ = \sum_ {y, a} \left[ \left(\frac {m _ {y , a}}{N} + \lambda \mathrm {s i g n} _ {y, a}\right) \mathcal {L} _ {c e} ^ {y, a} \right. \\ \left. - \lambda \operatorname {s i g n} _ {y, a} \mathcal {L} _ {c e} ^ {y} \right], \tag {4} \\ \end{array}
$$
where sign is a sign function, and  $\mathrm{sign}_{y,a} = \mathrm{sign}\left(\mathcal{L}_{ce}^{y,a} - \mathcal{L}_{ce}^{y}\right)$ . Noting that for binary protected attributes,  $\mathrm{sign}_{y,a} = -\mathrm{sign}_{y,\neg a}$ , and  $\sum_{y,a}\mathcal{L}_{ce}^{y} = 0, \forall y$  in this case:
$$
\mathcal {L} _ {\mathrm {e o}} ^ {\text {c l a s s}} = \sum_ {y, a} \left(\frac {m _ {y , a}}{N} + \lambda \operatorname {s i g n} _ {y, a}\right) \mathcal {L} _ {c e} ^ {y, a} \tag {5}
$$
By comparing Equations 3 and 5, we can see that for target label  $y$ , our method dynamically increases the weight for poorly-performing subsets (i.e.  $\mathrm{sign}_{y,g} = 1$ ) by  $\lambda$ , and decreases the weight for well-performing subsets ( $\mathrm{sign}_{y,g} = -1$ ) by  $\lambda$ , thereby leading to fairer predictions by adjusting the weight for instances with different protected attribute classes conditioned on a given target label.
3.4.3 From Binary Cross-Entropy to True Positive Rate
Using the definition of binary cross-entropy
$$
- [ y _ {i} \cdot \log (p (y _ {i})) + (1 - y _ {i}) \cdot \log (1 - p (y _ {i})) ],
$$
the loss for a certain subset, e.g., the subset of instances with target label 1 and protected attribute class 0, can be simplified as:
$$
\begin{array}{l} \mathcal {L} ^ {1, 0} = - \frac {1}{m _ {1 , 0}} \sum_ {j = 1} ^ {m _ {1, 0}} \left(y _ {j} \cdot \log \left(p \left(\hat {y} _ {j}\right)\right) \right. \\ \left. + \left(1 - y _ {j}\right) \cdot \log \left(1 - p \left(\hat {y} _ {j}\right)\right)\right) \tag {6} \\ = - \frac {1}{m _ {1 , 0}} \sum_ {j = 1} ^ {m _ {1, 0}} \log (p (\hat {y} _ {j})) \\ \end{array}
$$
Notice that  $p(\hat{y}_j)$  is equivalent to  $p(\hat{y}_j = 1)$ , making  $\mathcal{L}^{1,0} = -\frac{1}{m_{1,0}}\sum_{j=1}^{m_{1,0}}\log(p(\hat{y}_j))$  an unbiased
estimator of  $-\log p(\hat{y} = 1|y = 1)$ , which approximates  $-\log$  TPR.
Minimising the expectation of the absolute difference between  $\mathcal{L}^{1,0}$  and  $\mathcal{L}^{1,1}$  can approximate the true positive rate difference between two groups with the same target label 1:
$$
\begin{array}{l} \operatorname {a r g m i n} _ {\theta} \mathbb {E} \left(\left| \mathcal {L} ^ {1, 0} - \mathcal {L} ^ {1, 1} \right|\right) \\ = \operatorname {a r g m i n} | - \log p (\hat {y} = 1 | y = 1, g = 0) \\ - (- \log p (\hat {y} = 1 | y = 1, g = 1)) | \\ \approx \operatorname {a r g m i n} | \log \frac {T P R _ {1 , 0}}{T P R _ {1 , 1}} | \\ = \operatorname {a r g m i n} \left| T P R _ {1, 0} - T P R _ {1, 1} \right| \\ \end{array}
$$
This demonstrates that minimising the absolute difference between  $\mathcal{L}^{1,0}$  and  $\mathcal{L}^{1,1}$  is roughly equivalent to minimising the TPR difference between two groups with the same target label, which is precisely the formulation of equal opportunity, as described in Section 4.3. Therefore, the second term in our proposed method (Equation 1) is optimising directly for equal opportunity.
Our experiments compare the performance and fairness of our methods against various competitive baselines, and across two classification tasks.
We compare our proposed methods  $\mathsf{EO}_{\mathrm{CLA}}$  and  $\mathsf{EO}_{\mathrm{GLB}}$  against the following seven baselines:
1. CE: train the model with cross-entropy loss and no explicit bias mitigation.
2. INLP: first train the model with cross-entropy loss to obtain dense representations, and iteratively apply null-space projection to the learned representations to remove protected attribute information (Ravfogel et al., 2020). The resulting representations are used to make predictions.
3. Adv: jointly train the model with cross-entropy loss and an ensemble of three adversarial discriminators for the projected attribute, with an orthogonality constraint applied to the discriminators to encourage diversity (Han et al., 2021c).
4. DS: downsample the dataset corresponding to the protected attribute conditioned on a given target label (Han et al., 2021a).
5. RW: reweight instances based on the (inverse) joint distribution of the protected attribute classes and target classes (Han et al., 2021a).
6. Constrained: formulate the task as a constrained optimisation problem, where equal opportunity is incorporated as constraints (Subramanian et al., 2021a).
7. FairBatch: formulate the model training as a bi-level optimisation problem, as described in Section 2.2 (Roh et al., 2021).
For each task, we first obtain document representations from their corresponding pretrained models, which are not finetuned during training. Then document representations are fed into two fully-connected layers with a hidden size of 300d. For all experiments, we use the Adam optimiser (Kingma and Ba, 2015) to optimise the model for at most 60 epochs with early stopping and a patience of 5. All models are trained and evaluated on the same dataset splits, and models are selected based on their performance on the development set. We finetune the learning rate, batch size, and extra hyperparameters introduced by the corresponding debiasing methods for each model on each dataset (see the Appendix for details). Noting the complexity of model selection given the multi-objective accuracy-fairness tradeoff and the absence of a standardised method for selecting models based on both criteria in fairness research, we determine the best-achievable accuracy for a given model, and select the hyperparameter settings that reduce bias while maintaining accuracy as close as possible to the best-achievable value (all based on the dev set). We leave the development of a fair and robust model selection method to future work.
To evaluate the performance of models on the main task, we adopt  $\mathrm{F}_{1}^{\mathrm{micro}}$  and  $\mathrm{F}_{1}^{\mathrm{macro}}$  for all our datasets, taking class imbalance into consideration, especially in the multi-class setting.
To evaluate fairness, we follow previous work (De-Arteaga et al., 2019; Ravfogel et al., 2020) and adopt root mean square TPR gap over all classes, which is defined as
$$
\mathrm {G A P} = \sqrt {\frac {1}{| C |} \sum_ {y \in Y} \left(\mathrm {G A P} _ {y} ^ {\mathrm {T P R}}\right) ^ {2}},
$$
where  $\mathrm{GAP}_y^{\mathrm{TPR}} = |\mathrm{TPR}_{y,a} - \mathrm{TPR}_{y,\neg a}|$ ,  $y\in Y$  and  $\mathrm{TPR}_{a,y} = \mathbb{P}(\hat{y} = y|y,a)$ , indicating the proportion of correct predictions among instances with target label  $y$  and protected attribute label
$a$ .  $\mathrm{GAP}_{y}^{\mathrm{TPR}}$  measures the absolute performance difference between demographic subgroups conditioned on target label  $y$ , and a value of 0 indicates that the model makes predictions independent of the protected attribute.
4.4 Twitter Sentiment Analysis
4.4.1 Task and Dataset
For our first dataset, the task is to predict the binary sentiment for a given English tweet, where each tweet is also annotated with a binary protected attribute indirectly capturing the ethnicity of the tweet author as either African American English (AAE) or Standard American English (SAE). Following previous studies (Ravfogel et al., 2020; Han et al., 2021c; Shen et al., 2021), we adopt the dataset of Blodgett et al. (2016) (Moji hereafter), where the training dataset is balanced with respect to both sentiment and ethnicity but skewed in terms of sentiment-ethnicity combinations (40% HAPPY-AAE, 10% HAPPY-SAE, 10% SAD-AAE, and 40% SAD-SAE, respectively). The number of instances in the training, dev, and test sets are 100K, 8K, and 8K, respectively. The dev and test set are balanced in terms of sentiment-ethnicity combinations.
4.4.2 Implementation Details
Following previous work (Elazar and Goldberg, 2018; Ravfogel et al., 2020; Han et al., 2021c), we use DeepMoji (Felbo et al., 2017), a model pretrained over 1.2 billion English tweets, as the encoder to obtain text representations. The parameters of DeepMoji are fixed in training. Hyperparameter settings are provided in Appendix A.2.
4.4.3 Experimental Results
Table 1 presents the results over the Moji test set. Compared to CE, INLP and Adv moderately reduce model bias while simultaneously improving model performance. Surprisingly, both DS and RW reduce GAP substantially and achieve the joint best  $\mathrm{F}_{1}^{\mathrm{micro}}$ , indicating that the biased prediction is mainly due to the imbalanced distribution of protected attribute classes conditioned on a given target label, and the imbalanced distribution of sentiment-ethnicity combinations. $^{1}$  However,
<sup>1</sup>However, it does not hold the other way around as demonstrated by previous studies (Wang et al., 2019), indicating that a balanced dataset either in terms of target label and protected attribute combination, or in terms of protected attribute class distribution conditioned on target classes, can still lead to biased predictions.
Model	F1micro↑	GAP↓
CE	72.09±0.65	40.21±1.23
INLP	72.81±0.01	36.81±3.49
Adv	74.47±0.68	30.59±2.94
DS	76.16±0.28	14.96±1.08
RW	76.21±0.16†	14.70±0.86
Constrained	75.22±0.20	15.92±4.86
FairBatch	75.81±0.17	15.36±3.07
EOCLA	75.03±0.25	10.83±1.40†
EOGLB	75.20±0.20	11.49±1.07
Table 1: Experimental results on the Moji test set (averaged over 10 runs); Bold = Best Performance;  $\uparrow=$  the higher the better;  $\downarrow=$  the lower the better. The best result is marked with “†” if the difference over the next-best method is statistically significant (based on a one-tailed Wilcoxon signed-rank test;  $p<0.05$ ), noting that if the best method is one of our methods, we compare it to the next-best method which is not our own.