SlowGuess/ACL_2020_OCR huggingface dataset:
Guanhua Zhang $^{\ddagger \S *}$ , Bing Bai $^{\S *}$ , Junqi Zhang $^{\S}$ , Kun Bai $^{\S}$ , Conghui Zhu $^{\ddagger}$ , Tiejun Zhao $^{\ddagger}$
$^{\ddagger}$ Harbin Institute of Technology,  $^{\S}$ Tencent CSIG
{guanhzhang,icebai,benjqzhang,kunbai}@tencent.com, {chzhu,tjzhao}@hit-mtlab.net
With the recent proliferation of the use of text classifications, researchers have found that there are certain unintended biases in text classification datasets. For example, texts containing some demographic identity-terms (e.g., "gay", "black") are more likely to be abusive in existing abusive language detection datasets. As a result, models trained with these datasets may consider sentences like "She makes me happy to be gay" as abusive simply because of the word "gay." In this paper, we formalize the unintended biases in text classification datasets as a kind of selection bias from the non-discrimination distribution to the discrimination distribution. Based on this formalization, we further propose a model-agnostic debiasing training framework by recovering the non-discrimination distribution using instance weighting, which does not require any extra resources or annotations apart from a pre-defined set of demographic identity-terms. Experiments demonstrate that our method can effectively alleviate the impacts of the unintended biases without significantly hurting models' generalization ability.
With the development of Natural Language Processing (NLP) techniques, Machine Learning (ML) models are being applied in continuously expanding areas (e.g., to detect spam emails, to filter resumes, to detect abusive comments), and they are affecting everybody's life from many aspects. However, human-generated datasets may introduce some human social prejudices to the models (Caliskan-Islam et al., 2016). Recent works have found that ML models can capture, utilize, and even amplify the unintended biases (Zhao et al., 2017), which has raised lots of concerns about the
Identity-term	Count	Percentage Toxic
gay	868	57.4%
homosexual	202	34.4%
Mexican	116	21.6%
blind	257	14.8%
black	1,123	13.1%
overall	159,686	9.6%
Table 1: Percentage of toxic comments by some specific demographic identity-terms in the dataset released by Dixon et al. (2018).
discrimination problem in NLP models (Sun et al., 2019).
Text classification is one of the fundamental tasks in NLP. It aims at assigning any given sentence to a specific class. In this task, models are expected to make predictions with the semantic information rather than with the demographic group identity information (e.g., "gay", "black") contained in the sentences.
However, recent research points out that there widely exist some unintended biases in text classification datasets. For example, in a toxic comment identification dataset released by Dixon et al. (2018), it is found that texts containing some specific identity-terms are more likely to be toxic. More specifically,  $57.4\%$  of comments containing "gay" are toxic, while only  $9.6\%$  of all samples are toxic, as shown in Table 1.
Because of such a phenomenon, models trained with the dataset may capture the unintended biases and perform differently for texts containing various identity-terms. As a result, predictions of models may discriminate against some demographic minority groups. For instance, sentences like "She makes me happy to be gay" is judged as abusive by models trained on biased datasets in our experiment, which may hinder those minority groups who want to express their feelings on the web freely.
Recent model-agnostic research mitigating the
* Equal contributions from both authors. This work was done when Guanhua Zhang was an intern at Tencent.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4134-4145 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
unintended biases in text classifications can be summarized as data manipulation methods (Sun et al., 2019). For example, Dixon et al. (2018) propose to apply data supplementation with additional labeled sentences to make toxic/non-toxic balanced across different demographic groups. Park et al. (2018) proposes to use data augmentation by applying gender-swapping to sentences with identity-terms to mitigate gender bias. The core of these works is to transform the training sets to an identity-balanced one. However, data manipulation is not always practical. Data supplementation often requires careful selection of the additional sentences w.r.t. the identity-terms, the labels, and even the lengths of sentences (Dixon et al., 2018), bringing a high cost for extra data collection and annotation. Data augmentation may result in meaningless sentences (e.g., "He gives birth"), and is impractical to perform when there are many demographic groups (e.g., for racial bias cases).
In this paper, we propose a model-agnostic debiasing training framework that does not require any extra resources or annotations, apart from a pre-defined set of demographic identity-terms. We tackle this problem from another perspective, in which we treat the unintended bias as a kind of selection bias (Heckman, 1979). We assume that there are two distributions, the non-discrimination distribution, and the discrimination distribution observed in the biased datasets, and every sample of the latter one is drawn independently from the former one following a discrimination rule, i.e., the social prejudice. With such a formalization, mitigating the unintended biases is equivalent to recovering the non-discrimination distribution from the selection bias. With a few reasonable assumptions, we prove that we can obtain the unbiased loss of the non-discrimination distribution with only the samples from the observed discrimination distribution with instance weights. Based on this, we propose a non-discrimination learning framework. Experiments on three datasets show that, despite requiring no extra data, our method is comparable to the data manipulation methods in terms of mitigating the discrimination of models.
The rest of the paper is organized as follows. We summarize the related works in Section 2. Then we give our perspective of the problem and examine the assumptions of commonly-used methods in Section 3. Section 4 introduces our non-discrimination learning framework. Taking three datasets as ex
amples, we report the experimental results of our methods in Section 5. Finally, we conclude and present the future works in Section 6.
Non-discrimination and Fairness Non-discrimination focuses on a number of protected demographic groups, and ask for parity of some statistical measures across these groups (Chouldechova, 2017). As mentioned by Friedler et al. (2016), non-discrimination can be achieved only if all groups have similar abilities w.r.t. the task in the constructed space which contains the features that we would like to make a decision. There are various kinds of definitions of non-discrimination corresponding to different statistical measures. Popular measures include raw positive classification rate (Calders and Verwer, 2010), false positive and false negative rate (Hardt et al., 2016) and positive predictive value (Chouldechova, 2017), corresponding to different definitions of non-discrimination. Methods like adversarial training (Beutel et al., 2017; Zhang et al., 2018) and fine-tuning (Park et al., 2018) have been applied to remove biasedness.
In the NLP area, fairness and discrimination problems have also gained tremendous attention. Caliskan-Islam et al. (2016) show that semantics derived automatically from language corpora contain human biases. Bolukbasi et al. (2016) show that pre-trained word embeddings trained on large-scale corpus can exhibit gender prejudices and provide a methodology for removing prejudices in embeddings by learning a gender subspace. Zhao et al. (2018) introduce the gender bias problem in coreference resolution and propose a general-purpose method for debiasing.
As for text classification tasks, Dixon et al. (2018) first points out the unintended bias in datasets and proposes to alleviate the bias by supplementing external labeled data. Kiritchenko and Mohammad (2018) examines gender and race bias in 219 automatic sentiment analysis systems and finds that several models show significant bias. Park et al. (2018) focus on the gender bias in abusive language detection task and propose to debias by augmenting the datasets with gender-swapping operation. In this paper, we propose to make models fit a non-discrimination distribution with calculated instance weights.
Instance Weighting Instance weighting has been broadly adopted for reducing bias. For example, the Inverse Propensity Score (IPS) (Rosenbaum and Rubin, 1983) method has been successfully applied for causal effect analyses (Austin and Stuart, 2015), selection bias (Schonlau et al., 2009), position bias (Wang et al., 2018; Joachims et al., 2017) and so on. Zadrozny (2004) proposed a methodology for learning and evaluating classifiers under "Missing at Random" (MAR) (Rubin, 1976) selection bias. Zhang et al. (2019) study the selection bias in natural language sentences matching datasets, and propose to fit a leakage-neutral distribution with instance weighting. Jiang and Zhai (2007) propose an instance weighting framework for domain adaptation in NLP, which requires the data of the target domain.
In our work, we formalize the discrimination problem as a kind of "Not Missing at Random" (NMAR) (Rubin, 1976) selection bias from the non-discrimination distribution to the discrimination distribution, and propose to mitigate the unintended bias with instance weighting.
3 Perspective
In this section, we present our perspective regarding the discrimination problem in text classifications. Firstly, we define what the nondiscrimination distribution is. Then, we discuss what requirements non-discrimination models should meet and examine some commonly used criteria for non-discrimination. After that, we analyze some commonly used methods for assessing discrimination quantitatively. Finally, we show that the existing debiasing methods can also be seen as trying to recover the nondiscrimination distribution and examine their assumptions.
3.1 Non-discrimination Distribution
The unintended bias in the datasets is the legacy of the human society where discrimination widely exists. We denote the distribution in the biased datasets as discrimination distribution  $\mathcal{D}$ .
Given the fact that the real world is discriminatory although it should not be, we assume that there is an ideal world where no discrimination exists, and the real world is merely a biased reflection of the non-discrimination world. Under this perspective, we assume that there is an non-discrimination distribution  $\hat{\mathcal{D}}$  reflecting the ideal world, and the discrimination distribution  $\mathcal{D}$  is
drawn from  $\widehat{\mathcal{D}}$  but following a discriminatory rule, the social prejudice. Attempting to correct the bias of datasets is equivalent to recover the original nondiscrimination distribution  $\widehat{\mathcal{D}}$ .
For the text classification tasks tackled in this paper, we denote  $X$  as the sentences,  $Y$  as the (binary) label indicator variable<sup>1</sup>,  $Z$  as the demographic identity information (e.g. "gay", "black", "female") in every sentence. In the following paper, we use  $P(\cdot)$  to represent the probability of the discrimination distribution  $\mathcal{D}$  in datasets, and  $Q(\cdot)$  for non-discrimination distribution  $\widehat{\mathcal{D}}$ . Then the non-discrimination distribution  $\widehat{\mathcal{D}}$  should meet that,
$$
Q (Y | Z) = Q (Y),
$$
which means that the demographic identity information is independent of the labels $^2$ .
3.2 Non-Discrimination Model
For text classification tasks, models are expected to make predictions by understanding the semantics of sentences rather than by some single identity-terms. As mentioned in Dixon et al. (2018), a model is defined as biased if it performs better for sentences containing some specific identity-terms than for ones containing others. In other words, a non-discrimination model should perform similarly across sentences containing different demographic groups. However, "perform similarly" is indeed hard to define. Thus, we pay more attention to some criteria defined on demographic groups.
A widely-used criterion is Equalized Odds (also known as Error Rate Balance) defined by Choudechova (2017), requiring the  $\widehat{Y}$  to be independent of  $Z$  when  $Y$  is given, in which  $\widehat{Y}$  refers to the predictions of the model. This criterion is also used by Borkan et al. (2019) in text classifications.
Besides the Equalized Odds criterion, a straightforward criterion for judging non-discrimination is Statistical Parity (also known as Demographic Parity, Equal Acceptance Rates, and Group Fairness) (Calders and Verwer, 2010; Dwork et al., 2012), which requires  $\widehat{Y}$  to be independent of  $Z$ ,
<sup>1</sup>In this paper, we focus on binary classification problems, but the proposed methodology can be easily extended to multiclass classifications.
There may be a lot of distributions satisfying the equation. However, as we only focus on the discrimination problem in the text classification task, we suppose that there is a unique non-discrimination distribution  $\widehat{\mathcal{D}}$  which reflects the ideal world in the desired way and the observed biased dataset is drawn from it following a discriminatory rule.
i.e.,  $\operatorname{Pr}(\widehat{Y} | Z) = \operatorname{Pr}(\widehat{Y})$ . Another criterion is Predictive Parity (Chouldechova, 2017), which requires  $Y$  to be independent of  $Z$  when condition  $\widehat{Y} = 1$  is given, i.e.,  $\operatorname{Pr}(Y | \widehat{Y} = 1, Z) = \operatorname{Pr}(Y | \widehat{Y} = 1)$ . Given the definitions of the three criterions, we propose the following theorem, and the proof is presented in Appendix A.
Theorem 1 (Criterion Consistency). When tested in a distribution in which  $\operatorname{Pr}(Y|Z) = \operatorname{Pr}(Y)$ ,  $\widehat{Y}$  satisfying Equalized Odds also satisfies Statistical Parity and Predictive Parity.
Based on the theorem, in this paper, we propose to evaluate models under a distribution where the demographic identity information is not predictive of labels to unify the three widely-used criteria. Specifically, we define that a non-discrimination model should meet that,
$$
\operatorname * {P r} (\widehat {Y} | Y, Z) = \operatorname * {P r} (\widehat {Y} | Y),
$$
when tested in a distribution where  $\operatorname{Pr}(Y|Z) = \operatorname{Pr}(Y)$ .
3.3 Assessing the Discrimination
Identity Phrase Templates Test Sets (IPTTS) are widely used as non-discrimination testing sets to assess the models' discrimination (Dixon et al., 2018; Park et al., 2018; Sun et al., 2019; Kritchenko and Mohammad, 2018). These testing sets are generated by several templates with slots for each of the identity-terms. Identity-terms implying different demographic groups are slotted into the templates, e.g., "I am a boy." and "I am a girl.", and it's easy to find that IPTTS satisfies  $\operatorname{Pr}(Y|Z) = \operatorname{Pr}(Y)$ . A non-discrimination model is expected to perform similarly in sentences generated by the same template but with different identity-terms.
For metrics, False Positive Equality Difference (FPED) and False Negative Equality Difference (FNED) are used (Dixon et al., 2018; Park et al., 2018), as defined below.
$$
\mathrm {F P E D} = \sum_ {z} \left| \mathrm {F P R} _ {z} - \mathrm {F P R} _ {\text {o v e r a l l}} \right|,
$$
$$
\mathrm {F N E D} = \sum_ {z} \left| \mathrm {F N R} _ {z} - \mathrm {F N R} _ {\text {o v e r a l l}} \right|,
$$
in which,  $\mathrm{FPR}_{\text{overall}}$  and  $\mathrm{FNR}_{\text{overall}}$ , standing for False Positive Rate and False Negative Rate respectively, are calculated in the whole IPTTS. Correspondingly,  $\mathrm{FPR}_z$  and  $\mathrm{FNR}_z$  are calculated on each subset of the data containing each specific
identity-term. These two metrics can be seen as a relaxation of Equalized Odds mentioned in Section 3.2 (Borkan et al., 2019).
It should also be emphasized that FPED and FNED do not evaluate the accuracy of models at all, and models can get lower FPED and FNED by making trivial predictions. For example, when tested in a distribution where  $\operatorname{Pr}(Y|Z) = \operatorname{Pr}(Y)$ , if a model makes the same predictions for all inputs, FPED and FNED will be 0, while the model is completely useless.
3.4 Correcting the Discrimination
Data manipulation has been applied to correct the discrimination in the datasets (Sun et al., 2019). Previous works try to supplement or augment the datasets to an identity-balanced one, which, in our perspective, is primarily trying to recover the non-discrimination distribution  $\widehat{\mathcal{D}}$ .
For data supplementation, Dixon et al. (2018) adds some additional non-toxic samples containing those identity-terms which appear disproportionately across labels in the original biased dataset. Although the method is reasonable, due to high cost, it is not always practical to add additional labeled data with specific identity-terms, as careful selection of the additional sentences w.r.t. the identity-terms, the labels, and even the lengths of sentences is required (Dixon et al., 2018).
The gender-swapping augmentation is a more common operation to mitigate the unintended bias (Zhao et al., 2018; Sun et al., 2019). For text classification tasks, Park et al. (2018) augment the datasets by swapping the gender-implying identity-terms (e.g., "he" to "she", "actor" to "actress") in the sentences of the training data to remove the correlation between  $Z$  and  $Y$ . However, it is worth mentioning that the gender-swapping operation additionally assumes that the non-discrimination distribution  $\hat{\mathcal{D}}$  meets the followings,
$$
Q (X ^ {\neg} | Z) = Q (X ^ {\neg}),
$$
$$
Q (Y | X ^ {\neg}, Z) = Q (Y | X ^ {\neg}),
$$
in which  $X^{\neg}$  refers to the content of sentences except for the identity information. And we argue that these assumptions may not hold sometimes. For example, the first assumption may result in some meaningless sentences (e.g., "He gives birth.") (Sun et al., 2019). Besides, this method is not practical for situations with many demographic groups.
4137
4 Our Instance Weighting Method
In this section, we introduce the proposed method for mitigating discrimination in text classifications. We first make a few assumptions about how the discrimination distribution  $\mathcal{D}$  in the datasets are generated from the non-discrimination distribution  $\hat{\mathcal{D}}$ . Then we demonstrate that we can obtain the unbiased loss on  $\hat{\mathcal{D}}$  only with the samples from  $\mathcal{D}$ , which makes models able to fit the non-discrimination distribution  $\hat{\mathcal{D}}$  without extra resources or annotations.
4.1 Assumptions about the Generation Process
Considering the perspective that the discrimination distribution is generated from the nondiscrimination distribution  $\widehat{\mathcal{D}}$ , we refer  $S \in [0,1]$  as the selection indicator variable, which indicates whether a sample is selected into the biased dataset or not. Specifically, we assume that every sample  $(x,z,y,s)^3$  is drawn independently from  $\widehat{\mathcal{D}}$  following the rule that, if  $s = 1$  then the sample is selected into the dataset, otherwise it is discarded, then we have
Assumption 1.  $P(\cdot) = Q(\cdot |S = 1)$
and as defined in Section 3.1, the non-discrimination distribution  $\widehat{\mathcal{D}}$  satisfies
Assumption 2.  $Q(Y|Z) = Q(Y)$ .
Ideally, if the values of  $S$  are entirely at random, then the generated dataset can correctly reflect the original non-discrimination distribution  $\widehat{\mathcal{D}}$  and does not have discrimination. However, due to social prejudices, the value of  $S$  is not random. Inspired by the fact that some identity-terms are more associated with some specific labels than other identity-terms (e.g., sentences containing "gay" are more likely to be abusive in the dataset as mentioned before), we assume that  $S$  is controlled by  $Y$  and  $Z^4$ . We also assume that, given any  $Z$  and  $Y$ , the conditional probability of  $S = 1$  is greater than 0, defined as,
Assumption 3.  $Q(S = 1|X,Y,Z) = Q(S = 1|Y,Z) > 0$ .
Meanwhile, we assume that the social prejudices will not change the marginal probability distribution of  $Z$ , defined as,
Assumption 4.  $P(Z) = Q(Z)$ ,
which also means that  $S$  is independent with  $Z$  in  $\widehat{\mathcal{D}}$ , i.e.,  $Q(S|Z) = Q(S)$ .
Among them, Assumption 1 and 2 come from our problem framing. Assumption 3 helps simplify the problem. Assumption 4 helps establish the non-discrimination distribution  $\hat{\mathcal{D}}$ . Theoretically, when  $Z$  is contained in  $X$ , which is a common case, consistent learners should be asymptotically immune to this assumption (Fan et al., 2005). A more thorough discussion about Assumption 4 can be found in Appendix B.
4.2 Making Models Fit the
Non-discrimination Distribution  $\widehat{\mathcal{D}}$
Unbiased Expectation of Loss Based on the assumptions above, we prove that we can obtain the loss unbiased to the non-discrimination distribution  $\widehat{\mathcal{D}}$  from the discrimination distribution with calculated instance weights.
Theorem 2 (Unbiased Loss Expectation). For any classifier  $f = f(x,z)$ , and for any loss function  $\Delta = \Delta(f(x,z),y)$ , if we use  $w = \frac{Q(y)}{P(y|z)}$  as the instance weights, then
$$
E _ {x, y, z \sim \mathcal {D}} \left[ w \Delta (f (x, z), y) \right] = E _ {x, y, z \sim \hat {\mathcal {D}}} \left[ \Delta (f (x, z), y) \right].
$$
Then we present the proof for Theorem 2.
Proof. We first present an equation with the weight  $w$ , in which we use numbers to denote the assumptions used in each step and bayes for the Bayes' Theorem.
$$
\begin{array}{l} w = \frac {Q (y)}{P (y | z)} \\ = \frac {Q (y)}{Q (y | z , S = 1)} \quad \begin{array}{c} \swarrow \\ \searrow \\ O (y) \end{array} \quad \begin{array}{c} \swarrow \\ \searrow \\ b a y e s \end{array} \\ = \frac {Q (y)}{Q (S = 1 | z , y) Q (y | z) / Q (S = 1 | z)} \quad \left. \right\} 2. 4 \\ \begin{array}{l} = \frac {Q (S = 1)}{Q (S = 1 | z , y)} \\ - \quad Q (S = 1) \end{array} \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \\ \begin{array}{l l} = \frac {Q (S = 1)}{Q (x , z , y | S = 1) Q (S = 1) / Q (x , z , y)} & \downarrow^ {3}, \text {d a y e s} \\ - \frac {Q (x , z , y)}{} & \downarrow^ {1} \end{array} \\ \end{array}
$$
$$
\begin{array}{l} E _ {x, z, y \sim \mathcal {D}} \left[ w \Delta (f (x, z), y) \right] \\ = \int \frac {Q (x , z , y)}{P (x , z , y)} \Delta (f (x, z), y) d P (x, z, y) \\ = \int \Delta (f (x, z), y) d Q (x, z, y) \\ = E _ {x, y, z \sim \widehat {\mathcal {D}}} \left[ \Delta (f (x, z), y) \right]. \\ \end{array}
$$
<sup>3</sup>Definitions of  $x, z$  and  $y$  are in Section 3.1.
As we only focus on the discrimination problem in this work, we ignore selection bias on other variables like topic and domain.
Algorithm 1: Non-discrimination Learning
Input: The dataset  $\{x,z,y\}$ , the number of fold  $K$  for cross prediction and the prior probability  $Q(Y = 0)$  and  $Q(Y = 1)$
01 Train classifiers and use  $K$ -fold cross-predictions to estimating  $P(y|z)$  with the dataset
02 Calculate the weights  $w = \frac{Q(y)}{P(y|z)}$  for all samples
03 Train and validate models using  $w$  as the instance weights