SlowGuess/NAACL_2021_OCR huggingface dataset:
Liwen Wang $^{1*}$ , Yuanmeng Yan $^{1*}$ , Keqing He $^{1,2}$ , Yanan Wu $^{1}$ , Weiran Xu $^{1}$ $^{1}$ Beijing University of Posts and Telecommunications, Beijing, China  
 $^{2}$ Meituan Group, Beijing, China
{w_liwen, yanyuanmeng, yan.wu, xuweiran}@bupt.edu.cn {hekeqing} @meituan.com
Representation learning is widely used in NLP for a vast range of tasks. However, representations derived from text corpora often reflect social biases. This phenomenon is pervasive and consistent across different neural models, causing serious concern. Previous methods mostly rely on a pre-specified, user-provided direction or suffer from unstable training. In this paper, we propose an adversarial disentangled debiasing model to dynamically decouple social bias attributes from the intermediate representations trained on the main task. We aim to denoise bias information while training on the downstream task, rather than completely remove social bias and pursue static unbiased representations. Experiments show the effectiveness of our method, both on the effect of debiasing and the main task performance.
Supervised neural networks have achieved remarkable success in a wide range of natural language processing (NLP) tasks. The fundamental capability of these neural models is to learn effective feature representations (Bengio et al., 2013) for the downstream prediction task. Unfortunately, the learned representations frequently contain undesirable biases with respect to things that we would rather not use for decision making. We refer to such inappropriate factors as protected attributes (Elazar and Goldberg, 2018a). Biased information has serious real-world consequences. For example, concerns have been raised about automatic resume filtering systems giving preference to male applicants when the only distinguishing factor is the applicants' gender (Sun et al., 2019). In this paper, we focus on social bias, such as gender bias which is the preference or prejudice towards one gender over the other (Moss-Racusin et al., 2012), race bias and age bias.
Figure 1: A demonstration of dynamically disentangling gender bias from the representations of downstream representations. The left figure shows a biased profession classifier where the model prefers men as engineers and women as nurses. Our adversarial disentanglement model makes the example features approach the boundary of the gender classifier. Therefore, the boundary of the profession classifier in the right figure shifts closer to the horizontal state where gender information is decoupled from the representations in the main task.
From the perspective of the debiasing target, previous debiasing works can be approximately classified into two types, word embedding (Bolukbasi et al., 2016; Caliskan et al., 2017; Zhao et al., 2018; Manzini et al., 2019; Wang et al., 2020; Kumar et al., 2020) and sentence embedding (Xu et al., 2017; Elazar and Goldberg, 2018a; Zhang et al., 2018; Ravfogel et al., 2020). The former aims to reduce the gender bias in word embedding, either as a post-processing step (Bolukbasi et al., 2016) or as part of the training procedure (Zhao et al., 2018). The latter focuses on removing these protected attributes from the downstream intermediate representations (Elazar and Goldberg, 2018a; Ravfogel et al., 2020). In this paper, we consider the latter setting and focus on how to mitigate undesirable social bias from the encoded representations without hurting the performance of the main task.
In terms of debiasing methods, previous models are either based on projection on a pre-specified, user-provided direction (Bolukbasi et al., 2016) or null-space (Xu et al., 2017; Ravfogel et al., 2020),
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3740-3750 June 6-11, 2021. ©2021 Association for Computational Linguistics
or on adding an additional gender discriminator (Xie et al., 2017; Elazar and Goldberg, 2018a). The former first trains an intermediate feature extractor on the main task, then using a separate projection method to remove social bias from the representations, finally fine-tuning on the main task. The debiasing procedure can be regarded as static because of no direct interaction between the main task and the debiasing task. Therefore, these methods have no guarantee that the representations for predicting the main task do not contain any bias information. Existing work, (Gonen and Goldberg, 2019), has shown that these methods only cover up the bias and that in fact, the information is deeply ingrained in the representations. Compared to these static debiasing methods, gender discriminator based methods (Elazar and Goldberg, 2018a; Zhang et al., 2018) use the traditional generative adversarial network (GAN) (Goodfellow et al., 2014) to distinguish protected gender attributes from encoded representations. However, they are notoriously hard to train (Ganin and Lempitsky, 2015). Elazar and Goldberg (2018a) has shown that the complete removal of the protected information is nontrivial: even when the attribute seems protected, different classifiers of the same architecture can often still succeed in extracting it. Hence, we aim to dynamically disentangle the social bias from the encoded representations while jointly training on the main task in a more stable way, rather than directly remove protected attributes. In fact, we show that bias information always remains even after adversarial debiasing and can be reconstructed from the encoded representations. The main goal of debiasing is to prevent downstream models from utilizing these social bias in the representations, that is, dynamic disentanglement instead of complete removal, as Fig 1 displays.
In this paper, we propose an adversarial disentangled debiasing model to dynamically decouple social bias attributes from the intermediate representations trained on the main task. Our motivation is to denoise bias information while training on the downstream task, rather than completely remove social bias and pursue static unbiased representations. Previous works (Elazar and Goldberg, 2018a; Gonen and Goldberg, 2019) show that even debiasing models achieve high fairness (Hardt et al., 2016), a fair amount of protected information still remains and can be extracted from the encoded representations. We argue that one can hardly re
Figure 2: The overall architecture of our proposed approach.
move all gender or race directions in the latent space but only preserve bias-free prediction on the downstream task. Specifically, we use a protected attribute classifier to generate model-agnostic adversarial worst-case perturbations to the representations in the direction that significantly increases the classifier's loss. Then we apply the perturbations to train the model of the downstream task end-to-end. The main difference between our method and GAN-based counterparts is that GANs suffer from unstable training for the two-stage min-max procedure but our method directly computes gradient-based perturbations to disentangle bias information from the representations. We hope to provide new insights and directions towards solving social bias issues. $^{1}$
Our main goal is to disentangle protected attributes from the representations of downstream tasks so that biased information can not affect the decision of the model on the main task. In other words, we aim to achieve fairness by equalizing the opportunity (Hardt et al., 2016) between individuals with different protected attributes (e.g. gender/race). Given a set of input samples  $x_{i}$ , and corresponding discrete attributes  $Z, z_{i} \in \{1, \dots, k\}$  (e.g. gender or race)  $^2$ , we aim to learn unbiased representations
1Our source code is available at https://github. com/W-lw/debias_adv.
2Although we focus on the discrete protected attributes in this paper, our method can be also applied to continuous attributes.
$h_i \in \mathbb{R}^d$ , so that  $z_i$  can pose as minimal negative effect as possible on the main task performance.
2.2 Overall Architecture
Fig 2 shows the overall architecture of our proposed method, including four core steps: protected forward, debiasing backward, main task forward, and update parameters. (1) protected forward: We first pre-train a protected attribute classifier then compute the classification cross-entropy loss  $\mathcal{L}_{\text{protected}}$  for each input sample  $x$ . (2) debiasing backward: We maximize the loss  $\mathcal{L}_{\text{protected}}$  of the protected attribute classifier to obtain the adversarial decoupling perturbation  $\delta$ . (3) main task forward: Then we sum the original input  $x$  and perturbation  $\delta$  to get a new adversarial sample  $x_{adv}$ . We forward the sample  $x_{adv}$  to the main task classifier to compute the loss  $\mathcal{L}_{\text{main}}$  of the downstream task. (4) update parameters: Finally, the overall model is updated by the sum of two losses  $\mathcal{L}_{\text{protected}}, \mathcal{L}_{\text{main}}$ . We will dive into the details of each procedure in the following section.
2.3 Adversarial Semantic Disentanglement
Protected Forward In Fig 2, we adopt BiLSTM as the shared context encoder by the main task classifier and protected attribute classifier. We first feed each token to an embedding layer to get token embedding  $e$ , then a BiLSTM encoder is adopted to get the context-aware representation  $h_i$  for each token  $x_i$ . Then, we use an attentive pooling layer to calculate the sentence embedding  $h$ . After that, a fully-connected layer followed by a softmax output layer is used to predict the protected attribute  $\hat{y}_i$ . Finally, we can get the classification cross-entropy loss  $\mathcal{L}_{\text{protected}}$ . In the experiment, we observe that pre-training the protected attribute classifier can effectively accelerate the whole training progress of debiasing. We also demonstrate that jointly training the protected attribute classifier and the main task classifier achieves superior performance in Section 4.2.
Debiasing Backward This is the primary step of our adversarial semantic disentanglement. Our main idea is to perform adversarial attacks (Goodfellow et al., 2015; Kurakin et al., 2016; Miyato et al., 2016; Jia and Liang, 2017; Zhang et al., 2019; Ren et al., 2019) to dynamically decouple social bias attributes from the intermediate representations trained on the main task. Specifically, we
need to compute a worst-case perturbation  $\delta$  that maximizes the original classification cross-entropy loss  $\mathcal{L}_{\text{protected}}$  of the protected attribute classifier:
$$
\boldsymbol {\delta} = \underset {\| \boldsymbol {\delta} ^ {\prime} \| \leq \epsilon} {\arg \max } \quad \mathcal {L} _ {\text {p r o t e c t e d}} \left(\boldsymbol {\theta}, x + \boldsymbol {\delta} ^ {\prime}\right) \tag {1}
$$
where  $\theta$  represents the parameters of the protected attribute classifier and  $x$  denotes a given sample.  $\epsilon$  is the norm bound of the perturbation  $\delta$ . However, due to model complexity, accurate computation for  $\delta$  is costly and inefficient. Similar to Vedula et al. (2020) and Ru et al. (2020), we apply Fast Gradient Value (FGV) (Rozsa et al., 2016) to approximate a worst-case perturbation  $\delta$ :
$$
\boldsymbol {\delta} = \epsilon \frac {g}{| | g | |}; \text {w h e r e} g = \nabla_ {\boldsymbol {e}} \mathcal {L} (f (\boldsymbol {e}; \boldsymbol {\theta}), Y) \tag {2}
$$
where  $f$  represents the protected attribute classifier. We perform normalization to  $g$  and then use a small  $\epsilon$  to ensure the approximate is reasonable. Section 4.3 validates a proper value of  $\epsilon$  can balance the debiasing effect and the main task performance. Finally, we can obtain the pseudo adversarial sample  $x_{adv} = x + \delta$ . Intuitively we aim to obtain a debiased representation  $x_{adv}$  by confusing the protected attribute classifier. Thus, the main task classifier can make a fair decision conditioned on the disentangled representation.
Main Task Forward After obtaining the pseudo adversarial sample  $x_{adv}$ , we forward the sample  $x_{adv}$  to the main task classifier to compute the loss  $\mathcal{L}_{main}$  of the downstream task, similar to protected forward. We find the location of adding adversarial perturbation plays a role in debiasing performance in Section 4.4. In a nutshell, adding noise to the word embedding layer achieves the best debiasing performance.
Update Parameters Finally, we apply the two classification objectives to update the parameters of the model as the dashed lines in Fig 2 show. Note that the loss  $\mathcal{L}_{\text{protected}}$  of the protected attribute classifier only updates the MLP and softmax layers while the loss  $\mathcal{L}_{\text{main}}$  of the main task classifier updates all the model parameters, including the low-level encoding layers. The setting aims to avoid the negative effect of the protected attribute classifier on main task performance.
Datasets Following the setup of (Ravfogel et al., 2020), we test the performance of our debiasing
3If the protected attribute is continuous, we can apply the regression objectives.
Ratio	Sentiment (Main Task)	TPR-GAP (Debias)
Original	INLP	Random Noise	Ours	Original	INLP	Random Noise	Ours
0.5	0.76	0.75	0.75	0.75	0.14	0.12	0.14	0.09
0.6	0.75	0.71	0.73	0.72	0.23	0.18	0.17	0.11
0.7	0.74	0.65	0.72	0.72	0.31	0.16	0.26	0.10
0.8	0.71	0.62	0.71	0.73	0.40	0.16	0.37	0.09
method on the dialectal tweets (DIAL) corpus collected by Blodgett et al. (2016) in a controlled setup, and the biography corpus (De-Arteaga et al., 2019) in a wild setup. The dialectal tweets corpus consists of 59.2 million tweets, where each tweet contains "race" information, and emojis correspond with specific emotion groups. According to the label of race and sentiment, we split the data into four classes: African American English (AAE) speaker with "happy" sentiment, Standard American English (SAE) speaker with "happy" sentiment, AAE speaker with "sad" sentiment and SAE speaker with "sad" sentiment. Following (Elazar and Goldberg, 2018b), we filter the corpus and 176K tweets left (44k for each class). Then we divide them into 40k samples for training, 2k for developing, and 2k for testing, following (Ravfogel et al., 2020). In the controlled setup, we introduce a bias ratio relevant to the sentiment and race to control the imbalance proportion of samples in four groups, following (Ravfogel et al., 2020). e.g., in the 0.8 condition, the AAE class contains  $80\%$  happy /  $20\%$  sad samples, while the SAE class contains  $80\%$  sad /  $20\%$  happy samples. And in the 0.5 conditions, all four categories contain the same number of samples. In all experiments, the unbalance factor of the development set and test set is set to 0.5.
The biography corpus contains 393,423 biographies, the corresponding professions (28 classes) labels and gender (protected attributes) labels. We split the dataset into 255,710, 39,369, 98,344 samples for training, validation and testing, as consistent with (De-Arteaga et al., 2019; Ravfogel et al., 2020).
Baselines We compare our model with these baselines as follow:
- Original is the main task classifier without any debiasing procedure as a baseline.
INLP (Ravfogel et al., 2020) is a linear debi-asaing method, which removes the protected
information from neural representations by iterative training the linear classifiers which predict the protected attributes.
- Random Noise replaces the debiasing perturbation generated by the protected classifier with random noise.
Implementation Details To demonstrate the effectiveness of our method, we use the same model structure of the main task (sentiment classification) as (Ravfogel et al., 2020), where the DeepMoji encoder (Felbo et al., 2017) and an one-hidden-layer MLP constitute the classifier. Besides, for simplicity, we use the same structure of classifier for predicting protected attributes. Both the unbalanced training data and the pre-trained DeepMoji model which has been proven that encodes demographic information would lead the downstream MLP classifier to make biased predictions. We then perform debiasing training for the main-task model following the process described in section 2.3 on the imbalanced training set with the imbalance factor and test the debiased model on the balanced test set.
Besides, we follow (Ravfogel et al., 2020) to evaluate our debiasing method on the biography corpus as a wild setup to verify the validity of our method in a less artificial setting. In this wild setup, we construct a similar model structure to the DeepMoji encoder, with a two-layer bidirectional RNN as the encoder, except for the attention operation. There are two input representation types of the encoder: FastText and BERT (Devlin et al., 2019). In the FastText experiments, we directly use the trained word embedding that provided by (Ravfogel et al., 2020), to represent each biography as a sequence of vectors. And in the BERT experiments, we use BERT as a sequence-to-sequence encoder
4Note that the original results reported in the published version contain some mistakes. We rerun the updated evaluation scripts according to the official code and report all the results for a fair comparison.
to obtain the representation of each word in the sentence. Then we feed the sentence representations into the model and perform the debiasing training process.
For all the experiments, we train and test our model on single 2080Ti GPU, and we use AllenNLP framework (Gardner et al., 2017) to implement our model. The hidden size of the 1-hidden-layer MLP classifier used in all of the above experiments is set to 300. In a controlled experiment, our debiasing method takes an average of ten minutes to run, and the total parameters of our models are 23M, including a DeepMoji encoder, a main task classifier, and a protected classifier. In the wild experiment, the model size of the FastText experiment is 127M, which takes an average of 15 minutes to run. While the model size of the BERT experiment is 114M, and it takes an average of 55 minutes to run, due to the use of BERT to encode the sentences. It's worth mentioning that our method converges with only one or two epochs, which is faster than other debiasing methods. In practice, we empirically find that the debiasing performance can reach the best when the L2-Norm of perturbation is between 1/3 and 2/3 of the corresponding disturbed vectors' L2-Norm. For example, in the first experiment, the L2-Norm size of the embedding vector is around 4, then we could set the normalized scale to (1.2, 1.8).
Metrics To evaluate the bias in the model, following (Ravfogel et al., 2020; De-Arteaga et al., 2019), we calculate TPR-GAP to measure the difference (GAP) in the True Positive Rate (TPR) between the groups with different protected attributes which can reflect the unfairness existing in NLP models:
$$
T P R _ {p, y} = P [ \hat {Y} = y | P = p, Y = y ] \qquad (3)
$$
$$
G A P _ {p, y} ^ {T P R} = T P R _ {p, y} - T P R _ {p ^ {\prime}, y} \tag {4}
$$
where  $y$  is the main task label of the input representation  $X$ , and  $p, p'$  denote the protected attribute  $P$ 's two values. Then we use TPR-GAP to measure the degree of bias, which calculate the root-mean square of  $GAP_{p,y}^{TPR}$  over all main task label  $y$ :
$$
G A P ^ {T P R, R M S} = \sqrt {\frac {1}{| N |} \sum_ {y \in N} \left(G A P _ {p , y} ^ {T P R}\right) ^ {2}} \tag {5}
$$
where  $N$  is the label set of all main task (sentiment or profession). De-Arteaga et al. (2019) did the experiment on the biography corpus, and proved
FastText	BERT
Accuracy (profession)	Original	78.1	80.9
INLP	73.0	75.2
Ours	80.1	77.8
TPR-GAP	Original	0.184	0.184
INLP	0.089	0.095
Ours	0.082	0.092
the indicator  $GAP_{p,y}^{TPR}$  have a strong correlation with the percentage of a certain gender group in different profession  $y$ , therefore  $GAP^{TPR,RMS}$  can reflect an overview of bias across all different main attributes. We use  $GAP^{TPR,RMS}$  to measure the bias existing in the models.
Table 1 displays the experimental results on the DIAL dataset under different ratios of data imbalance proportion which can reflect the degree of dataset bias. We analyze the results from two perspectives, TPR-GAP (Debias) and Sentiment (Main Task). For TPR-GAP (Debias), our method consistently outperforms other baselines under all ratios, especially on the more biased dataset. It demonstrates the effectiveness of our proposed adversarial semantic disentanglement. We also observe Random Noise can hardly mitigate social bias which confirms the necessity of the protected attribute classifier. For the performance of the main sentiment classification task, our method reaches close to the original baseline while INLP suffers from a severe drop under a large ratio. The results prove that our method can better avoid the negative effect of the debiasing procedure on main task performance. To further evaluate the debiasing effect, we also show the results of the wild biography classification dataset in Table 2. Results show that our method both achieves superior performance than other baselines on Accuracy of the main task and TPR-GAP of debiasing. Compared to the significant improvements on the DIAL dataset, we hypothesize that the bias degree of the dataset makes a difference to the range of improvements.
4.1 Fixed Encoder vs. Non-fixed Encoder
In previous works, it is common to pre-train the sentence encoder in advance and keep the encoder fixed while applying the debias algorithm. However, it is unclear whether this conventional experiment setup is applicable to our approach. Since
Figure 3: Performance comparison between fixed and non-fixed encoders. The above figure shows the bias decrement under different perturbation intensities while the below figure shows the classification performance of the main task. The experiments are adopted in the DIAL dataset, with the bias ratio set to 0.6.
our approach dynamically generates perturbation to decouple social bias from context via adversarial attacks, we expect the non-fixed encoder to generate perturbation of higher quality. To check this, we conduct two groups of experiments in the DIAL dataset, where one group uses a fixed encoder while the other group keeps the contextual encoder trainable. Note that we set the bias ratio to 0.6 in both two groups of experiments.
Fig 3 shows the experimental results. In Fig 3 above, we observe that our approach with the non-fixed encoder consistently achieves better debias effectiveness compared to the fixed encoder counterpart with a large margin. When the perturbation intensity increases, both experimental settings achieve an increasingly better debias effect.
On the other hand, as shown in Fig 3 below, the fixed encoder approach suffers a severe performance drop in classification accuracy with increasing perturbation intensity. Meanwhile, the classification accuracy under the non-fixed encoder setting is still increasing, and even outperforms the fixed encoder one when a relatively large perturbation intensity is applied. We argue that, with a non-fixed encoder, our approach can learn a high-quality perturbation for representation debias, and meanwhile continuously optimize for the main task.
Figure 4: Performance comparison between the static protected classifier and the training on-the-fly protected classifier. The above figure shows the bias decrement under different perturbation intensities while the below figure shows the classification performance of the main task. The experiments are adopted in the DIAL dataset, with the bias ratio set to 0.8.
4.2 Protected Classifier: "Static" vs. Training on-the-fly