SlowGuess/NAACL_2021_OCR huggingface dataset:
As discussed in the previous section, our proposed adversarial disentangled debiasing method requires the protected classifier to learn an accurate decision boundary of the protected attributes, such that the debiasing perturbation approximates the direction that mostly eliminates the model's discrimination of the protected attributes. Naturally, we have two options: either fix the parameters of protected classifier to generate the relatively static debiasing perturbation, or train the protected classifier on-the-fly during the main classifier training process to offer a relatively dynamic perturbation.
To verify which one performs better, we adopt two groups of experiments. In the "static" setting, we keep the parameters of the protected classifier fixed. Whether the parameters of the encoder are fixed or not, the debiasing perturbation generated by the protected classifier would be relatively static. It's worth noting that if the parameters of the encoder are fixed, the debiasing perturbation would be totally static. While in the training on-the-fly setting, we reserve the gradient of the protected classifier and update its parameters together with the main task model (context encoder and main task classifier). According to the conclusions in
Figure 5: The debias effectiveness (above) as well as the classification accuracy on the main task (below) of our proposed approach in the DIAL dataset, with the perturbation intensity increases from 0.1 to 7.0. We set the bias ratio to 0.8 and all parameters trainable.
section 4.1, we make the context encoder trainable in both settings and use the same objective to train the main classifier.
The results are displayed in Fig 4. We can find that both settings have the ability to debiasing in the DIAL dataset, showing the effectiveness of our approach in both settings. However, the training on-the-fly strategy consistently outperforms the "static" strategy under various perturbation intensities. We hypothesize that the difference is mainly because under the training on-the-fly strategy, the protected classifier will have a chance to adjust the decision boundary when the context encoder updates, and thus continuously generates better dynamic debiasing perturbation via adversarial attacks.
4.3 Influence of Perturbation Intensity
To explore how the perturbation intensity influences the debias effectiveness and the performance of main task, we run multiple experiments with only changing the perturbation intensity. We experiment with a wide range of perturbation intensity, from 0.1 to 7.0.
The experimental results are illustrated in Fig 5. From the figure above, we find that the bias decrement rapidly increases at the beginning period with the intensity increasing from 0.1 to 0.7. Then, between a wide range from 0.7 to 6.6, the bias decrement keeps relatively stable, oscillate in a
DIAL bias ratio →	0.5	0.6	0.7	0.8
Accuracy	Original	0.75	0.75	0.74	0.71
To sent emb	0.75	0.72	0.72	0.72
To word emb	0.73	0.72	0.72	0.73
TPR-GAP	Original	0.14	0.23	0.31	0.40
To sent emb	0.09	0.14	0.19	0.21
To word emb	0.09	0.11	0.10	0.09
Table 3: Analysis on which representation space is best for debiasing. "To sent emb" indicates the perturbation is added to the sentence embedding space, while "To word emb" indicates the perturbation is added to the word embedding space. The perturbation intensity is set to 0.7.
small range of 0.275 - 0.325, reflecting the stability of our approach. However, when the perturbation intensity exceeds some threshold (6.6 in this case), the bias decrement drops again. Meanwhile, with the perturbation intensity increasing, the classification accuracy of main task keeps falling (figure below), indicating that the perturbation with high intensity will also disturb the main task, leading to a low classification accuracy. The result provides a principle of how to choose a suitable perturbation intensity - the minimal intensity while effective enough for debiasing.
4.4 Which Representation Space to Apply Debiasing
Another pivotal consideration for our dynamically disentangling approach is - which representation space should we add the perturbation to? Typically, we have two choices: a) adding the perturbation to the sentence embedding space or b) adding the perturbation to the word embedding space. The sentence embedding is closer to the output space with the key information condensed into a single vector, while the word embedding is closer to the input side, keeping separated for each token. To check out which one performs better for social debiasing, we conduct experiments in the DIAL dataset with different bias ratio.
Table 3 illustrates the experiment results. Compared the result of "To sent emb" to "To word emb", we found adding the perturbation to word embedding space often gains better debiasing results, especially when the bias ratio of the dataset is large. For example, when the bias ratio is 0.8, adding to word embedding space achieves a  $GAP^{TPR,RMS}$  of 0.09, while adding to sentence embedding space achieves 0.21. We believe that, when applying our debiasing approach to a deeper representation
DIAL bias ratio →	0.5	0.6	0.7	0.8
Accuracy	Original	0.75	0.75	0.74	0.71
Entropy	0.74	0.71	0.70	0.72
Cross entropy	0.75	0.72	0.72	0.73
TPR-GAP	Original	0.14	0.23	0.31	0.40
Entropy	0.13	0.15	0.17	0.17
Cross entropy	0.09	0.11	0.10	0.09
space, the perturbation is also context-aware (since the context encoder is also related when calculating the gradient) and thus more dynamic for the complex data distribution.
4.5 Cross-Entropy vs. Entropy
As mentioned in Section 2.3, we need to calculate a cross-entropy loss  $\mathcal{L}_{\text{protected}}$  to generate the debiasing perturbation via FGV. Thus, during the training of the main task, we must obtain the protected attribute for each training example to calculate the cross-entropy loss. This severely limits the usefulness of our approach, as it may be difficult to obtain the ground truth protected attribute when training the main task. To this end, we also propose to use the entropy loss (Zheng et al., 2020) to substitute the cross-entropy loss:
$$
\mathcal {L} _ {\text {p r o t e c t e d}} = - \mathcal {H} (P (\boldsymbol {y} _ {\text {p r o t e c t e d}} | x)) \tag {6}
$$
where  $\mathcal{H}$  indicates the Shannon entropy and  $P(\mathbf{y}_{\text{protected}} | x)$  is the distribution output by protected classifier. This objective forces the protected classifier to obtain high entropy, which means the classifier is not confident and almost distributed uniformly across all values of the protected attributes.
In Table 4, we compare the debiasing effectiveness of using entropy with cross-entropy. From the table, we observe that using the entropy objective also works for debiasing as the TPR-GAP also drops compared with the baseline. However, the debiasing effect still can't exceed our approach with cross-entropy. This seems reasonable since the cross-entropy objective introduces extra information about the protected attribute. With the extra supervision signal, our approach generates pertur
Table 4: Experimental results on accuracy and debiasing effect with different objectives of the protected classifier. We respectively apply the Cross-Entropy loss and the Entropy loss to the protected classifier when calculating the objective of the protected classifier for generating the perturbation for debiasing. Note that the protected classifier is pre-trained and fixed, and the entropy loss doesn't require ground truth protected attributes during the training of the main task.
DIAL bias ratio →	0.5	0.6	0.7	0.8
RIM	INLP	0.143	0.164	0.362	0.473
Ours	0.357	0.482	0.650	0.814
Table 5: The debiasing effect under our proposed Relative Improve Metric (RIM). We show that our approach is far beyond the INLP under the evaluation of RIM.
(b) Ours
Figure 6: t-SNE projection of BiLSTM sentence representations for the positive sentiment. The left represents the baseline and the right represents our method. We display the representation distribution of different races in the latent space. Different races are colored in the figure.
bation towards a more precise direction for eliminating the representation of the protected attributes.
4.6 Performance on Different Bias Ratio
To more clearly show the performance differences of our model over data sets with varying degrees of bias, we introduce a new metric named Relative Improve Metric (RIM):
$$
R I M = \frac {A c c ^ {\prime} - A c c}{A c c} + \frac {G A P - G A P ^ {\prime}}{G A P} \tag {7}
$$
where  $Acc$  and  $Acc'$  represent the main task accuracy of the model before and after debiasing respectively, and  $GAP$ ,  $GAP'$  represent the TPR-GAP indicator of the model before and after debiasing respectively. RIM could synthetically reflect the stability of the main task and the debiasing performance of a debiasing method. We calculate the RIM indicator of our model and INLP based on the results in Table 1, and the new results are shown in Table 5. We can observe that the stronger bias in the dataset, the better performance of the two methods. Besides, we can find that our debiasing method is more robust.
To better understand the effectiveness of our method, we display a feature visualization of sentence representations in Fig 6. We can observe that the different race classes are no longer linearly
separable after debiasing. Therefore, downstream tasks can not make decisions conditioned on the race information in the representations.
The objective of controlled removal of specific types of information from neural representations is tightly related to the task of disentanglement of the representations (Bengio et al., 2013), that is, controlling and separating the different kinds of information encoded in them. Previous models are either based on projection on a pre-specified, user-provided direction (Bolukbasi et al., 2016) or null-space (Xu et al., 2017; Ravfogel et al., 2020), or adding an additional gender discriminator (Xie et al., 2017; Elazar and Goldberg, 2018a), or the impact of data decisions (Beutel et al., 2017). The former first train an intermediate feature extractor on the main task, then use a separate projection method to remove social bias from the representations, finally finetune on the main task. Compared to these static debiasing methods, gender discriminator based methods (Elazar and Goldberg, 2018a; Zhang et al., 2018) use the traditional generative adversarial network (GAN) (Goodfellow et al., 2014) to remove protected gender attributes from encoded representations. However, they are notoriously hard to train (Ganin and Lempitsky, 2015). Elazar and Goldberg (2018a) has shown that the complete removal of the protected information is nontrivial: even when the attribute seems protected, different classifiers of the same architecture can often still succeed in extracting it. Therefore, in this paper, we aim to dynamically disentangle the social bias from the encoded representations while jointly training on the main task in a more stable way, rather than directly remove protected attributes. The main goal of debiasing is to prevent downstream models from utilizing these social biases in the representations, that is, dynamic disentanglement instead of complete removal.
In this paper, we focus on removing social bias in representation learning. We argue that the main goal of debiasing is to prevent downstream models from utilizing these social biases in the representation, that is, dynamic disentanglement instead of complete removal. Therefore, we propose an adversarial disentangled debiasing model to dynamically decouple social bias attributes from the
intermediate representation trained on the main task. We perform extensive experiments and analysis to demonstrate the effectiveness of our method. We hope to provide new insights and directions towards solving social bias.
In recent years, neural network based models have demonstrated remarkable performance in many natural language processing tasks and thus have been applied to a wide range of real-world applications. However, a lot of works reveal that such models are easily affected by social bias and thus makes incorrect and biased decisions. In domains with the greatest potential for societal impacts, using such biased models for real-world applications is dangerous and faces many problems such as human morality. The social bias implicit in the natural language processing model may be exposed and become a social hot spot, thus becoming an unstable factor that causes social unrest. Meanwhile, some existing debiasing methods, although able to slightly reduce bias in such model, often cause great damage to model performance in the main task, thus difficult to be applied in practice. This work proposes a new adversarial training method for end-to-end debiasing. Due to the robustness of the adversarial attack, the model can eliminates bias without losing much performance.
Yoshua Bengio, Aaron C. Courville, and P. Vincent. 2013. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35:1798-1828.
Alex Beutel, Jilin Chen, Zhe Zhao, and Ed H. Chi. 2017. Data decisions and theoretical implications when adversarially learning fair representations.
Su Lin Blodgett, Lisa Green, and Brendan O'Connor. 2016. Demographic dialectal variation in social media: A case study of African-American English. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages
1119-1130, Austin, Texas. Association for Computational Linguistics.
Tolga Bolukbasi, Kai-Wei Chang, James Y. Zou, Venkatesh Saligrama, and A. Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. ArXiv, abs/1607.06520.
A. Caliskan, J. Bryson, and A. Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356:183 - 186.
Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnamaram Kenthapadi, and Adam Tauman Kalai. 2019. Bias in bios. Proceedings of the Conference on Fairness, Accountability, and Transparency - FAT* '19.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Yanai Elazar and Y. Goldberg. 2018a. Adversarial removal of demographic attributes from text data. ArXiv, abs/1808.06640.
Yanai Elazar and Yoav Goldberg. 2018b. Adversarial removal of demographic attributes from text data. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 11-21, Brussels, Belgium. Association for Computational Linguistics.
Bjarke Felbo, Alan Mislove, Anders Søgaard, Iyad Rahwan, and Sune Lehmann. 2017. Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1615-1625, Copenhagen, Denmark. Association for Computational Linguistics.
Yaroslav Ganin and V. Lempitsky. 2015. Unsupervised domain adaptation by backpropagation. In ICML.
Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke S. Zettlemoyer. 2017. Allennlp: A deep semantic natural language processing platform.
H. Gonen and Y. Goldberg. 2019. Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them. In NAACL-HLT.
Ian J. Goodfellow, Jean Pouget-Abadie, M. Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In NIPS.
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and harnessing adversarial examples. CoRR, abs/1412.6572.
M. Hardt, E. Price, and Nathan Srebro. 2016. Equality of opportunity in supervised learning. In NIPS.
Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. arXiv preprint arXiv:1707.07328.
V. Kumar, Tenzin Singhay Bhotia, and Tanmoy Chakraborty. 2020. Nurse is closer to woman than surgeon? mitigating gender-biased proximities in word embeddings. Transactions of the Association for Computational Linguistics, 8:486-503.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. 2016. Adversarial examples in the physical world. arXiv preprint arXiv:1607.02533.
Thomas Manzini, Yao Chong Lim, Yulia Tsvetkov, and A. Black. 2019. Black is to criminal as caucasian is to police: Detecting and removing multiclass bias in word embeddings. In NAACL-HLT.
Takeru Miyato, Andrew M Dai, and Ian Goodfellow. 2016. Adversarial training methods for semi-supervised text classification. arXiv preprint arXiv:1605.07725.
C. Moss-Racusin, J. F. Dovidio, V. Brescoll, M. Graham, and J. Handelsman. 2012. Science faculty's subtle gender biases favor male students. Proceedings of the National Academy of Sciences, 109:16474 - 16479.
Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. 2020. Null it out: Guarding protected attributes by iterative nullspace projection. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7237-7256, Online. Association for Computational Linguistics.
Shuhuai Ren, Yihe Deng, Kun He, and Wanxiang Che. 2019. Generating natural language adversarial examples through probability weighted word saliency. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1085-1097.
Andras Rozsa, Ethan M Rudd, and Terrance E Boult. 2016. Adversarial diversity and hard positive generation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 25-32.
Dongyu Ru, Yating Luo, Lin Qiu, Hao Zhou, Lei Li, Weinan Zhang, and Yong Yu. 2020. Active sentence learning by adversarial uncertainty sampling in discrete space. arXiv preprint arXiv:2004.08046.
T. Sun, Andrew Gaut, Shirlyn Tang, Yuxin Huang, M. ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth M. Belding-Royer, Kai-Wei Chang, and William Yang Wang. 2019. Mitigating gender bias in natural language processing: Literature review. ArXiv, abs/1906.08976.
Nikhita Vedula, Nedim Lipka, Pranav Maneriker, and Srinivasan Parthasarathy. 2020. Open intent extraction from natural language interactions. In Proceedings of The Web Conference 2020, pages 2009-2020.
Tianlu Wang, Xi Victoria Lin, Nazneen Rajani, Vicente Ordonez, and Caimng Xiong. 2020. Double-hard debias: Tailoring word embeddings for gender bias mitigation. ArXiv, abs/2005.00965.
Qizhe Xie, Zihang Dai, Yulun Du, E. Hovy, and Graham Neubig. 2017. Controllable invariance through adversarial feature learning. In NIPS.
K. Xu, Tongyi Cao, S. Shah, Crystal Maung, and H. Schweitzer. 2017. Cleaning the null space: A privacy mechanism for predictors. In AAAI.
B. H. Zhang, B. Lemoine, and Margaret Mitchell. 2018. Mitigating unwanted biases with adversarial learning. Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society.
Huangzhao Zhang, Hao Zhou, Ning Miao, and Lei Li. 2019. Generating fluent adversarial examples for natural languages. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5564-5569.
Jieyu Zhao, Yichao Zhou, Z. Li, W. Wang, and Kai-Wei Chang. 2018. Learning gender-neutral word embeddings. In EMNLP.
Yinhe Zheng, Guanyi Chen, and Minlie Huang. 2020. Out-of-domain detection for natural language understanding in dialog systems. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:1198-1209.