Debiasing Pretrained Text Encoders by Paying Attention to Paying Attention

1 INTRODUCTION . Natural Language Processing ( NLP ) is increasingly penetrating real-world operations such as recruitment ( Hansen et al. , 2015 ) , legal systems ( Dale , 2019 ) , healthcare ( Velupillai et al. , 2018 ) and Web Search ( Nalisnick et al. , 2016 ) . Part of this success is attributed to the underlying embedding layer which encodes sophisticated semantic representations of language ( Camacho-Collados & Pilehvar , 2018 ) . The wide adoption of modern NLP models in critical domains has also inflicted a more thorough scrutiny . Recent research has uncovered some propensities of NLP models to replicate discriminatory social biases ( Bolukbasi et al. , 2016 ; Caliskan et al. , 2017 ; May et al. , 2019 ) which may cause unintended and undesired model behaviors with respect to social groups . Social bias in NLP is mainly caused by unbalanced mentions of attributes near advantaged groups in training data ( Zhao et al. , 2018a ) . For example , in most existing text corpora , very few cooks are referred to by male pronouns ( e.g . he , him , himself ) ( Zhao et al. , 2017 ) . Accordingly , text encoders or language models trained on such data may use this shortcut to inadvertently disassociate cooks from men , and learn that cooking is a female attribute . Worse , NLP models may even amplify social biases if left unchecked ( Bommasani et al. , 2021 ) . Methods to debias static word embeddings such as Word2vec ( Mikolov et al. , 2013 ) or GloVe ( Pennington et al. , 2014 ) have been applied for gender , race and religion ( Bolukbasi et al. , 2016 ; Zhao et al. , 2018b ; Kaneko & Bollegala , 2019 ; Ravfogel et al. , 2020 ) . However , by the time NLP practitioners started casting more attention to the fairness problem of their models , they had already switched to the more powerful sentence-level transformers in the likes of BERT ( Devlin et al. , 2018 ) , GPT3 ( Brown et al. , 2020 ) or T5 ( Raffel et al. , 2020 ) which owe their success to the novel self-attention mechanism ( Vaswani et al. , 2017 ) . This leap in accuracy in several NLP tasks does not extend to fairness since research discovered social stereotypes in modern text encoders ( May et al. , 2019 ; Nadeem et al. , 2020 ; Nangia et al. ) . To date , debiasing them remains comparatively under-explored . Mitigating biases in text encoders is difficult for four reasons : ( 1 ) They are expensive to retrain , so conventional methods based on Counterfactual Data Augmentation ( CDA ) to rebalance groupattribute mentions ( Zhao et al. , 2018a ; Webster et al. , 2020 ) become prohibitive as they generate more training data , and all debiasing attempts might be limited to either finetuning or adapting ( Houlsby et al. , 2019 ; Lauscher et al. , 2021 ) . ( 2 ) Static embeddings encode words whereas text encoders need context1 . Thus , it is not straightforward to use existing debiasing techniques for static embeddings off-the-shelf as it is not clear how to generate context for single words . Previous work tackled this problem by either designing bleached sentence templates ( May et al. , 2019 ; Kurita et al. , 2019 ) where they fill in the blanks with words of interest , or sampling sentences from large corpora where the words are mentioned ( Liang et al. , 2020a ; Cheng et al. , 2020 ) , thus creating context . The former betrays the expressiveness of natural language while the latter suffers from sampling and preprocessing bias ( Liang et al. , 2020a ) . ( 3 ) The input space of text encoders is the set of all possible sentences , so we can not debias every single input as it is done with static embeddings . ( 4 ) Text encoders are larger in capacity and complexity . This suggests that they can accommodate subtler and more sophisticated forms of stereotype , especially in their attention component , which renders bias imperceptible to existing detection methods as they are not designed to operate on attention . Despite these difficulties , previous works ( Liang et al. , 2020a ; Webster et al. , 2020 ; Liang et al. , 2020b ; Cheng et al. , 2020 ; Kaneko & Bollegala , 2021 ; Lauscher et al. , 2021 ) addressed the problem of reducing bias from modern text encoders with different techniques . However , most of them make strong assumptions about the linearity of bias . Moreover , they operate on the embeddings produced by text encoders , and leave their most important block - attention - largely unrectified . In this paper we explore attention-based debiasing . This approach stems from our observation that attention exhibits a great deal of social biases . We empirically show that this is the case , propose a novel method to reduce stereotypes from attention blocks , and demonstrate that it is effective in mitigating biases from sentence representations as a whole . Given an input sentence , our method compels the text encoder of interest to redistribute its internal attention scores such that each word in the input allocates the same attention for different social groups . Thus , it learns to forget previously encoded preferences , and generate fair representations , free of stereotypical influence . We also keep semantic information loss at a minimum while debiasing by distilling knowledge ( Hinton et al. , 2015 ; Gou et al. , 2021 ) from an unaltered teacher text encoder . In this setting , we encourage the debiased model to copy the original attention from its teacher to minimize semantic offset . Unlike most previous work which focus only on gender , we address five bias types in our experiments ( gender , race , religion , age and sexual orientation ) . We conduct likelihood- and inference-based evaluations to measure the intensity of bias in our final debiased models . Experiments demonstrate that the technique we propose effectively reduces bias , and outperforms existing debiasing methods . 2 RELATED WORK . In this section , we discuss related work about debiasing static word embeddings and sentence-level text encoders . Then , we shed some light on work done on the attention mechanism in general . It should be noted that bias at data level ( Pryzant et al. , 2020 ; Cryan et al. , 2020 ) and in language generation tasks ( Sheng et al. , 2020 ; Sap et al. , 2020 ; Dhamala et al. , 2021 ) are also active and complementary areas of research . However , due to space limitations , they will not be discussed in this paper . 2.1 BIAS IN STATIC WORD EMBEDDINGS . The work of Bolukbasi et al . ( 2016 ) pioneered bias research in NLP by discovering that static word embeddings such as Word2Vec ( Mikolov et al. , 2013 ) or GloVe ( Pennington et al. , 2014 ) encode significant amounts of binary gender bias . They proposed Hard-Debias : a simple method to remove biases by projecting gender-neutral word embeddings onto a gender-free direction . Manzini et al . ( 2019 ) extended Hard-Debias to the multiclass setting where they also treat racial and religious stereotypes . In both works , the bias direction is defined by a manually pre-compiled list of stereotyped words . In contrast , Ravfogel et al . ( 2020 ) suggest a data-driven approach to learn bias directions with a linear classifier . Debiasing is then conducted by iteratively projecting word embeddings on the null space of the classifier ’ s matrix . On the other hand , finetuning is the debiasing approach that attracted the widest adoption , either by using an autoencoder ( Kaneko & Bollegala , 2019 ) , attraction-repulsion mechanism ( Kumar et al. , 2020 ) , or adversarial attacks ( Xie et al. , 2017 ; 1A word needs to be in a context ( sentence or paragraph ) in order to be correctly encoded Li et al. , 2018 ; Elazar & Goldberg , 2018 ) . Unlike these post-processing methods , Zhao et al . ( 2018b ) added a new fairness constraint to GloVe loss function , and retrained their fair word embeddings from scratch . 2.2 BIAS IN TEXT ENCODERS . Research on biases in sentence representations is dominated by detection rather than correction and mitigation . To date , there are three main approaches to detect stereotypes in text encoders : ( 1 ) representation-based : where vector relationships between different types of inputs are measured . For example , May et al . ( 2019 ) extended the WEAT test ( Caliskan et al. , 2017 ) into sentence vector space ( SEAT ) , and compared the cosine similarity between representations of two sets of targets and two sets of attributes . All sentences in SEAT follow a predefined template . ( 2 ) likelihood-based : These approaches examine how often text encoders prefer stereotypes over anti-stereotypes . Preferences in this case are defined in terms of higher likelihoods as produced by language models using embeddings of the text encoders under study . Two benchmarks are widely used for measuring bias : StereoSet ( Nadeem et al. , 2020 ) and Crows-Pairs ( Nangia et al. ) . Both datasets are organized in pairs or triples of minimally-distant sentences which differ only in the word ( s ) carrying a stereotypical connotation . ( 3 ) inference-based : These methods employ text encoders in downstream NLP tasks ( Blodgett et al. , 2020 ) such as natural language inference ( Dev et al. , 2020 ) , sentiment analysis ( Dı́az et al. , 2018 ) or language generation ( Sap et al. , 2020 ; Sheng et al. , 2020 ) . Bias in such settings is declared as the difference in outcome when the models are tested with the same input sentence , differing only in social groups . Bias mitigation approaches are mostly inspired by debiasing static embeddings . In projection-based methods , Liang et al . ( 2020a ) contextualize words into sentences by sampling them from existing corpora before applying Hard-Debias . Kaneko & Bollegala ( 2021 ) minimize the projection of sentence representations on a learned bias subspace , while Qian et al . ( 2019 ) ; Bordia & Bowman ( 2019 ) ; Liang et al . ( 2020b ) add bias-reduction objectives to their loss functions . Another line of research uses CDA ( Webster et al. , 2020 ) to balance gender correlations in training data , while Lauscher et al . ( 2021 ) use adapters to reduce the large training time that CDA incurs . Finally , Cheng et al . ( 2020 ) use contrastive learning , and add a fair filter that minimizes mutual information between stereotypes and anti-stereotypes . In our work , rather than extending approaches from static embeddings , we focus on the self-attention mechanism which is characteristic of many text encoders , and show that fair attention leads to fair representations . 2.3 ATTENTION ANALYSIS IN TEXT ENCODERS . Clark et al . ( 2019 ) analyzed BERT ’ s attention heads and found that some of them correspond remarkably well to linguistic patterns of coreference and syntax without additional training . Michel et al . ( 2019 ) observe that not all attention heads within a model are made equal . They also propose a pruning algorithm to reduce the energy footprint of these models by eliminating the least important heads without much attenuation to the overall performance . Given the convenient interpretabilty of attention , it has also been used in a myriad of visualization works ( Vig , 2019 ; Hoover et al. , 2020 ; Tenney et al. , 2020 ; Bastings & Filippova , 2020 ) in an attempt to dissect and explain the inner functioning of text encoders . Most attention studies in text encoders are designed for analysis purposes . In contrast , we are the first to leverage the attention mechanism in order to make text encoders fairer and less stereotyped .

Summary: This paper addresses potential biases introduced by attention models by re-weighing the attention weights. First, the paper provides a few examples that demonstrate attention weights correlated with social stereotype (e.g., doctor attending to he and nurse attending to she). Then, it proposes to reduce this type of bias by "calibrating" the attention weights. To do so, each sample is augmented with pairs of words corresponding to the groups for which the model is intended to mitigate bias. The augmented samples are used in training, and, thus, each token in the input will attend to tokens in the augmented portion of the sample. The weights are changed such that the weights attending on the augmented, group-relevant tokens are similar. This type of re-weighing can lead to some of the semantic meaning encoded in the network to change. To prevent this change, the attention weights corresponding to the tokens from the original sample are forced to follow the weights in an unaltered model that is trained in parallel. Last, the paper suggested also utilizing negative sampling: using random words for augmentation and forcing all attention weights to follow the ones of the teacher.  For evaluation, the paper shows the performance of several transformer-based models.  The evaluation of the method includes three different benchmarks: two benchmarks that measure bias intrinsically (Crows-Pairs and StereoSet) and an NLI task designed for measuring bias. To make sure that the semantic strengths of the model are not lost, the models are also evaluated using the GLUE tasks. The model obtains competitive performance for the GLUE tasks, while reducing bias compared to original models and a couple of related works.