Year: 2021

Abstract:
Recent studies in fair Representation Learning have observed a strong inclination for Natural Language Processing (NLP) models to exhibit discriminatory stereotypes across gender, religion, race and many such social constructs. In comparison to the progress made in reducing bias from static word embeddings, fairness in sentence-level text encoders received little consideration despite their wider applicability in contemporary NLP tasks. In this paper, we propose a debiasing method for pre-trained text encoders that both reduces social stereotypes, and inflicts next to no semantic damage. Unlike previous studies that directly manipulate the embeddings, we suggest to dive deeper into the operation of these encoders, and pay more attention to the way they pay attention to different social groups. We find that most stereotypes are also encoded in the attention layer. Then, we work on model debiasing by redistributing the attention scores of a text encoder such that it forgets any preference to historically advantaged groups, and attends to all social classes with the same intensity. Our experiments confirm that we successfully reduce bias with little damage to semantic representation.

heading: 1 INTRODUCTION
text: Natural Language Processing (NLP) is increasingly penetrating real-world operations such as recruitment (Hansen et al., 2015), legal systems (Dale, 2019), healthcare (Velupillai et al., 2018) and Web Search (Nalisnick et al., 2016). Part of this success is attributed to the underlying embedding layer which encodes sophisticated semantic representations of language (Camacho-Collados & Pilehvar, 2018). The wide adoption of modern NLP models in critical domains has also inflicted a more thorough scrutiny. Recent research has uncovered some propensities of NLP models to replicate discriminatory social biases (Bolukbasi et al., 2016; Caliskan et al., 2017; May et al., 2019) which may cause unintended and undesired model behaviors with respect to social groups. Social bias in NLP is mainly caused by unbalanced mentions of attributes near advantaged groups in training data (Zhao et al., 2018a). For example, in most existing text corpora, very few cooks are referred to by male pronouns (e.g. he, him, himself) (Zhao et al., 2017). Accordingly, text encoders or language models trained on such data may use this shortcut to inadvertently disassociate cooks from men, and learn that cooking is a female attribute. Worse, NLP models may even amplify social biases if left unchecked (Bommasani et al., 2021).
Methods to debias static word embeddings such as Word2vec (Mikolov et al., 2013) or GloVe (Pennington et al., 2014) have been applied for gender, race and religion (Bolukbasi et al., 2016; Zhao et al., 2018b; Kaneko & Bollegala, 2019; Ravfogel et al., 2020). However, by the time NLP practitioners started casting more attention to the fairness problem of their models, they had already switched to the more powerful sentence-level transformers in the likes of BERT (Devlin et al., 2018), GPT3 (Brown et al., 2020) or T5 (Raffel et al., 2020) which owe their success to the novel self-attention mechanism (Vaswani et al., 2017). This leap in accuracy in several NLP tasks does not extend to fairness since research discovered social stereotypes in modern text encoders (May et al., 2019; Nadeem et al., 2020; Nangia et al.). To date, debiasing them remains comparatively under-explored.
Mitigating biases in text encoders is difficult for four reasons: (1) They are expensive to retrain, so conventional methods based on Counterfactual Data Augmentation (CDA) to rebalance groupattribute mentions (Zhao et al., 2018a; Webster et al., 2020) become prohibitive as they generate
more training data, and all debiasing attempts might be limited to either finetuning or adapting (Houlsby et al., 2019; Lauscher et al., 2021). (2) Static embeddings encode words whereas text encoders need context1. Thus, it is not straightforward to use existing debiasing techniques for static embeddings off-the-shelf as it is not clear how to generate context for single words. Previous work tackled this problem by either designing bleached sentence templates (May et al., 2019; Kurita et al., 2019) where they fill in the blanks with words of interest, or sampling sentences from large corpora where the words are mentioned (Liang et al., 2020a; Cheng et al., 2020), thus creating context. The former betrays the expressiveness of natural language while the latter suffers from sampling and preprocessing bias (Liang et al., 2020a). (3) The input space of text encoders is the set of all possible sentences, so we cannot debias every single input as it is done with static embeddings. (4) Text encoders are larger in capacity and complexity. This suggests that they can accommodate subtler and more sophisticated forms of stereotype, especially in their attention component, which renders bias imperceptible to existing detection methods as they are not designed to operate on attention.
Despite these difficulties, previous works (Liang et al., 2020a; Webster et al., 2020; Liang et al., 2020b; Cheng et al., 2020; Kaneko & Bollegala, 2021; Lauscher et al., 2021) addressed the problem of reducing bias from modern text encoders with different techniques. However, most of them make strong assumptions about the linearity of bias. Moreover, they operate on the embeddings produced by text encoders, and leave their most important block - attention - largely unrectified.
In this paper we explore attention-based debiasing. This approach stems from our observation that attention exhibits a great deal of social biases. We empirically show that this is the case, propose a novel method to reduce stereotypes from attention blocks, and demonstrate that it is effective in mitigating biases from sentence representations as a whole. Given an input sentence, our method compels the text encoder of interest to redistribute its internal attention scores such that each word in the input allocates the same attention for different social groups. Thus, it learns to forget previously encoded preferences, and generate fair representations, free of stereotypical influence. We also keep semantic information loss at a minimum while debiasing by distilling knowledge (Hinton et al., 2015; Gou et al., 2021) from an unaltered teacher text encoder. In this setting, we encourage the debiased model to copy the original attention from its teacher to minimize semantic offset. Unlike most previous work which focus only on gender, we address five bias types in our experiments (gender, race, religion, age and sexual orientation). We conduct likelihood- and inference-based evaluations to measure the intensity of bias in our final debiased models. Experiments demonstrate that the technique we propose effectively reduces bias, and outperforms existing debiasing methods.

heading: 2 RELATED WORK
text: In this section, we discuss related work about debiasing static word embeddings and sentence-level text encoders. Then, we shed some light on work done on the attention mechanism in general. It should be noted that bias at data level (Pryzant et al., 2020; Cryan et al., 2020) and in language generation tasks (Sheng et al., 2020; Sap et al., 2020; Dhamala et al., 2021) are also active and complementary areas of research. However, due to space limitations, they will not be discussed in this paper.

heading: 2.1 BIAS IN STATIC WORD EMBEDDINGS
text: The work of Bolukbasi et al. (2016) pioneered bias research in NLP by discovering that static word embeddings such as Word2Vec (Mikolov et al., 2013) or GloVe (Pennington et al., 2014) encode significant amounts of binary gender bias. They proposed Hard-Debias: a simple method to remove biases by projecting gender-neutral word embeddings onto a gender-free direction. Manzini et al. (2019) extended Hard-Debias to the multiclass setting where they also treat racial and religious stereotypes. In both works, the bias direction is defined by a manually pre-compiled list of stereotyped words. In contrast, Ravfogel et al. (2020) suggest a data-driven approach to learn bias directions with a linear classifier. Debiasing is then conducted by iteratively projecting word embeddings on the null space of the classifier’s matrix. On the other hand, finetuning is the debiasing approach that attracted the widest adoption, either by using an autoencoder (Kaneko & Bollegala, 2019), attraction-repulsion mechanism (Kumar et al., 2020), or adversarial attacks (Xie et al., 2017;
1A word needs to be in a context (sentence or paragraph) in order to be correctly encoded
Li et al., 2018; Elazar & Goldberg, 2018). Unlike these post-processing methods, Zhao et al. (2018b) added a new fairness constraint to GloVe loss function, and retrained their fair word embeddings from scratch.

heading: 2.2 BIAS IN TEXT ENCODERS
text: Research on biases in sentence representations is dominated by detection rather than correction and mitigation. To date, there are three main approaches to detect stereotypes in text encoders: (1) representation-based: where vector relationships between different types of inputs are measured. For example, May et al. (2019) extended the WEAT test (Caliskan et al., 2017) into sentence vector space (SEAT), and compared the cosine similarity between representations of two sets of targets and two sets of attributes. All sentences in SEAT follow a predefined template. (2) likelihood-based: These approaches examine how often text encoders prefer stereotypes over anti-stereotypes. Preferences in this case are defined in terms of higher likelihoods as produced by language models using embeddings of the text encoders under study. Two benchmarks are widely used for measuring bias: StereoSet (Nadeem et al., 2020) and Crows-Pairs (Nangia et al.). Both datasets are organized in pairs or triples of minimally-distant sentences which differ only in the word(s) carrying a stereotypical connotation. (3) inference-based: These methods employ text encoders in downstream NLP tasks (Blodgett et al., 2020) such as natural language inference (Dev et al., 2020), sentiment analysis (Dı́az et al., 2018) or language generation (Sap et al., 2020; Sheng et al., 2020). Bias in such settings is declared as the difference in outcome when the models are tested with the same input sentence, differing only in social groups.
Bias mitigation approaches are mostly inspired by debiasing static embeddings. In projection-based methods, Liang et al. (2020a) contextualize words into sentences by sampling them from existing corpora before applying Hard-Debias. Kaneko & Bollegala (2021) minimize the projection of sentence representations on a learned bias subspace, while Qian et al. (2019); Bordia & Bowman (2019); Liang et al. (2020b) add bias-reduction objectives to their loss functions. Another line of research uses CDA (Webster et al., 2020) to balance gender correlations in training data, while Lauscher et al. (2021) use adapters to reduce the large training time that CDA incurs. Finally, Cheng et al. (2020) use contrastive learning, and add a fair filter that minimizes mutual information between stereotypes and anti-stereotypes. In our work, rather than extending approaches from static embeddings, we focus on the self-attention mechanism which is characteristic of many text encoders, and show that fair attention leads to fair representations.

heading: 2.3 ATTENTION ANALYSIS IN TEXT ENCODERS
text: Clark et al. (2019) analyzed BERT’s attention heads and found that some of them correspond remarkably well to linguistic patterns of coreference and syntax without additional training. Michel et al. (2019) observe that not all attention heads within a model are made equal. They also propose a pruning algorithm to reduce the energy footprint of these models by eliminating the least important heads without much attenuation to the overall performance. Given the convenient interpretabilty of attention, it has also been used in a myriad of visualization works (Vig, 2019; Hoover et al., 2020; Tenney et al., 2020; Bastings & Filippova, 2020) in an attempt to dissect and explain the inner functioning of text encoders. Most attention studies in text encoders are designed for analysis purposes. In contrast, we are the first to leverage the attention mechanism in order to make text encoders fairer and less stereotyped.

heading: 3 DEBIASING METHOD

heading: 3.1 MOTIVATING EXAMPLE & INTUITION
text: Despite the applicability of our work on any model that is built upon self-attention, we focus in this paper on models based on the encoder side of the transformer architecture, such as BERT (Devlin et al., 2018), RoBERTa (Liu et al., 2019), or ALBERT (Lan et al., 2019). This owes to the decoder side being usually used in auto-regression tasks, and less often to encode text. Transformers consist of multiple layers, each composed of a self-attention block followed by a feed-forward block to make embeddings. A self-attention block contains multiple heads. Each head transforms the input into
attention weights between all pairs of tokens in the input sequence (Vaswani et al., 2017), such that each token learns to attend to its most related tokens, hence the prevalence of attention in defining the understanding of natural language. In this work, we assume that undesired social stereotypes are primarily encoded in the self-attention block. In order to verify this hypothesis, we show and analyze some attention maps of BERT in Figure 12. Consider the following sentence ”The doctor asked the nurse a question.” Aiming to analyze how every word representation3 relates to different social groups, we add a dummy second input consisting of words representing two distinct genders (he and she after the [SEP] token). Figure 1(a) illustrates that the token doctor pays much more attention to he than to she4. Figure 1(b) reveals that nurse attends to she. This finding suggests that gender stereotypes are deeply encoded in attention weights. Likewise, in Figure 1(c), the token math is more related to asian than to white or black, conforming to the famous racial stereotype casting asians as really good mathematicians. These examples align with our intuition that social stereotypes are first and foremost encoded in the self-attention block of text encoders before they propagate to their embeddings or predictions in downstream NLP tasks. Consequently, we propose Att-D, a finetuning method for reducing undesired biases of text encoders from their attention component.
The intuition of our debiasing strategy is as follows: Given that attention weights conform with undesired biases (e.g., doctor attending to he, and nurse to she in Figure 1), we aim to equalize the attention scores of every word in the input sentence with respect to social groups. Following the examples of Figure 1, Att-D redistributes the attention scores of doctor such that it attends to he and she with the same intensity, thus eliminating any preference toward one of the groups. However, alterations to the attention of doctor on the remaining words of the input sentence must be kept to a minimum in order not to corrupt the semantic understanding of the original text encoder.

heading: 3.2 DEBIASING WORKFLOW
text: Att-D consists of three steps: First, for each input sentence s in the training corpus S, we make an artificial second input sg consisting of words related to social groups (similar to the examples on Figure 1). Pretrained text encoders expect either one or two inputs before they produce attentions and embeddings. In this work, we use both s (as first input) and sg (as second input) separated by [SEP] token. Consequently, the resulting self-attention includes both s and sg . The second step of Att-D equalizes the attention weights of all heads in all the layers of the text encoder of interest such that each token in s pays the same amount of attention to tokens of sg , thus eliminating preferences and stereotypes. Finally, we minimize semantic loss by compelling our model to learn the original semantics from an unaltered teacher model by copying its internal attention in a knowledge distillation setting (Hinton et al., 2015).
We schematize the operation of Att-D in Figure 2. Gr1, Gr2 and Grn in the figure correspond to the tokens of sg . Both matrices represent one attention head of the text encoder before (left) and after (right) debiasing. The matrices should be read in rows. Each row depicts the attention weights of the corresponding token on all the other tokens of the input (s + sg)5.
2The figures are produced using bertviz tool: https://github.com/jessevig/bertviz 3According to BERT in this example 4Dark colors correspond to high attention scores, and light colors indicate low attention scores 5[CLS] token (vector representation of s) is also included for attention calibration. Details in Appendix
The matrices are conceptually split in four blocks: (1) attentions of s on s, (2) attentions of s on sg , (3) attentions of sg on s, and (4) attentions of sg on sg . Debiasing consists in making the columns of block 2 equal. In other words, each token in s pays the same amount of attention to all the groups as indicated in the right side of Figure 2. Ideally, debiasing should also preserve the semantics of the original text encoder. That is why block 1 of Figure 2 should be kept unchanged. Both blocks 3 and 4 are irrelevant to the results, since they denote attentions of our artificially inserted second input sg . Besides, they do not participate in defining neither fairness nor representativeness of the text
encoder. So, we do not impose any restrictions on them. In the following, we describe each step of Att-D in detail.

heading: 3.2.1 GENERATING AUGMENTED INPUTS
text: The first step involves identifying bias types that we want to mitigate from pretrained text encoders, such as gender, race, religion. This is achieved by defining a set of tuples G such that G = {T1, T2, ..., Tk} where each Ti describes social groups of a given bias type, or their attributes. In the case of binary gender, Ti = {”male”,”female”} or Ti = {”he”,”she”} are both possible definitions (and many others are possible for non-binary cases). Table 1 shows some
tuples that we use in Att-D. Unlike previous work (Liang et al., 2020a; Kaneko & Bollegala, 2021), we do not aim to learn bias subspaces. Therefore, Att-D charges next to no constraint in the definition of G, contrary to existing methods which need group definitions to be formulated in a certain way, and given in enough quantities. On the other hand, providing a single tuple per bias type is sufficient for Att-D to function, even though more contributes to better precision.
During debiasing, we pick a tuple Ti from G randomly and construct sg , a bleached sentence formed by words of Ti. For example, given Table 1, sg can be ”man, woman” or ”old, young”. The input to the text encoder is both the original input sentence s and the artificial one sg . Pretrained text encoders separate the two halves of the input with a special token [SEP] as illustrated in Figure 1, and compute attention maps for the entire sequence (s + sg).

heading: 3.2.2 EQUALIZING ATTENTIONS ON SOCIAL GROUPS
text: After obtaining attention maps of the augmented input from Section 3.2.1, which is produced by the pretrained text encoder of interest E, we make each token of s pay equal amounts of attention to the tokens of sg which define social groups. The rationale is to eliminate any inclination forE to prefer a social group to the detriment of others. Suppose Al,h,s,sg = Attn(s, sg; l, h) is the attention matrix at layer l, head h of the encoder E, computed from the input s + sg . Here, we make the reasonable assumption that sg contains at least two social groups.6 In this spirit, equalizing attention vectors of block 2 (as defined in Figure 2) is equivalent to making them equal to a pivot vector. In our method, we consider the attention vector of s on the first social group as the pivot (first column in block 2 of Figure 2), and minimize the mean square error between the pivot and the attention vectors of s on the other groups, one at a time. The equalization loss is given by Equation 1.
Lequ = ∑ s∈S L∑ l=1 H∑ h=1 ||sg||∑ i=2 ||Al,h,s,sg:σ,σ+1 −A l,h,s,sg :σ,σ+i || 2 2 (1)
6Biases are usually about making one or more groups (dis)advantaged with respect to the others, hence the existence of at least two groups per bias type
where L is the number of layers of the text encoder, H the number of heads, ||sg|| the number of social groups in sg and σ is the position of the special token [SEP] that marks the end of s and the beginning of sg . As can be seen, A l,h,s,sg :σ,σ+1 is the pivot vector containing attention scores of s on the first social group token (whose position is directly after [SEP], i.e., σ + 1). Equation 1 forces attention scores on subsequent social groups to be the same as on the first one, thus making them all equal.

heading: 3.2.3 PRESERVING SEMANTIC INFORMATION
text: Text encoders must preserve their ability to represent natural language and keep the same performance on downstream NLP tasks. For this reason, debiasing must ensure that useful information is preserved as much as possible. We minimize semantic information loss in a knowledge distillation setting (Hinton et al., 2015; Gou et al., 2021). We cast the text encoder that we want to debias as the student model, and recruit another model to play the role of the teacher. We do not apply our debiasing strategy on the teacher since it provides a reference to the original unaltered language representations. We distill semantic information in the form of attention maps from the teacher and instill it in the student. Stated differently, we compel the student to learn from the teacher and reproduce its attention scores for every input sentence in the training corpus S. In this work, the student and the teacher models must share the exact same architecture in order to make knowledge distillation flow correctly from source to target, unlike other works in knowledge distillation (Gou et al., 2021) wherein the student is much smaller and should be more compact than the teacher for model compression purposes.
As in Section 3.2.2, let Al,h,s,sg be the attention of the student model at layer l, head h with s and sg as input. Likewise, let Ol,h,s,sg define the same attention matrix, but for the original teacher model. We formalize the preservation of semantic information as a regularizer where we minimize the squared l2 distance between the student’s and the teacher’s attention scores according to the following:
Ldistil = ∑ s∈S L∑ l=1 H∑ h=1 ||Al,h,s,sg:σ,:σ −Ol,h,s,sg:σ,:σ ||22 (2)
where L is the number of layers, H is the number of heads, and σ is the position of the [SEP] token. As can be seen from Equation 2, the student learns only to replicate block 1 (as in Figure 2) of the attention matrices. This is because block 1 contains attention scores of the original input sentence s on itself, thus encoding an important aspect of semantics. We force the student not to reproduce the attention distribution on social groups (block 2) from the teacher since these are fundamentally biased, and are left to the care of our debiasing objective. We describe the overall training objective as a linear combination of the previously defined losses, with λ as a hyperparameter to control the weight of debiasing over semantic preservation.
Loss = Ldistil + λLequ (3)

heading: 3.3 NEGATIVE SAMPLING & LAYER SELECTION
text: The strict application of Att-D as discussed so far may accidentally lead to some undesired spurious phenomena. While learning to equalize attention on social groups that constitute the second half of the input, the text encoder might potentially bear the risk of distributing its attention uniformly on any second half of the input, no matter what it is. This is particularly alarming when the text encoder is subsequently employed in double-sentence tasks (Wang et al., 2018) such as semantic textual similarity, paraphrase detection or sentence entailment.
To overcome the above obstacle, we introduce negative sampling. Instead of using words related to social groups in order to generate the artificial second input sg , we randomly sample words (negative examples) from the vocabulary. In this case, we do not equalize the attentions but compel the student to copy its teacher even for blocks 2, 3 and 4. We do this in order to prevent the text encoder from learning to assign the same attention weight to all tokens of the second input when these do not define social groups. We control the ratio of negative examples with a hyperparameter η.
Another concern when debiasing text encoders is that their layers do not necessarily encode the same information. Bhardwaj et al. (2021) found that BERT layers display widely different reactions when probed with a gender-detection classifier. This means that they do not encode gender stereotype identically. Therefore, it is not clear which layers and/or attention heads are best for debiasing. To investigate this issue, we consider seven settings: debiasing all layers, first 6, first 3, last 6, last 3, and alternating layers with strides of one7 or two8. We find that working with all layers yields the best results. So, unless otherwise specified in this paper, debiasing concerns all the layers.

heading: 4 EVALUATION
text: In this section, we first describe our experimental setup, then evaluate Att-D from two viewpoints: fairness and representativeness. Unlike previous work in debiasing text encoders (Liang et al., 2020a; Cheng et al., 2020; Kaneko & Bollegala, 2021), we do not conduct representation-based evaluations - SEAT (May et al., 2019) and WEAT (Caliskan et al., 2017) - in this paper to measure bias since their effectiveness for such a purpose has been questioned (Silva et al., 2021). On the other hand, we evaluate fairness with both likelihood- and inference-based methods. Other quantitative and qualitative evaluations can be found in the appendix9.

heading: 4.1 DEBIASING SETUP
text: In the experimental evaluation we focus on five different bias types which are commonly addressed by the bias literature and debiasing approaches. To facilitate comparison, we also follow existing literature in defining the social groups for each type of bias, although the approach presented here is not restricted to that, and can be leveraged for both other kinds of biases and for a more inclusive definition of the groups. In the experiments we show results of debiasing based on (binary) gender (male, female), race (white, black, asian, hispanic), religion (muslim, christian, jewish, buddhist), age (young, old) and sexual orientation (heterosexual, non-heterosexual). We leverage the definition words from previous work (Liang et al., 2020a) and augment them with our own tuples. The full list can be found in the appendix. We apply our debiasing method on BERT10 (Devlin et al., 2018), and use the News-commentary-v15 corpus11 as training data. It contains 223,153 sentences of which we use 80% for training and 20% for development.

heading: 4.2 EVALUATIONS OF FAIRNESS

heading: 4.2.1 LIKELIHOOD-BASED EVALUATION
text: We use the publicly available subsets of two stereotype benchmarks: StereoSet (Nadeem et al., 2020) and Crows-Pairs (Nangia et al.). Both provide likelihood-based diagnostics to measure how often stereotypes are considered more likely than anti-stereotypes by the text encoder of interest12. CrowsPairs is organized in pairs of minimally distant sentences, i.e. sentences that differ only in one word, usually the one describing a social group. We pick the following example from the dataset: Jews are such greedy people (Stereotype) versus Christians are such greedy people (Anti-stereotype). CrowsPairs uses word likelihoods produced by the text encoder to measure how often stereotypes are ranked higher than anti-stereotypes. An ideal unbiased text encoder should score 50% in the CrowsPairs challenge, meaning that it prefers neither stereotypes nor anti-stereotypes. 100% and 0% are both undesired extreme outcomes. In contrast, StereoSet adds a third absurd sentence to capture the language modeling capabilities of the text encoder in addition to measuring bias. The triple of sentences in StereoSet are also minimally distant but differ in the attribute word, not in the group. For example, in Girls tend to be more [MASK] than boys, StereoSet generates the evaluation sentences by replacing the mask with the following attributes: soft to denote stereotype, determined for antistereotyping, and fish to make an unrelated sentence. As in Crows-Pairs, StereoSet quantifies how
7Layers 2, 4, 6, 8, 10 and 12 of BERT 8Layers 4, 8 and 12 of BERT 9We provide code and data as supplementary material to this submission
10In the appendix, we also apply Att-D on ALBERT, RoBERTa, DistilBERT and SqueezeBERT 11http://www.statmt.org/wmt20/translation-task.html 12The text encoder must first be fine-tuned on a language modeling objective
much the model prefers stereotypes over anti-stereotypes. Moreover, it also measures how often meaningful sentences rank higher than unrelated sentences in order to assess the quality of the text encoder in terms of representativeness. An ideal model should have a stereotype score (ss) of 50% and a language modeling (lm) of 100%.
We compare our method against the original BERT base model to see the effect of debiasing13. Also, for fair comparisons against previous work, we decided to include the baselines whose final debiased models have been published in order to avoid errors of training and/or tuning hyperparameters. Thus, we compare Att-D against Sent-D (Liang et al., 2020a) and the debiasing procedure proposed by Kaneko & Bollegala (2021). Sent-D currently holds state-of-the-art performance in debiasing. We also conduct a simple ablation study by training without negative examples (Att-D-). We finetune Att-D and the baselines on language modeling to be able to conduct likelihood-based experiments. Tables 2 and 3 report the evaluation results on StereoSet and Crows-Pairs respectively.
We see that the original BERT contains significant levels of biases (56.04 in StereoSet and 60.48 in Crows-Pairs). It is important to note that Kaneko & Bollegala (2021) only focused on gender bias, which is clear in Table 2 where only gender stereotype has been reduced. We observe that focusing on one bias type can make text representations even more biased for the other dimensions as can be seen in Kaneko for race and religion (Table 2), and for sexual orientation and disability (Table 3). In contrast, Att-D always reduces the intensity of stereotyping in BERT (up to 9.53%), and yields the best
results overall14. We also notice that we manage to reduce biases linked to dimensions we did not include in our design such as profession, nationality and disability. We speculate that these bias types are connected to those we worked on mitigating. Therefore, we conjecture that reducing multiple biases at the same time meets better success in mitigating unforeseen stereotypes than working on every bias type separately.

heading: 4.2.2 INFERENCE-BASED EVALUATION
text: This approach of measuring bias builds on the intuition of Dev et al. (2020) stating that biased representations lead to invalid inferences, whose ratio quantifies bias. They construct a challenge benchmark for the natural language inference task where every hypothesis should be neutral to its premise. For example, suppose that the premise is The driver owns a van and the hypothesis is The man owns a van. The hypothesis neither entails nor contradicts the premise. If the predictions of a classifier deviate from neutrality, the underlying text encoder is doomed as biased. All the data points in Dev et al. (2020) follow the same structure of the example above, and span across gender, religion and race. Suppose that the set contains M instances, and let the predictor’s probabilities of the ith instance for entail, contradict and neutral be ei, ci and ni. Following Dev et al. (2020), we report three measures of inference-based bias: (1) Net Neutral (NN): NN = 1M ∑M i=1 ni; (2) Fraction
Neutral (FN): FN = 1M ∑M i=1 1ni=max(ei,ci,ni); (3) Threshold τ (T:τ ): T : τ = 1 M ∑M i=1 1ni>τ .
13Results of BERT large, ALBERT, RoBERTa, DistilBERT and SqueezeBERT are in the appendix 14The closer the stereotype score is to 50%, the better
In this experiment, we finetune text encoders of interest with the MNLI dataset for natural language inference (Wang et al., 2018). An ideal bias-free model should score 1 in all three measures. We report our findings in Table 4 after transforming them into percentages. Our method outperforms the original model and the baselines. This result shows that Att-D not only reduces bias at surface-level as reflected by likelihood-inspired evaluation methods, but also mitigates stereotypes in real world inference settings, unlike Sent-D which produces positive results with Crows-Pairs but comes short of meeting the same success in this experiment. In the next section, we show that these findings are meaningful since the entailment accuracy is not hurt after debiasing.

heading: 4.3 EVALUATIONS OF REPRESENTATIVENESS
text: We use GLUE benchmark (Wang et al., 2018) to verify whether the debiased text encoder still holds enough semantic information to be applicable in downstream NLP tasks. In essence, GLUE assesses the natural language understanding capabilities of NLP models. So, it constitutes a suitable stack to evaluate the semantic preservation of Att-D. In this experiment, we finetune BERT on seven different tasks from GLUE and show the results in Table 5. We also report the difference in accuracy between original BERT and each of the debiasing baselines. Surprisingly, Att-D not only preserves semantic information, but enhances it in most GLUE tasks as reflected in an increase in accuracy from BERT.

heading: 5 CONCLUSION AND FUTURE WORK
text: In this paper, we proposed a finetuning approach to debiasing that trains the text encoder to distribute its attention equally on different social groups. Experiments demonstrate that bias is successfully reduced without harm to semantic representativeness. However, we are aware of the following limitations: (1) our definitions of biases are simplified. There are more social divisions in the real world than the five dimensions we studied. Besides, bias types can be correlated in intricate ways such as the links between race, nationality and ethnicity. Moreover, it is not clear which or how many groups to include. For these reasons, we follow previous work and constrain our study to simple definitions targeting the most widely spread groups. We plan to study the effect that the choice of definition tuples and their order impose overall. (2) We equalize the attention scores of every word in the input. However, some words are inherently charged with a strong inclination toward one group, e.g., beard to male or pregnant to female. Such words need not be debiased, which requires compiling expensive lists of related words for every social group and protecting them from attention equalization. In this work, we rely on knowledge distillation to retain as much useful semantic information as possible. (3) Current bias detection experiments have positive predictive ability, which means that they can only detect the presence of bias, not the absence of it. Although contemporary evaluation tools demonstrate that our method successfully reduced social biases from sentence representations, it is possible that bias is still lurking in shapes and forms that our experimental lenses failed to detect. We plan to address these limitations in future work.

heading: A APPENDIX

heading: A.1 TRAINING HYPERPARAMETERS
text: We used Adam optimizer (Kingma & Ba, 2014) with a learning rate of 5e−6 for 3 epochs. We keep the betas to their default values (0.9, 0.999) as in PyTorch implementation (Paszke et al., 2017). We set the loss coefficient λ to 2.0 and the negative ratio η to 0.8 meaning that in 80% of the iterations, we use negative examples whose number we set to 5 in each negative iteration. We only finetuned the values of λ, η, the learning rate, and the number of epochs. We conducted the hyperparameter search manually on the development set.
As for GLUE experiments, we follow the experimental setup of Devlin et al. (2018) and train each task for 3 epochs with a learning rate of 2e−5 on their respective training data.

heading: A.2 SOCIAL GROUPS USED IN THE EXPERIMENTS
text: While the approach is independent of the definition of social groups and categories (it could work for any kind of grouping, e.g., cuisine styles or sports), in the experiment we focus on groups commonly used in the debiasing literature: binary gender, religion, race, age and sexual orientation. This is to facilitate comparison, but nothing in the approach prevent it from being used with broader and more inclusive groups. This being said, we have not experimented yet with debiasing where a dimension is divided in dozens of categories.
We list the definition tuples that we used in Table 6. We show that Att-D does not incur strict rules for defining social groups, unlike previous work (Bolukbasi et al., 2016; Kaneko & Bollegala, 2019; 2021) that require the definition words to be organized in a predefined format (pairs of words or bag of words for every group), and provided in relatively large quantities. We can see from Table 6 that it is sufficient to define one tuple per bias type (e.g., race) if the tuples are hard to come by. Also, the tuples need not be of the same size (e.g., in religion there is a missing word for buddhist group since it is not clear which word to use in that tuple). This desired property owes to the fact that Att-D does not learn subspaces or directions for every bias type as previous works do (Bolukbasi et al., 2016; Kaneko & Bollegala, 2019; Kumar et al., 2020; Kaneko & Bollegala, 2021). In contrast, Att-D uses the tuples in order to equalize the attentions of the input sentence, and make the words therein attend to the groups with the same intensity. These example categories used in experiments are neither complete nor exhaustive, and in some experiments also include terms possibly considered inappropriate but that appear in the corpus and we may still want to debias from (such as using ”straight” to define heterosexual).

heading: A.3 CHOICE OF LAYERS
text: Transformer-based text encoders consist of many layers. It is not clear which layers to choose for debiasing since bias information is spread out across all of them. In this experiment, we try different
debiasing settings in which we select different layer combinations of BERT to work on: all layers, first 6, first 3, last 6, last 3, and alternating layers with strides of 1 (layers 2, 4, 6, 8, 10 and 12) or 2 (layers 4, 8 and 12). We apply the debiasing method proposed in this paper, and report both language modeling and stereotype scores of StereoSet benchmark in Table 7.
The results show that it is safest to equalize attention heads of all layers of the text encoder under study, since it produces the best scores both in terms of language modeling and stereotype. Our findings go in tandem with those of Liang et al. (2020b); Kaneko & Bollegala (2021); Bhardwaj et al. (2021) who found that reducing bias from all layers usually is the best option.

heading: A.4 EFFECT OF NEGATIVE EXAMPLES ON REPRESENTATIVENESS
text: We remind that the introduction of negative examples to training serves in forcing the text encoder not to rely on a dangerous shortcut which is distributing its attention uniformly on all the tokens constituting the second half of the input, no matter what the input is. This is particularly important in double-sentence tasks where the text encoder is given two input sentences. In addition to Tables 2 and 3 which highlighted the effect of negative sampling on the final stereotype scores, the primary goal of using negative examples remains the preservation of the text encoder’s representativeness. In Table 8, we report the performance of Att-D and Att-D- with and without negative examples respectively on GLUE tasks. Unsurprisingly, the lack of negative examples does not damage the performance of single-sentence tasks since these ignore the second half of the input altogether. However, in double-sentence tasks where both halves are used for prediction, Table 8 shows that negative sampling plays a pivotal role in preserving the semantics of text encoders, and bypassing the side effects inflicted by attention equalization.
A.5 VISUALIZING DEBIASING RESULTS
In this experiment, we aim to visualize the effects of debiasing on attention weights. We only focus on binary gender bias for two reasons: First, it is easier to visualize binary variables on a 2D plane than multiclass variables (such as race, religion...). Second, gender is the most well studied bias type (Bolukbasi et al., 2016; Caliskan et al., 2017; May et al., 2019), so linguistic resources and vocabularies for gender exist and are well documented. We use the vocabulary words compiled by Kaneko & Bollegala (2019) and categorized into three non-overlapping subsets: (1) Male-definition ΩM whose corresponding words are exclusively male-gendered such as father, king or uncle. (2) Female-definition ΩF which is a set of inherently female words (mother, queen, aunt...). (3) Gender-stereotype ΩS which is constituted of words that are not gendered by definition, but that carry a strong gender stereotype such as doctor being attributed to male or nurse to female.
For every word w ∈ ΩM ∪ΩF ∪ΩS , we extract sentences from the News-commentary-v15 corpus where w is mentioned. We denote this set as Sw. Then, for every sentence s ∈ Sw, we append the dummy input ”man, woman” as explained in Sections ?? and 3.2.1. The augmented input s′ is then fed to the text encoder of interest (BERT base in this experiment), and we collect the attention scores ofw on the second-half tokens man and woman. Finally, for every wordw ∈ ΩM∪ΩF∪ΩS , we take the mean of its attention scores in Sw. By the end of this procedure, we have for every word w its attention score on the words man (awm) and woman (awf ) as computed on the News-commentary-v15 corpus which includes overall 223,153 sentences. We take the difference awm − awf which indicates the preference of the text encoder to consider w as male (positive difference) or female (negative difference). The absence of gender bias is reflected in difference scores near zero.
We plot the results in Figure 3 where the x-axis represents the differences awm − awf , and the y-axis random values to separate the words vertically. Stereotype words (green dots) should have values near 0, which is not the case in Figure 3(b). This means that BERT has a strong preference for one of the genders, and is thus heavily biased. In contrast, our method brings the attention of stereotype words near 0, meaning that they prefer neither male nor female connotations. Moreover, the spread of stereotype words in Figure 3(d) is narrower than male- or female-oriented words, which is desired since these are inherently gendered and must pick a side. This result strengthens the claim that AttD preserves semantic information, and is less severe in reducing bias from gendered words as it is on gender-neutral words. The difference in spread is less apparent in the original BERT model. We also note that debiasing the embeddings of BERT rather than the attention mechanism as in Kaneko & Bollegala (2021) (Figure 3(c)) is not enough since bias information is still lurking (and perhaps made worse for some words) in the attention component. Thus, we conclude that working on attention directly constitutes our best option for debiasing to date.

heading: A.6 EFFECT OF ATTENTION-BASED DEBIASING ON OTHER TRANSFORMER-BASED TEXT ENCODERS
text: We evaluate five widely used sentence-level text encoders: BERT (Devlin et al., 2018), ALBERT (Lan et al., 2019), RoBERTa (Liu et al., 2019), DistilBERT (SANH et al.) and SqueezeBERT (Iandola et al., 2020). For each model, we evaluate both its base and large variants (except for DistlBERT and SqueezeBERT since these are not available in HuggingFace’s transformers library15), original and debiased; which gives a total of sixteen evaluated models. We use Crows-Pairs dataset (Nan-
15https://huggingface.co/transformers/index.html
gia et al.) to quantify the intensity of undesired stereotypes encoded therein. As a reminder, ideal stereotype scores according to Crows-Pairs benchmark should be close to 50, i.e. models preferring neither stereotypes nor anti-stereotypes. Tables 9, 10, 11 and 12 show the bias results for BERT, ALBERT, RoBERTa and DistillBERT/SqueezeBERT respectively.
All five models exhibit substantial levels of bias, and in each of the bias types with differing intensities (religion, sexual orientation and disability being the bias categories with the most severe stereotyping). Also, we find that the large variants are more biased than their base counterparts mainly because large models, with their larger capacity and greater number of parameters, can capture more intricate and more sophisticated aspects of training data, exposing them to learn more bias. This finding corresponds well to results of previous work (Nangia et al.; Nadeem et al., 2020). The tables also show that Att-D is effective in mitigating bias from BERT, ALBERT, RoBERTa, DistilBERT and SqueezeBERT, and produces a reduction of up to 25%. We note that Att-D succeeds in debiasing all models, with varying effectiveness across bias types. We also note that Att-D meets the best success with ALBERT as reductions are greater on this particular text encoder. We believe this is because ALBERT is composed of a single transformer layer (Lan et al., 2019) with substantially less parameters than BERT or RoBERTa; which makes debiasing easier since there is no interference between different attention layers. Finally, we see from the tables that Att-D sometimes contributes to adding a bit of bias. We observe that this phenomenon is rare, and happens especially with bias types we did not include in our design16. We assume that not explicitly compelling the text encoder to equalize attention heads corresponding to these overlooked bias types gave it green light to adjust these attentions in a way to facilitate solving the optimization problem; even if it entails adding bias. We plan to include all bias types present in Crows-Pairs dataset to our debiasing design as a future work.

heading: A.7 APPLICATION-ORIENTED BIAS EVALUATION
text: Recent studies show that intrinsic metrics of bias do not necessarily correlate with bias measures on concrete real-world applications (Goldfarb-Tarrant et al., 2020). In the body of this paper, we already conducted intrinsic and extrinsic bias evaluations. In this experiment, we validate the efficacy of our debiasing method on a concrete real-world hate speech detection application where an input snippet of text is classified as either offensive (toxic, harmful, disrespectful...) or not. We use hate speech detection because it is well studied in the literature (Burnap & Williams, 2016; Ribeiro et al., 2018; Zhang et al., 2018), and high-quality datasets which are tagged with social groups already exist (Borkan et al., 2019; Mathew et al., 2021).
16In the current version of this work, we remind that we only consider five bias types: gender, race, religion, age and sexual orientation
Admittedly, common social biases have also been shown to exist in hate speech detection models, for example in associating toxicity to frequently attacked groups (such as ”muslim” or ”gay”) even if the text itself is not toxic (Dixon et al., 2018; Park et al., 2018). In this experiment, we adopt the bias definition of Borkan et al. (2019) which casts bias as a skewing in the hate speech detector scores based solely on the social groups mentioned in the text. In other words, we consider a model to exhibit unintended social stereotypes if the model’s performance varies across groups. We use the bias measures proposed by Borkan et al. (2019) which are based on the Area Under the Receiver Operating Characteristic Curve (ROC-AUC, or AUC) metric. AUC measures the probability that a randomly chosen negative example (not offensive) receives a lower toxicity score than a randomly chosen positive example (offensive), meaning that a perfect model should always have an AUC score of 1.0. Stated differently, all negative examples have lower toxicity scores than positive examples. While AUC is used to measured the general performance of classifiers, Borkan et al. (2019) propose three extensions of AUC to measure bias. We summarize them in the following:
Subgroup (Sub) AUC: where AUC is computed only on the group under consideration and not on all the examples of the test benchmark, i.e. only positive and negative examples of the target group are considered. This metric represents the model’s performance on a given group. A higher value means that the model is good at distinguishing between toxic and non-toxic texts specific to the group.
Background Positive Subgroup Negative (BPSN) AUC: where AUC is calculated on the negative examples of the target group, and the positive examples of the background (all other groups except the group under consideration). This metric computes whether the model discriminates against the target group with respect to the others. This value is reduced when non-toxic examples of the group have higher toxicity scores than actually toxic examples of the background.
Background Negative Subgroup Positive (BNSP) AUC: where AUC is calculated on the positive examples of the target group, and the negative examples of the background. This metric computes whether the model favors the target group with respect to the others. This value is reduced when toxic examples of the group have lower toxicity scores than non-toxic examples of the background.
In this experiment, we finetune the text encoder under study on hate speech detection task using the training set of HateXplain dataset (Mathew et al., 2021). We also use the test portion of HateXplain for the evaluation, which contains posts from Twitter17 and Gab18 annotated with their ground-truth toxicity scores and the social groups and communities they target. Fundamentally, the three metrics described above give bias scores per group. In order to combine the per group scores in one overall measure, we apply the Generalized Mean of Bias (GMB) introduced by the Google Conversation AI Team as part of their Kaggle competition19, and later used by Mathew et al. (2021) in their own evaluations. The formula of GMB is as the following:
GMB(b) = ( 1
|b| |b|∑ g=1 bpg) 1/p (4)
where b is an array of AUC scores per group, and bg is the AUC score of group g. We follow Mathew et al. (2021) and set p to -5. We compute the GMB of all three metrics: Subgroup, BPSN and BNSP. As for Subgroup, we also add the standard deviation as it gives valuable information about how much the performance of the hate speech detection model varies across groups. We report our results in Table 13, in addition to classic performance measures.
We observe that Att-D provides competitive results across the four bias metrics, and largely outperforms the baselines. Especially with GMB-BNSP, where bias scores of the original model are very low (i.e. it is throttled by social biases), we observe the best improvements overall, and by a large margin compared to existing debiasing methods. Also, the variance in model performance is lowest with Att-D, which confirms that the corresponding hate speech detection model has less stereotypes about different social groups. Finally, the general performance (Accuracy, F1 score and AUC) of the hate speech detection model after debiasing is not hurt.
17https://twitter.com 18https://gab.com 19https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification/overview/evaluation

heading: A.8 WORD-LEVEL VS SENTENCE-LEVEL DEBIASING
text: As previously explained in the paper, Att-D calibrates the attention weighs of all tokens of the input sentence on group-related words. Since we used BERT-based models in our experiments, the first token in the input is the special [CLS] token, which is considered by the NLP community as a vector representation for the entire input sentence. In the current version of Att-D, we also calibrate the attention weighs of the special [CLS] token on groups, in addition to calibrating the other tokens of the sentence. One can see this notion as a combined word-level and sentence-level debiasing. In this experiment, we motivate this design choice by comparing it to word-level and sentence-level debiasing separately. For word-level, we exclude the [CLS] token from the attention equalization process, whereas in sentence-level we only calibrate the attention of [CLS]. We use all the bias evaluations run so far to understand the difference in performance. Tables 14, 15, 16, 17 and 18 report the results of StereoSet, Crows-Pairs, inference, hate speech and GLUE experiments respectively. We denote word-level debiasing by No [CLS], and sentence-level debiasing by Only [CLS] in the tables. The combination of both is referred to as Att-D, and is the variant that we promote in this paper. We observe that while the three settings are good at reducing bias from text encoders, Att-D is superior than word-level and sentence-level debiasing since it capitalizes on the benefits of both. It enjoys the fine granularity of reducing bias from every word, while it also mitigates biases that manifest at sentence-level.

heading: A.9 STATIC VS RANDOM ORDERING OF GROUP-RELATED WORDS
text: In the preprocessing step of our method (as explained in Section 3.2.1), we use a preset ordering of group-related words of a given bias type to form the second input. For example, if we have the groups Muslim, Christian, Jew and Buddhist defining the religion bias type, Att-D constructs the second input using the same preset ordering of groups across all samples of the training data. Continuing the example above, Att-D appends the following artificial sentence ”muslim, christian, jew, buddhist”. In this experiment, we change the ordering of groups in a random way. Tables 14, 15, 16, 17 and 18 also report the bias scores of Att-D (static ordering) and Att-D with random ordering.
Although the semantic performance of Att-D with random ordering is better, we notice that it suffers from a stronger presence of bias than in its static counterpart. In Table 17, Att-D with random ordering has an AUC score of 0 in one of the groups, which made the GMB extremely small. We suspect that the relatively poor fairness of random ordering owes to the fact that the model might be confused by different orderings throughout the iterations. A more serious analysis of the impact of group order on the overall performance (fairness and semantics) of Att-D motivates the direction of future work.

heading: Models Att-D No [CLS] Only [CLS] Random Order

Review summaries:
This paper proposes a new debiasing method for contextualized word embeddings, specifically for attention-based text encoders. At a very high level, the proposed method tries to calibrate the attention scores of words from different groups, e.g. to reduce gender bias, the method forces the model (text encoder) to allocate same attentions to word "man" and "woman". Experimentally, the paper also demonstrates relatively good results on both likelihood-based evaluation (StereoSet and Crows-Pairs) and inference-based evaluation (NLI).

This paper addresses potential biases introduced by attention models by re-weighing the attention weights. First, the paper provides a few examples that demonstrate attention weights correlated with social stereotype (e.g., doctor attending to he and nurse attending to she). Then, it proposes to reduce this type of bias by "calibrating" the attention weights. To do so, each sample is augmented with pairs of words corresponding to the groups for which the model is intended to mitigate bias. The augmented samples are used in training, and, thus, each token in the input will attend to tokens in the augmented portion of the sample. The weights are changed such that the weights attending on the augmented, group-relevant tokens are similar. This type of re-weighing can lead to some of the semantic meaning encoded in the network to change. To prevent this change, the attention weights corresponding to the tokens from the original sample are forced to follow the weights in an unaltered model that is trained in parallel. Last, the paper suggested also utilizing negative sampling: using random words for augmentation and forcing all attention weights to follow the ones of the teacher.  For evaluation, the paper shows the performance of several transformer-based models.  The evaluation of the method includes three different benchmarks: two benchmarks that measure bias intrinsically (Crows-Pairs and StereoSet) and an NLI task designed for measuring bias. To make sure that the semantic strengths of the model are not lost, the models are also evaluated using the GLUE tasks. The model obtains competitive performance for the GLUE tasks, while reducing bias compared to original models and a couple of related works.