Debiasing Pretrained Text Encoders by Paying Attention to Paying Attention
Authors: Yacine GACI
Boualem Benatallah
Fabio Casati
Khalid Benabdeslem
Venue: ICLR 2022 Submitted (2022)
Keywords: Fairness
Pretrained Text Encoders
Self-Attention
Knowledge Distillation
Social Biases
Debiasing

Abstract: Recent studies in fair Representation Learning have observed a strong inclination for natural language processing (NLP) models to exhibit discriminatory stereotypes across gender, religion, race and many such social constructs. In comparison to the progress made in reducing bias from static word embeddings, fairness in sentence-level text encoders received little consideration despite their wider applicability in contemporary NLP tasks. In this paper, we propose a debiasing method for pre-trained text encoders that both reduces social stereotypes, and inflicts next to no semantic offset. Unlike previous studies that directly manipulate the embeddings, we suggest to dive deeper into the operation of these encoders, and pay more attention to the way they pay attention to different social groups. We find that the attention mechanism is the root of all stereotypes. Then, we work on model debiasing by redistributing the attention scores of a text encoder such that it forgets any preference to historically advantaged groups, and attends to all social classes with the same intensity. Our experiments confirm that we successfully reduce bias with little damage to semantic representation.

Under review as a conference paper at ICLR 2022
DEBIASING PRETRAINED TEXT ENCODERS BY PAY- ING ATTENTION TO PAYING ATTENTION
Anonymous authors Paper under double-blind review
ABSTRACT
Recent studies in fair Representation Learning have observed a strong inclination for Natural Language Processing (NLP) models to exhibit discriminatory stereo- types across gender, religion, race and many such social constructs. In comparison to the progress made in reducing bias from static word embeddings, fairness sentence-level text encoders received little consideration despite their wider appli- cability in contemporary NLP tasks. In this paper, we propose a debiasing method for pre-trained text encoders that both reduces social stereotypes, and inflicts next to no semantic damage. Unlike previous studies that directly manipulate the em- beddings, we suggest dive deeper into the operation of these encoders, and pay more attention to the way they pay attention to different social groups. We find also encoded in the attention layer. Then, we work
that most stereotypes model debiasing by redistributing the attention scores of a text encoder such that it forgets any preference to historically advantaged groups, and attends all so- cial classes with the same intensity. Our experiments confirm that we successfully reduce bias with little damage to semantic representation.
INTRODUCTION
Natural Language Processing (NLP) increasingly penetrating real-world operations such re- cruitment (Hansen et al.] 2015), legal systems (Dale) 2019). healthcare (Velupillai et al.] 2018)) and Web Search (Nalisnick et al.J 2016). Part of this success attributed to the underlying embedding layer which encodes sophisticated semantic representations of language (Camacho-Collados & Pile- hvar 2018). The wide adoption of modern NI.P models in critical domains has also inflicted more thorough scrutiny. Recent research has uncovered some propensities of NL.P models to replicate discriminatory social biases (Bolukbasi et al. 2016: Caliskan et al.) 2017 May et al., 2019) which may cause unintended and undesired model behaviors with respect social groups. Social bias in NLP is mainly caused by unbalanced mentions of attributes near advantaged groups in training data (Zhao et al. 2018a). For example, in most existing text corpora, very few cooks are referred to by male pronouns (e.g. he, him, himself) (Zhao et al. (017). Accordingly, text encoders or language models trained such data may use this shorteut to inadvertently disassociate cooks from men, and learn that cooking is female attribute. Worse, NLP models may even amplify social biases if left unchecked (Bommasani al. 2021).
Methods to debias static word embeddings such as Word2vec (Mikolov et al. 2013) or GloVe (Pen- nington al. 2014) have been applied for gender, race and religion (Bolukbasi al. 2016 Zhao et al. 2018b; Kaneko & Bollegala, 2019; Raafogel et al. 2020). However, by the time NLP prac- titioners started casting more attention the fairness problem of their models, they had already switched to the more powerful sentence-level transformers in the likes of BERT (Devlin et al.) 2018). GPT3 (Brown et al., (020) or T5 (Raffel et al.) 2020) which owe their success to novel self-attention mechanism (Vaswani et al. 2017). This leap in accuracy in several NLP tasks does not extend to fairness since research discovered social stereotypes in modern text encoders (May al. 2019. Nadeem et al. 2020 Nangia al.). To date, debiasing them remains comparatively under-explored.
Mitigating biases in text encoders is difficult for four reasons: (1) They expensive to retrain, so conventional methods based on Counterfactual Data Augmentation (CDA) to rebalance group- attribute mentions (Zhao et al. 2018al Webster et al. 2020 become prohibitive they generate

---

Under review as a conference paper at ICLR 2022
more training data, and all debiasing attempts might be limited to either finetuning or adapting (Houlsby et al. 2019; Lauscher et al. 202I). (2) Static embeddings encode words whereas text
encoders need context Thus, it is not straightforward to use existing debiasing techniques for static
emheddings off-the-shelf as it is not clear how to generate context for single words. Previous work
tackled this problem by either designing bleached sentence templates (May al. [019 Kurita al.)
2019) where they fill in the blanks with words of interest, or sampling sentences from large corpora where the words are mentioned (Liang et al. 2020a Cheng et al. 2020), thus creating context. The former betrays the expressiveness of natural language while the latter suffers from sampling and pre- processing bias (Liang et al., 2020a). (3) The input space of text encoders is the set of all possible sentences, so we cannot debias every single input as it is done with static embeddings. (4) Text encoders are larger capacity and complexity. This suggests that they can accommodate subtler and more sophisticated forms of stereotype, especially in their attention component, which renders bias imperceptible to existing detection methods as they are not designed to operate on attention.
Despite these difficulties, previous works (Liang et al., 2020a; Webster et al. 2020: Liang et al. 2020b [Ceng et al. 2020 Kaneko & Bollegala 202 Lauscheree et al. 202 1) addressed the problem of reducing bias from modern text encoders with different techniques. However, most of them make strong assumptions about the linearity of bias. Moreover, they operate on the embeddings produced
by text encoders, and leave their most important block attention largely unrectified.
In this paper we explore attention-based dehiasing. This approach stems from our observation that attention exhibits great deal of social biases. We empirically show that this is the case, propose a novel method to reduce stereotypes from attention blocks, and demonstrate that it is effective in mitigating biases from sentence representations as a whole. Given an input sentence, our method compels the text encoder of interest to redistribute its internal attention scores such that each word in the input allocates the same attention for different social groups. Thus, it learns to forget previously encoded preferences, and generate fair representations, free of stereotypical influence. We also keep semantic information loss at a minimum while debiasing by distilling knowledge (Hinton et al.) 2015 Gou et al. (02T) from an unaltered teacher text encoder. In this setting, we encourage the debiased model to copy the original attention from its teacher to minimize semantic offset. Unlike most previous work which focus only on gender, we address five bias types in our experiments (gender, race, religion, age and sexual orientation). We conduct likelihood- and inference-based evaluations to measure the intensity of bias in our final debiased models. Experiments demonstrate that the technique we propose effectively reduces bias, and outperforms existing debiasing methods.
2 RELATED WORK
In this section, we discuss related work about debiasing static word embeddings and sentence-level text encoders. Then, we shed some light on work done on the attention mechanism in general.
should be noted that bias at data level (Pryzant al. 2020 Cryan al. 2020) and in language generation tasks (Sheng et al. 2020 Sap et al.] 20200] Dhamala et al. 2021) are also active and
complementary areas of research. However, due to space limitations, they will not be discussed in this paper.
2.1
BIAS STATIC WORD EMBEDDINGS
The work of Bolukbasi et al. (216)) pioneered bias research in NLP by discovering that static word embeddings such as Word Vec (iikolov et al. 2013) or GloVe (Pennington et al., 2014) encode significant amounts of binary gender bias. They proposed Hard-Debias: a simple method move biases by projecting gender-neutral word embeddings onto a gender-free direction. Maazinn] et al. (2019) extended Hard-Debias to the multiclass setting where they also treat racial and reli- gious stereotypes. In both works, the bias direction is defined by a manually pre-compiled list of stereotyped words. In contrast, Raviogel al. (2020) suggest data-driven approach to learn bias directions with linear classifier. Debiasing is then conducted by iteratively projecting word em- beddings on the null space of the classifier's matrix. On the other hand, finetuning is the debiasing approach that attracted the widest adoption, either by using autoencoder (Kaneko & Bollegala 2019, attraction-repllsion mechanism (Kumar et al. 2020), or adversarial attacks (Xie et al. 2017
A word needs be in context (sentence or paragraph) in order to be correctly encoded
2

---

Under review as a conference paper at ICLR 2022
Liet 2018 Elazar & Goldberg, 2018 1 Unlike these post-processing methods, Zhao et al. (2018b)
added a new fairness constraint to GloVe loss function, and retrained their fair word embeddings from scratch.
2.2 BIAS IN TEXT ENCODERS
Research on biases in sentence representations is dominated by detection rather than correction and mitigation. To date, there are three main approaches to detect stereotypes in text encoders: (1) representation-baze
where vector relationships between different types of inputs are measured. For example, |May et al. (2019) extended the WEAT test (Caliskan et al. 2017) into sentence vector space (SEAT), and compared the cosine similarity between representations of two sets of targets and two sets of attributes. All sentences in SEAT follow predefined template. (2) likelihood-basee These approaches examine how often text encoders prefer stereotypes over anti- anti-stereotypes. Prefer- ences in this case are defined in terms of higher likelihoods as produced by language models using embeddings of the text encoders under study. Two benchmarks are widely used for measuring bias: StereoSet (Nadeem et al., (220)) and Crows-Pairs (Nangia et al.). Both datasets are organized in pairs or triples of minimally- distant sentences which differ only in the word(s) carrying a stereotyp- ical connotation.
(3) inference-based: These methods employ text encoders in downstream NLP tasks (Blodgett et al., 2020) such as natural language inference (Dev et al. 2020), sentiment analysis (Diaz et al. 2018) or language generation (Sap et al. 2020 Sheng et al. 2020). Bias in such settings declared as the difference in outcome when the models tested with the same input sentence,
differing only in social groups.
Bias mitigation approaches are mostly inspired by debiasing static embeddings. In projection-based methods, [iang et al.] (2020a) contextualize words into sentences by sampling them from exist- ing corpora before applying Hard-Debias. Kaneko & Bollegala (2021) minimize the projection of sentence representations on a learned bias subspace, while Qian et al. (2019); Bordia & Bowman
2019 Liang et al. (2020b) add bias-reduction objectives their loss functions. Another line of
research uses CDA (Webster et al., (020)) to balance gender correlations in training data, while Lauscher et al. (2021) use adapters to reduce the large training time that CDA incurs. Finally, Cheng et al. (2020) use contrastive learning, and add fair filter that minimizes mutual information between stereotypes and anti-stereotypes In our work, rather than extending approaches from static embed- dings, we focus on the self-attention mechanism which is characteristic of many text encoders, and show that fair attention leads to fair representations.
2.3 ATTENTION ANALYSIS IN TEXT ENCODERS
Clark et al. (019)) analyzed BERT's attention heads and found that some of them correspond re- markably well to linguistic patterns of coreference and syntax without additional training. [Michel et al. (2019) observe that not all attention heads within model are made equal. They also propose pruning algorithm to reduce the energy footprint of these models by eliminating the least important heads without much attenuation to the overall performance. Given the convenient interpretabilty of attention, it has also been used myriad of visualization works (Vig) 2019 Hoover et al.) 2020)) Tenney et al., 2020 Bastinng&& & Filippova 2020) in an attempt dissect and explain the inner func- tioning of text encoders. Most attention studies in text encoders are designed for analysis purposes. In contrast, we are the first leverage the attention mechanism in order to make text encoders fairer and less stereotyped.
DEBIASING METHOD
3.1 MOTIVATING EXAMPLE & INTUITION
Despite the applicability of our work on any model that is built upon self-attention, we focus in this paper on models based on the encoder side of the transformer architecture, such as BERT (Devlin) et al. 2018). RoBERTa (Liu et al.] 2019), or ALBERT (Lan et al. 2019). This owes to the decoder side being usually used in nuto-regression tasks, and less often to encode text. Transformers consist of multiple layers, each composed of self-attention block followed by a feed- feew-forwwrd block to make embeddings. A self-attention block contains multiple heads. Each head transforms the input into
3

---

Under review as a conference paper at ICLR 2022
Laco
Logent 11
Lawer [2
121
[0.49
thy
may
aded
good
goodd
0
[S0P]
1id 1o
ate
Joft
(b)
Figure 1: Attention patterns in BERT suggest the existence of potential gender and racial biases
attention weights between all pairs of tokens in the input sequence (aswwnn et al., 2017), such that each token learns to attend its most related tokens, hence the prevalence of attention defining the understanding of natural language. In this work, we assume that undesired social stereotypes are primarily encoded in the self-attention block. In order to verify this hypothesis, we show and analyze some attention maps of BERT in Figure [F] Consider the following sentence "The doctor to different social groups, we add a dummy second input consisting of words representing two distinct genders (he and she after the [SEP] token). Figure [(a) illustrates that the token doctor pays much more attention to he than she Figure ((b) reveals that nurse attends to she. This finding suggests that gender stereotypes are deeply encoded in attention weights. Likewise, in Vigure[T(c) the token math more related to asian than to white or black, conforming the famous racial stereotype casting asians as really good mathematicians. These examples align with our intuition that social stereotypes are first and foremost encoded in the self-attention block of text encoders before they propagate to their embeddings or predictions in downstream NLP tasks. Consequently, we propose Art-D, a finetuning method for reducing undesired biases of text encoders from their attention component.
asked the murse a question." " Aiming to analyze how every word eppresentationrrelateee
The intuition of our debiasing strategy is as follows: Given that attention weights conform with undesired biases (e.g., doctor attending to he, and nurse to she Figure we aim to equalize the attention scores of every word in the input sentence with respect to social groups. Following the examples of Figure [] Att-D redistributes the attention scores of doctor such that it attends to he and she with the same intensity, thus eliminating any preference toward one of the groups However, alterations to the attention of doctor on the remaining words of the input sentence must be kept to a minimum in order not to corrupt the semantic understanding of the original text encoder.
3.2 DEBIASING WORKFLOW
Att-D consists of three steps: First, for each input sentence s in the training corpus S, we make an artificial second input 8, consisting of words related to social groups (similar to the examples on Figure D. Pretrained text encoders expect either one or two inputs before they produce attentions and embeddings. In this work, we use both (as first input) and 5g (as second input) separated by [SEP] token. Consequently, the resulting self-attention includes both s and 8g. The second step of Att-D equalizes the attention weights of all heads in all the layers of the text encoder of interest such that each token in 8 pays the same amount of attention to tokens of 8g, thus eliminating pref- erences and stereotypes. Finally, we minimize semantic loss by compelling our model to learn the original semantics from an unaltered teacher model by copying its internal attention in a knowledge distillation setting (Hinton al. 2015).
We schematize the operation of Att-D in Figure 2 spond to the tokens of sg. Both matrices represent one attention head of the text encoder be- fore (left) and after (right) debiasing. The matrices should be read in rows. Each row depicts the attention weights of the corresponding token on all the other tokens of the input (s + $g
Gr Gra and Gr, in the figure corre-
2 'The figures are produced using bertviz tool:
AAcordinng BERT in this example
"Dark colors correspond to high attention scores, and light colors indicate low attention scores
$[CT S] token (vector representation of s) is also included for attention calibration. Details in Appendix
4

---

Under review as a conference paper at ICLR 2022
The matrices are conceptually split in four blocks: (1) attentions of on 8, (2) attentions of Sg. (3) attentions of sy on s, and (4) attentions of sy on 8g- Debiasing consists in making the columns of block 2 equal. In other words, each token in s pays the same amount of attention all the groups as indicated in the right side of Figure Ideally, debiasing should also preserve the semanties of the original text encoder. That is why block of Figure should be kept unchanged. Both blocks and are ir- relevant to the results, since they denote atten- tions of our artificially inserted second input sg. Besides, they do not participate in defining nei- ther fairness nor representativeness of the text encoder. So, do not impose any restrictions them. In the following, describe each step of Att-D in detail.
(a)
(b)
Figure 2: Overview an attention head before (a)
and after (b) debiasing
3.2.1 GENERATING AUGMENTED INPUTS
The first step involves identifying bias types that we want to mitigate from pretrained text encoders, such as gender, race, religion. This achieved by defin- ing a set of tuples G such that G (T1, T2...., TiH where each T7 describes social groups of a given bias type, or their attributes. In the case of binary gender,
Table Examples of group tuples per bias
binary gender
age
man, woman he, she
old, young
Ti T= {"male", "female") or Ti = {"he", "she") both possible definitions (and many others are possible for non-binan binabi cases). Table Table shows some tuples that we do not aim to learn bias subspaces. Therefore, Att-D charges next to no constraint in the defini- tion of G, contrary existing methods which need group definitions way, and given in enough quantities. On the other hand, providing a single tuple per bias type is sufficient for Att-D function, even though more contributes to better precision.
elderly, youth
in Att-D. Unlike previous work (Liang et al. 2020a Kaneko & Bollegala 202
be formulated in certain
During debiasing, we pick tuple T. from G randomly and construct 8g. a bleached sentence formed by words of T. For example, given Table [ 5g can be "man, woman" "old, young". The input to the text encoder is both the original input sentence and the artificial one $g. Pretrained text encoders separate the two halves of the input with special token [SEP] as illustrated in Figure ! and compute attention maps for the entire sequence (s + 8g)-
3.2.2 EQUALIZING ATTENTIONS SOCIAL GROUPS
After obtaining attention maps of the augmented input from Section 3.2.1 which is produced by the pretrained text encoder of interest E, make each token of pay equal amounts of attention to the tokens of 5, which define social groups. The rationale is to eliminate any inclination for to prefer a social group to the detriment others. Suppose (.h.s.s, Atin(s, 8yi h) is the attention matrix at layer 1, head of the encoder E, computed from the input + Sg. Here, make the reasonable assumption that 5, contains at least two social groups In this spirit, equalizing attention vectors of block 2 (as defined in Figure] 2 equivalent to making them equal to a pivot vector. In our method, consider the attention vector of the first social group the pivot (first column in block 2 of Figure 2 and minimize the mean square error between the pivot and the attention vectors on the other groups,
time. The equalization loss is given by Equation
H || Legu >Σ
"Biases are usually about making one or more groups (dis)advantaged with respect to the others, hence the existence of at least two groups per bias type
5

---

Under review as a conference paper at ICLR 2022
where L is the number of layers of the text encoder, H the number of heads, ||sy|| the number of social groups in sy and a is the position of the special token [SEP] that marks the end of s and the beginning of Sy. As be seen, A A.o.0+1 is the pivot vector containing attention scores of the first social group token (whose position is directly after [SEP], i.e., o + 1). Equation | forces attention scores on subsequent social groups be the same on the first one, thus making them all equal.
3.2.3 PRESERVING SEMANTIC INFORMATION
Text encoders must preserve their ability to represent natural language and keep the same perfor- mance downstream NLP tasks. For this reason, debiasing must ensure that useful information is preserved as much as possible. We minimize semantic information loss in a knowledge distillation setting (Hinton et al. 2013 Gou et al., 202T). We cast the text encoder that we want debias as the student model, and recruit another model to play the role of the teacher. We do not apply our debiasing strategy on the teacher since it provides a reference to the original unaltered language representations. We distill semantic information in the form of attention maps from the teacher and instill it in the student. Stated differently, we compel the student to learn from the teacher and reproduce its attention scores for every input sentence in the training corpus S. In this work, the student and the teacher models must share the exact same architecture in order to make knowledge distillation flow correctly from source to target, unlike other works in knowledge distillation (Gou et al. 2021 wherein the student is much smaller and should be more compact than the teacher for model compression purposes.
As in Section 3.2.2 let 1.,,5,5 be the attention of the student model at layer I, head h with s and $g as input. Likewise, let Olhh.s,3, define the same attention matrix, but for the original teacher model. We formalize the preservation of semantic information as a regularizer where we minimize the squared l2 distance between the student's and the teacher's attention scores according to the following:
1.
H
Litistil ΣΣΣ
|| 1.h,s,3 0,50 sa.:0
RES 1=1 h=1
where L is the number of layers, is the number of heads, and o is the position of the [SEP] token. As can be seen from Equation [ the student learns only replicate block (as in Figure 2 of the attention matrices. This is because block contains attention scores of the original input sentence on itself, thus encoding an important aspect of semantics. We force the student not to reproduce the attention distribution on social groups (block 2) from the teacher since these are fundamentally biased, and are left to the care of our debiasing objective. We deseribe the overall training objective as linear combination of the previously defined losses, with À as hyperparameter to control the weight of debiasing over semantic preservation.
Loss Lusstti + ALegu
3.3 NEGATIVE SAMPLING & LAYER SELECTION
The strict application of Att-D discussed so far may accidentally lead some undesired spurious phenomena. While learning to equalize attention on social groups that constitute the second half of the input, the text encoder might potentially bear the risk of distributing its attention uniformly on any second half of the input, no matter what it is. This is particularly alarming when the text encoder is subsequently employed in double-sentence tasks (Wang et al. 2018) such as semantic textual similarity, paraphrase detection sentence entailment.
To overcome the above obstacle, we introduce negative sampling. Instead of using words related to social groups in order to generate the artificial second input sn. we randomly sample words (negative examples) from the vocabulary. In this case, we do not equalize the attentions but compel the student to copy its teacher even for blocks 2, 3 and 4. We do this in order to prevent the text encoder from learning to assign the same attention weight to all tokens of the second input when these do not define social groups We control the ratio of negative examples with hyperparameter 7).

---

Under review as a conference paper at ICLR 2022
Another concern when debiasing text encoders is that their layers do not necessarily encode the same information. Bhardwaj et al. (2021) found that BERT layers display widely different reactions when probed with a gender-detection classifier. This means that they do not encode gender stereotype iddetiiclll.. Therefore, it is not clear which layers and/or attention heads are best for debiasing. To investigate this issue, we consider seven settings: debiasing all layers. first 6. first 3, last 6, last 3, and alternating layers with strides of one or two We find that working with all layers yields the best results, So, unless otherwise specified in this paper, debiasing concerns all the layers.
4 EVALUATION
In this section, we first describe our experimental setup, then evaluate Att-D from two viewpoints: fairness and representativeness. Unlike previous work in debiasing text encoders (Liang et al.) 2020a Cheng et al.) 2020, Kaneko & Bollegala| (202T) we do not conduct representation-basee evaluations- SEAT May et | (019) and WEAT (Caliskan et al., 2017 in this paper to measure bias since their effectivvness for such a purpose has been questioned (Silva et al. (02I). On the other hand, we evaluate fairness with both likelihood- and inference-based methods. Other quantitative and qualitative evaluations can be found in the appendix
4.1 DEBIASING SETUP
In the experimental evaluation we focus on five different bias types which are commonly addressed by the bias literature and debiasing approaches. To facilitate comparison, we also follow existing literature in defining the social groups for each type of bias, although the approach presented here is not restricted to that, and can be leveraged for both other kinds of biases and for a more inclusive definition of the groups. the experiments we show results of debiasing based on (binary) gender imale.femalee race (white, black, asian, hispanic), religion (muslim, christian, jewish, buddhist), age (young, old) and sexual orientation (heterosexual, non-heterosessal). We leverage the defini- tion words from previous work (Liang et al., 2020a) and augment them with our tuples. The full list can be found in the appendix. We apply our debiasing method on BERT" (Devlin et al., 2018), and use the News-commeetary--11. corpus training data. It contains 223, 153 sentences of which we use 80% for training and 20% for development.
4.2 EVALUATIONS OF FAIRNESS
4.2.1
LIKEL EVALUATION
We use the publicly available subsets of two stereotype benchmarks: StereoSet (Nademm al. 2020 and Crows-Pairs (Nangia et al.). Both provide likelihood-bass diagnostics to measure how often stereotypes are considered more likely than anti-stereotypes by the text encoder of interes [2 Crows- Pairs is organized in pairs of minimally distant sentences, i.e. sentences that differ only in one word, usually the one describing social group. We pick the following example from the dataset: Jews are such greedy people (Stereotype) versus Christians are such greedy people (Anti-stereotype). Crows- Pairs uses word likelihoods produced by the text encoder to measure how often stereotypes are ranked higher than anti- strtissttrroy.ss.... An ideal unbiased text encoder should score 50% in the Crows- Pairs challenge, meaning that it prefers neither stereotypes nor anti anti-stereotypes. 100% and 0% are both undesired extreme outcomes. In contrast, StereoSet adds third absurd sentence capture the language modeling capabilities of the text encoder in addition to measuring bias. The triple of sentences in StereoSet example. in Girls tend to be more [MASK] than boys, StereoSet generates the evaluation sentences by replacing the mask with the following attributes: soft denote stereotype, determined for anti- stereotyping, and fish to make an unrelated sentence. As in Crows-Pairs, StereoSet quantifies how
also minimally distant but differ in the attribute word, not in the group. For
'Layers 2, 4. 6, 8, 10 and 12 of BERT *Layers 4, 8 and 12 of BERT
'We provide code and data as supplementary material to this submission "In the appendix, we also apply Au-DD ALBERT, RoBERTa, DistiBBERT and SqueezeBERT "htp://www.statmt wwwtaatt.trggwwmmm.. http///m///ansslaionntaaaaa html The text encoder must first be fine-tuned on language modeling objective
7

---

Under review as a conference paper at ICLR 2022
Table 2: Language modeling (lm) and Stereotype scores (ss) of different text encoders StereoSet
Models
Original BERT
Sent-D
Kaneko
Att-D*
Att-D
Overall (Im/ss)
83.70
56.04
81.39
54.71
85.58
56.04
80.92
53.37
83.34
53.04
gender (Im/ss)
82,35
62.75
76.67
53.33
83.73
58.82
73.33
51.37
78.24
53.73
race (Im/ss)
86.28
54.68
85,40
55.09
87.47
56.24
85.03
54.37
86.28
51.87
religion (Im/ss)
87.82
56,41
88.46
51.28
85,90
57.69
85.90
55.13
88.46
53.85
profession (Im/ss)
80.66
55.50
77.44
55.01
83.87
54.76
77.94
52.66
80.96
54.14
much the model prefers stereotypes over anti-stereotypes. Moreover, it also measures how often meaningful sentences rank higher than unrelated sentences in order to assess the quality of the text encoder in terms of representativeness. An ideal model should have a stereotype score (ss) of 50% and a language modeling (Im) of 100%.
We compare method against the original BERT base model for fair comparisons against previous work, we decided to include the baselines whose final debiased models have been published in order avoid errors of training and/or tuning hyperparameters. Thus, we compare Att-D against Sent-D (Liang et al.] (020a) and the debiasing procedure proposed by Kaneko & Bollegala| (202I). Sent-D currently holds state-of-the-art performance debiasing. We also conduct simple ablation study by training without negative examples (Att-D'). We finetune Att-D and the baselines on language modeling to able conduct likelihood- basdd experiments. Tables 2 3 the evaluation results on StereoSet and Crows-Pain respectively.
the effect of debiasing Also,
We see that the original BERT contains signif- icant levels of biases (56.04 in StereoSet and It is important to note
60.48 in Crows-Pairs that Kaneko & Bollegala (202T) only focused on gender bias, which is clear in Table 2 where only gender stereotype has been reduced. We observe that focusing on one bias type can make text representations even more biased for the other dimensions as can be seen in Kaneko
Table 3: Bias measurements of different text en-
coders on Crows-Pain
Models
BERT
Sent-D
Kaneko
Att-D
Att-D
Overall
60.48
$6.90
$7.82
57.23
55.7
gender
57.63
57.36
race
58.14
55.23
53.68
53.68
51.15
religion
71.43
60.0
64,76
69.52
64.76
age
55.17
51.72
$4,02
$4.02
43.68
for race and religion (Table 2). and for sexual orientation and disability (Table In contrast, Att-D always reduces the intensity of stereotyp- ing in BERT (up to 9,53%) and yields the best results overal We also notice that we manage to reduce biases linked dimensions we did not in- clude in our design such as profession, nationality and disability. We speculate that these bias types are connected to those we worked on mitigating. Therefore, we conjecture that reducing multiple biases at the same time meets better success in mitigating unforeseen stereotypes than working on every bias type separately.
sexual orientation
67.86
70,24
69.05
66.67
58.33
nationality
62.89
56.6
59.12
61.01
57.86
disubility
61.67
65.0
68.33
63.33
60.0
4.2.2 INFERENCE-BASEE BASED EVALUATION
This approach of measuring bias builds on the intuition of Dev et al. (2020) stating that biased representations lead to invalid inferences, whose ratio quantifies bias. They construct a challenge benchmark for the natural language inference task where every hypothesis should be neutral to its premise. For example, suppose that the premise is The driver owns van and the hypothesis is The man owns van. The hypothesis neither entails nor contradicts the premise. If the predictions of classifier deviate from neutrality, the underlying text encoder is doomed as biased. All the data points in(Dev et al. (2020) follow the same structure of the example above. and span across gender, religion and race. Suppose that the set contains M instances, and let the predictor's 's probabilities of the ith instance for entail, contradict and neutral be c,, 6; and n,. Following Dev et al. (2020), we report three measures of inference-basee
1 bias: (1) Net Neutral (NN): NN = M Σ Mi; (2) Fraction Σ. 1,427-
Neutral (FN): FN
ΣM M
Giml mn:=maa(ssemmm)) (3) Threshold 7 (T:r): T
13 Results of BERT large, ALBERT, RoBERTa, BEsT and SqueezeBERT are in the appendix
The closer the sterentype score is to 50%, the better
8

---

Under review as a conference paper at ICLR 2022
Table Performance of different models on GLUE tasks. The table shows accuracy scores for sst2, rte, wnli, and mnli for both matched and mismatched instances; fl for mrpe: spearman correlation for stsb: and matthews correlation for cola
Models
Single sentence tasks
Double sentence tasks mali (mm)
s522
cola
stab
murpe
mali (m)
wnli
BERT
92.78
56.05
|| 888.97
92.25
83,54
82.68
70.04
45.07
Sent-D
9163
-1.15
59.08
+3.03
89.58
+0.61
90.12
-2.13
84.97
+1.43
83.51
+0.83
68.95
-1.09
28.17
-16.9
Kaneko
91.97
-081
56.50
40.55
88.44
-0.53
-1.56
84.48
40.94
83,66
40.98
-10.11
52.11
47.04
AII-
92.66
-0.12
55.22
-0.83
89.62
+0.65
91.22
-1.03
84.63
+1.09
84.19
+1.51
70.40
+0.36
53.52
+8.45
In this experiment, we finctune text encoders of in- terest with the MNLI dataset for natural language inference (Wang et al. 2018). An ideal bias-free model should score in all three measures. We re- port our findings in Table 4 after transforming them into percentages. Our method outperforms the orig- inal model and the baselines. This result shows that Att-D not only reduces bias at surface-level as re- flected by likelihood-inspired evaluation methods, but also mitigates stereotypes in real world inference settings, unlike Sent-D which produces positive re- sults with Crows-Pairs but comes short of meeting the same success in this experiment. In the next section, we show that these findings are meaningful since the entailment accuracy is not hurt after debi- asing.
Table 4: Inference- bias measurements. Best scores are highlighted with bold char- acter, underlined, or marked with i for gen- der, race and religiont respectively
Model
Bius type
NN
FN
7:0.5
r:0.7
gender
00.59
00.
BERT
race
75.96
76.57
76.51
74.91
religion
43.47
43.55
geader
00.94
00.38
00.33
00.24
Sent-D
race
59.61
59.28
59,20
56.22
religion
29.64
29.08
29.02
27.24
gender
00.57
00.14
00.12
00.08
Kaneko
race
84.24
84,84
84,80
83,26
religion
69.271
60.801
69.72!
67. 67.66'
gender
01.31
00.43
00.35
00.21
Au-D
race
93.31
93.94
93.90
93.04
religion
68.51
69.08
68.95
66.97
4.3 EVALUATIONS OF REPRESENTATIVENESS
We use GLUE benchmark (Wang et al. 2018 to verify whether the debiased text encoder still holds enough semantic information to be applicable in downstream NLP tasks. In essence, GLUE assesses the natural language understanding capabilities of NLP models. So, it constitutes a suitable stack to evaluate the semantic preservation of Att-D. In this experiment, we finetune BERT on seven different tasks from GLUE and show the results in Table We also report the difference in accuracy between original BERT and each of the debiasing baselines. Surprisingly. Att-D not only preserves semantic information, but enhances it in most GLUE tasks as reflected in an increase in accuracy from BERT.
5 CONCLUSION AND FUTURE WORK
In this paper, we proposed finetuning approach to debiasing that trains the text encoder to distribute its attention equally on different social groups Experiments demonstrate that bias is successfully reduced without harm to semantic representativennss.. However, we are aware of the following limi- tations: (1) our definitions of biases are simplified. There are more social divisions in the real world studied. Besides, bias types can be correlated in intricate ways such as the links between race, nationality and ethnicity. Moreover, it is not clear which or how many groups include. For these reasons, follow previous work and constrain our study simple definitions targeting the most widely spread groups. We plan to study the effect that the choice of definition tuples and their order impose overall. (2) We equalize the attention scores of every word in the input. However, some words are inherently charged with a strong inclination toward one group, e.g., heard male or pregnant to female. Such words need not be debiased, which requires compiling expensive lists of related words for every social group and protecting them from attention cqualization. In this work, we rely on knowledge distillation to retain as much useful semantic in- formation as possible. (3) Current bias detection experiments have positive predictive ability, which means that they can only detect the presence of bias, not the absence of it. Although contemporary evaluation tools demonstrate that our method successfully reduced social biases from sentence rep- resentations, it is possible that bias is still lurking in shapes and forms that our experimental lenses failed to detect. We plan to address these limitations in future work.
than the five dimensions

---

Under review as a conference paper at ICLR 2022
REFERENCES
Jasmijn Bastings and Katja Filippova. The elephant in the interpretability room: Why use attention as explanation when we have saliency methods? In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pp. 149-155, 2020.
Rishabh Bhardwaj, Navonil Majumder, and Soujanya Poria. Investigating gender bias in bert. Cog- nitive Computation, pp. 1-11, 2021.
Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. Language (technology) is power: A critical survey of "bias" in nlp. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5454-5476, 2020.
Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. Man is to computer programmer as woman homemaker? debiasing word embeddings. Advances in neural information processing systems, 19:4349-43577 2016.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportu- nities and risks of foundation models. arXiv preprint arXiv:210.007258 2021.
Shikha Bordia and Samuel Bowman. Identifying and reducing gender bias in werd-level language models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop, pp. 7-15, 2019.
Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Nuanced metrics for measuring unintended bias with real data for text classification. In Companion pro- ceedings of the 2019 world wide web conference, pp. 491-500, 2019.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.00. /11.0005..111.. /4165, 2020.
Pete Burnap and Matthew I. Williams. Us and them: identifying cyber hate on twitter across multiple protected characteristics. EPJ Data science, 5:1-15, 2016.
Aylin Caliskan, Joanna Bryson, and Arvind Narayanan. Semanties derived automatically from language corpora contain human-like biases. Science, 356((6334) 2017.
Jose Camacho-Collados and Mohammad Taher Pilehvar. From word to sense embeddings: A survey on vector representations of meaning. Journal of Artificial Intelligence Research, 63:743-788,
2018.
Pengyu Cheng, Weituo Hao, Siyang Yuan, Shijing Si, and Lawrence Carin. Fairfil: Contrastive neural debiasing method for pretrained text encoders. In International Conference on Learning Representations, 2020.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. What does bert look at? an analysis of bert's attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 276-286, 2019.
Jenna Cryan, Shiliang Tang, Xinyi Zhang, Miriam Metzger, Haitao Zheng, and Ben Zhao. De- tecting gender stereotypes: Lexicon supervised learning methods. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pp. 1-11, 2020.
Robert Dale. Law and word order: NIp in legal tech. Natural Language Engineering, 25(1):211- 217, 2019.
Sunipa Dev, Tao Li, Jeff M Phillips, and Vivek Srikumar. On measuring and mitigating biased inferences of word embeddings. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 7659-7666, 2020.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: /0.0: 04.05, 2018.
10

---

Under review as a conference paper at ICLR 2022
Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai- kei Chang, and Rahul Gupta. Bold: Dataset and metries for measuring biases in open-ended language genera- tion. In Proceedings of the 2021 ACM Conference Fairness, Accountability, and Transparency, pp. 862-872, 2021.
Mark Diaz, Isaac Johnson, Amanda Lazar, Anne Marie Piper, and Darren Gergle. Addressing age- related bias in sentiment analysis. In Proceedings of the 2018 chi conference on human factors in computing systems, pp. 1-14, 2018.
Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Measuring and miti- gating unintended bias in text classification. Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Sociery, pp. 67-73, 2018.
Yanai Elazar and Yoav Goldberg, Adversarial removal of demographic attributes from text data. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp- 11-21, 2018.
Seraphina Goldfarb- Tarran,, Rebecca Marchant, Ricardo Muñoz Sanchez, Mugdha Pandya, and Adam Lopez. Intrinsic bias metrics do not correlate with application bias. arXiv preprint arXiv:2012. 15859, 2020.
Jianping Gou, Baosheng Yu, Stephen Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision, 129(6): 189-1189, 2021.
Hansen, Tosik, G Goossen, Li, Bayeva, Berbain, and Rotaru. How to get the best word vectors for resume parsing. In SNN Adaptive Ineeliiennce/ymmposiu::: Machine Learning, 2015.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv:1503.0255/1 2015.
Benjamin Hoover, Hendrik Strobelt, and Sebastian Gehrmann. exbert: A visual analysis tool to explore learned representations in transformer models. In Proceedings the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 187-196, 2020.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly, Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pp. 2790-2799. PMLR, 2019.
Forrest landola, Albert Shaw, Ravi Krishna, and Kurt Keutzer. Squeezebert: What can computer vision teach nlp about efficient neural networks? In Proceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing, pp. 124-135, 2020.
Masahiro Kaneko and Danushka Bollegala. Gender-preserving debiasing for pre-trained word em- beddings. In Proceedings of the 57th Annual Meeting the Association for Computational Lin- guistics, pp. 1641-1650, 2019.
Masahiro Kaneko and Danushka Bollegala. Debiasing pre-trained contextualised embeddings. In Proceedings of the 16th Conference of the European Chapter of the Association for Computa- tional Linguistics: Main Volume, pp. 1256-1266, 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6968 2014.
Vaibhav Kumar, Tenzin Singhay Bhotia, and Tanmoy Chakraborty. Nurse closer to woman than surgeon? mitigating gender-biased proximities in word embeddings. Transactions of the Associ- ation for Computational Linguistics, 8:486-503, 2020.
Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black, and Yulia Tsvetkov. Quantifying social biases in contextual word representations. In Ist ACL Workshop on Gender Bias for Natural Language Processing, 2019.
11

---

Under review as a conference paper at ICLR 2022
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv: 11942, 2019.
Anne Lauscher, Tobias Lüken, and Goran Glavaš. Sustainable modular debiasing of language mod- els. arXiv preprint urXiv:2109.0360 00,443,6,, 2021.
Yitong Li, Timothy Baldwin, and Trevor Cohn. Towards robust and resentations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 25-30, 2018.
text rep-
Paul Pu Liang, Irene Mengze Li, Emily Zheng, Yao Chong Lim, Ruslan Salakhutdinov, and Louis- Philippe Morency. Towards debiasing sentence representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5502-5515, 2020a.
Sheng Liang, Philipp Dufter, and Hinrich Schütze. Monolingual and multilingual reduction of gen- der bias in contextualized representations. In Proceedings of the 28th International Conference on Computational Linguistics, pp. 5082-5093, 2020b.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. 1692, 2019.
Thomas Manzini, Yao Chong Lim, Yulia Tsvetkov, and Alan W Black. Black to criminal as cau- casian is to police: Detecting and removing multiclass bias in word embeddings. arXiv preprint arXiv:1904.04447 2019.
Binny Mathew, Punyajoy Saha, Seid Muhie Yimam, Chris Biemann, Pawan Goyal, and Animesh Mukherjee. Hatexplain: A benchmark dataset for explainable hate speech detection. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 14867-14875, 2021.
Chandler May, Alex Wang, Shikha Bordia, Samuel R Bowman, and Rachel Rudinger. On measuring social biases in sentence encoders. NAACL-HLT (1), 2019.
Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? 2019.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen- tations in vector space. arXiv preprint arXiv:1301.3781, 2013.
Moin Nadeem, Anna Bethke, and Siva Reddy. Stereoset: Measuring stereotypical bias in pretrained language models. arXiv preprint arXiv:2004.0944 09445, 2020.
Eric Nalisnick, Bhaskar Mitra, Nick Craswell, and Rich Caruana. Improving document ranking with dual word embeddings. In Proceedings of the 25th International Conference Companion on World Wide Web, pp. 83 84, 2016.
Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R Bowman. Crows-pairs: A challenge dataset for measuring social biases in masked language models.
Ho Park, Jamin Shin, and Pascale Fung. Reducing gender bias in abusive language detection. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp- 2018
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1543, 2014.
Reid Pryzant, Richard Diehl Martinez, Nathan Dass, Sadao Kurohashi, Dan Jurafsky, and Diyi Yang. Automatically neutralizing subjective bias in text. In Proceedings of the aaai conference on artificial intelligence, volume 34, pp. 480. 2020.
12

---

Under review as a conference paper at ICLR 2022
Yusu Qian, Urwa Muaz, Ben Zhang, and Jae Won Hyun. Reducing gender bias in word-level language models with a cender-equalizin loss function. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties: Student Research Workshop, pp. 223- 228, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21 21(140): 2020.
Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. Null it out: Guard- ing protected attributes by iterative nullspace projection. arXiv preprint 17XXix:0004076666 2020.
Manoel Horta Ribeiro, Pedro H Calais, Yuri Santos, Virgilio AF Almeida, and Wagner Meira Jr. Characterizing and detecting hateful users on twitter. In Twelfth international AAAI conference on web and social media, 2018.
Victor SANH, Lysandre DEBUT, Julien CHAUMOND, Thomas WOLF, and Hugging Face. Distil- bert, a distilled version of bert: smaller, faster, cheaper and lighter.
Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A Smith, and Yejin Choi. Social bias frames: Reasoning about social and power implications of language. In Association for Computational Linguistics, 2020.
Emily Sheng. Kai-Wei Chang, Prem Natarajan, and Nanyun Peng. Towards controllable biases in language generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings. pp. 3239-3254 2020.
Andrew Silva, Pradyumna Tambwekar, and Matthew Gombolay. Towards a comprehensive under- standing and accurate evaluation of societal biases in pre-trained transformers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2383-2389, 2021.
Ian Tenney, James Wexler, Jasmijn Bastings, Tolga Bolukbasi, Andy Coenen, Sebastian Gehrmann, Ellen Jiang, Mahima Pushkarna, Carey Radebaugh, Emily Reif, al. The language interpretabil- ity tool: Extensible, interactive visualizations and analysis for nlp models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstra- tions, pp. 107-118, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008 2017.
Sumithra Velupillai, Hanna Suominen, Maria Liakata, Angus Roberts, Anoop D Shah, Katherine Morley, David Osborn, Joseph Hayes, Robert Stewart, Johnny Downs, et al. Using clinical natural language processing for health outcomes research: overview and actionable suggestions for future advances. Journal of biomedical informatics, 88: 88:11-19, 2018.
Jesse Vig. Visualizing attention in transformer-based language representation models. arXiv preprint arXiv:1904 02679, 2019.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 353-355, 2018
Kellie Webster, Xuezhi Wang, Ian Tenney, Alex Beutel, Emily Pitler, Ellie Pavlick, Jilin Chen, Ed Chi, and Slav Petrov. Measuring and reducing gendered correlations in pre-trained models. arXiv preprint arXiv:2010.0603: 2020.
Qizhe Xie, Zihang Dai, Yulun Du, Eduard Hovy, and Graham Neubig. Controllable invariance through adversarial feature learning. Advances in Neural Information Processing Systems, 30: 585-596, 2017.
13

---

Under review as a conference paper at ICLR 2022
Ziqi Zhang. David Robinson, and Jonathan Tepper. Detecting hate speech twitter using convolution- gru based deep neural network. In European semantic web conference, pp. 745-760. Springer, 2018.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai- Wei Chang. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 2017.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai- Wei Chang. Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of the 2018 Confer- ence of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume 2 (Short Pupers), pp. 15-20, 2018a.
Jieyu Zhao, Yichao Zhou, Zeyu Li, Wei Wang, and Kai-Wei Chang. Learning gender-neutral word embeddings. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 4847-4853, 2018b.
A APPENDIX
A.1 TRAINING HYPERPARAMETERS
We used Adam optimizer Kingma& & Ba 2014) with learning rate of 5c for 3 epochs. We keep the betas to their default values (0.9, 0.999) as PyTorch implementation (Paszke et al. 2017). We set the loss coefficient À to 2.0 and the negative ratio v/ to 0.8 meaning that in 80% of the iterations, we use negative examples whose number we set to in each negative iteration. We only finetuned the values of X, 17, the learning rate, and the number of epochs. We conducted the hyperparameter search manually on the development set.
As for GLUE experiments, we follow the experimental setup of Devlin et al. (2018) and train each task for 3 epochs with learning rate of -5-5 on their respective training data.
A.2 SOCIAL GROUPS USED IN THE EXPERIMENTS
While the approach is independent of the definition social groups and categories (it could work for any kind of grouping, e.g., cuisine styles or sports), in the experiment we focus on groups commonly used in the debiasing literature: binary gender, religion, race, age and sexual orientation. This is to facilitate comparison, but nothing in the approach prevent it from being used with broader and more inclusive groups. This being said, we have not experimented yet with debiasing where a dimension is divided in dozens of categories.
We list the definition tuples that we used in Table We show that Att-D does not incur strict rules for defining social groups, unlike previous work (Bolukbasi al. 2016) Kaneko & Bollegala [019 2021) that require the definition words to be organized in a predetined format (pairs of words or bag of words for every group), and provided in relatively large quantities. We can see from Table that it is sufficient to define one tuple per bias type (c.g., race) if the tuples are hard to come by. Also, the tuples need not be of the same size (c.g., in religion there is a missing word for buddhist group since it is not clear which word to use in that tuple). This desired property owes to the fact that Att-D does not learn subspaces or directions for every bias type as previous works do (Bolukbasi et al., 2016) Kaneko & Bollegala, 2019 Kumar et al.) 2020 Kaneko & Bollegala, (021). In contrast, AttDD uses the tuples in order to equalize the attentions the input sentence, and make the words therein attend to the groups with the same intensity. These example categories used in experiments neither complete nor exhaustive, and in some experiments also include terms possibly considered inappropriate but that appear in the corpus and we may still want to debias from (such as using "straight" to define heterosexual).
A.3 CHOICE OF LAYERS
Transformer- based text encoders consist of many layers. is not clear which layers to choose for debiasing since bias information is spread out across all of them. In this experiment, try different
14

---

Under review as a conference paper at ICLR 2022
Table 6: Full list of definition tuples for bias types and social groups used in this work
gender
religion
male
female
muslim
christian
jewish
buddhist
man
woman
muslim
christian
jewish
buddhist
father girl
boy
muslims
christians
jews
buddhists
mother
islam
christianity
judaism
buddhism
brother sister
mosque
church
synagogue
temple
grandfather
grandmother
quran
bihle
torah
son daughter
imam
priest
rabbi
monk huddha
gentleman lady
mohammad
jesus
moses
his she
he
her
himself herself
race hispanic
age
sexual orientation
white black
asian
old
young
heterosexual
non-
heterosexual
white
black
asian
hispanic
young youth adult child senior junior teenager
old
straight straight heterosexual heterosexual
gay lesbian homosexual
elderly
bisexual
adult
Table 7: Language modeling (Im) and Stereotype scores (ss) of different layer combinations StereoSet. Underlined depicts the best language modeling score, while bold shows the best stereo-
type score
Models
first 3
first 6
Inst 3
last
I-stride
2-stride
all
Overall (lmiss))
83.17
54,28
78.80
54.04
82.51
54.13
81.92
54.33
82.70
54.42
82.68
54.04
83.34
53.04
gender (Im/ss)
78.04
55.29
71.96
55.69
78.43
56.08
77.65
55.69
76.47
55.29
78.43
54.51
78.24
53.73
race (lmn/ss)
87.11
54.05
83.16
54.16
85.71
54.05
85.40
54.57
85.76
54.26
86.38
52.91
86.28
51.87
religion (Inv's))
87.82
51.28
87.82
$7.69
85,90
52.56
88.46
$7.69
88.46
53.85
86.54
60.26
88.46
53.85
profession (In/ss)
79.67
54.51
74.91
$3.03
79.67
53.77
78.49
53.28
80.47
54.39
79.23
54.64
80.96
54.14
debiasing settings in which we select different layer combinations of BERT to work on: all layers, first 6. first last 6, last 3, and alternating layers with strides of (layers 2, 4, 6, 8. 10 and 12) or 2 (layers 4, 8 and 12). We apply the debiasing method proposed in this paper, and report both language modeling and stereotype scores StereoSet benchmark in Table
The results show that it is safest to equalize attention heads of all layers of the text encoder under study, since it produces the best scores both in terms of language modeling and stereotype. Our findings in tandem with those of Liang et al. (2020b); Kancko & Bollegalal (202I): Bhardwaj
ct al. 202T who found that reducing bias from all layers usually is the best option.
A.4
EFFECT OF NEGATIVE EXAMPLES ON REPRESENTATTIVENSS
We remind that the introduction of negative examples training serves in forcing the text encoder not to rely on a dangerous shortcut which is distributing its attention uniformly on all the tokens constituting the second half of the input, no matter what the input is. This is particularly important in double-sentent tasks where the text encoder is given two input sentences. In addition to Tables [ and 3 which highlighted the effect of negative sampling on the final stereotype scores, the primary goal of using negative examples remains the preservation of the text encoder's representativeness. In Table 8) we report the performance of Att-D and Att-D with and without negative examples respectively GLUE tasks. Unsurprisingly, the lack of negative examples does not damage the performance of single-sentence tasks since these ignore the second half of the input altogether. However, in double-sentence tasks where both halves are used for prediction, Table g shows that negative sampling plays a pivotal role in preserving the semantics of text encoders, and bypassing the side effects inflicted by attention equalization.
15

---

Under review as a conference paper at ICLR 2022
Table Effect of negative examples on GLUE tasks. The table shows accuracy scores for sst2, rte, wnli, and mnli for both matched and mismatched instances; fl for mrpe; spearman correlation for stsb; and matthews correlation for cola
Single sentence tasks
Double sentence tasks
Models
sst2
cola
stsb
mrpc
mnli (m/mm)
rtc
wnli
BERT
92.78
56.05
88.97
92.25
83.54 82.68
70.04
45.07
Att-D
92.66
55.22
89.62
91.22
84.63 84.19
70.40
53.52
Att-D
92.32
56.25
89.12
80.44
84.59 83.96
58.12
39.44
A.5 VISUALIZING DEBIASING RESULTS
In this experiment, we aim visualize the effects of debiasing on attention weights. We only focus on binary gender bias for two reasons: First, it is easier to visualize binary variables on 2D plane than multiclass variables (such as race, religion...). Second, gender is the most well studied bias type (Bolukbasi et al., 2016 Caliskan et al.] [217]] |Ma et al.) 2019), so linguistic resources and vocabularies for gender exist and are well documented. We use the vocabulary words compiled by Kaneko & Bollegala (2019) and categorized into three non-overlapping subsets: (1) Male-definition S2300 whose corresponding words are exclusively male-gendered such as father, king uncle. (2) Female-definition Ω which is set of inherently female words (mother, queen, aunt...). (3) Gender-stereotype ? which is constituted of words that are not gendered by definition, but that carry a strong gender stereotype such as doctor being attributed to male nurse to female.
For every word w E 92 U U we extract sentences from the News-commentarr.v// corpus where is mentioned. We denote this set as 5 Then, for every sentence s S append the dummy input "man, woman" as explained in Sections ?? and 3.2.1 The augmented input s' is then fed the text encoder of interest (BERT' base in this experiment), and we collect the attention scores of w on the second-hai tokens man and woman. Finally, for every word w E M UnS, we take the mean of its attention scores in S By the end of this procedure, we have for every word w its attention score on the words man (o and woman (af) as computed on the News-commentary-ve corpus which includes overall 223,15 sentences. We take the difference 0x ay which indicates the preference of the text encoder to consider w as male (positive difference) or female (negative difference). The absence of gender bias is reflected in difference scores near zero
We plot the results in Figure 3 where the x-axis represents the differences am aa and the y-axis random values to separate the words vertically. Stereotype words (green dots) should have values near 0, which is not the case in Figure ((b) This means that BERT has a strong preference for one of the genders, and is thus heavily biased. contrast, our method brings the attention of stereotype words near O, meaning that they prefer neither male nor female connotations. Moreover, the spread of stereotype words in iid narrower than male- or female-oriented words, which is desired since these are inherently gendered and must pick side. This result strengthens the claim that Att- D preserves semantic information, and is less severe in reducing bias from gendered words as on gender-neutral words. The difference in spread is less apparent in the original BERT model. We also note that debiasing the embeddings of BERT rather than the attention mechanism as in ((c) is not enough since bias information still lurking (and perbaps made worse for some words) in the attention component. Thus, we conclude that working on attention directly constitutes our best option for debiasing to date.
Kaneko & Bollegala| (202I) (Figure 3(c))
A.6 EFFECT OF ATTENTION- BAASED DEBIASING ON OTHER TRANSFORMER-BAN BBSED TEXT ENCODERS
We evaluate five widely used sentence-level text encoders: BERT (Devlin al. 2018). ALBERT (Lnn et al. 2019), RoBERTa (Liu et al. (201)), DistiBBERT (SANH et al.) and BERuT (Tan- dola et al. 2020). For each model, evaluate both its base and large variants (except for DiRT and SqUeezeBERT since these are not available in HuggingFace's 's transformers library 15 original and debiased: which gives total of sixteen evaluated models. We use Crows-Pain Pairr dataset (Nan-
15
16

---

Under review as a conference paper at ICLR 2022
cenegyy oricnted
fomale oriented
sterestype
Tamare one 'edd Suro0i1pp
(a)
(b)
catogery
lamralecceerted
stnnechyce
-0 012 -0.300 -0.003 -0 006 -d 001 032 c.000 0007
0.001
-0.012 -0 OLD -a 000 -0.36 -olcon -0.000 ncoo 0002
6.004
score
2007
(c)
(d)
Figure 3: Scatter plots of attention scores on male female direction. (a) Original BERT, (b) BERT debiased by Sent-D (c) BERT debiased by Kaneko & Bollegala (2021) (d) BERT debiased by Att-D
17

---

Under review as a conference paper at ICLR 2022
Table 9: Bias reduction in BERT base and large measured Crows-Pairs dataset. Each cell is organized as follows: + d +/-diff where o is the stereotype score of the original model, d is that of the debiased model using attention-based debiasing, and di is the difference stereotype score. Negative values correspond to reduction in bias (desired) where positive values mean addition of bias (undesired).
Models
BERT base
BERT large
Overall
60.48 55.70
-04.78
59.68 + 56.96
-02.72
race
58.14 51.15
-06.99
60.08
53.49
-06.59
gender
58.02 + 57.36
-00.66
55.34 + 53.05
-02.29
socioeconomic
59.88 + 51.16
-08.72
56.40 + 57.56
+01.16
nationality
62.89 + 57.86
-05.03
52.20 + 57.23
+05.03
religion
71.43 - 64.76
-06.67
68.57 + 66.67
-01.90
age
55.17 43.66
+01.15
55.17 + 54.02
-01.15
sexual orientation
67.86 - 58.33
-09.53
65.48 + 67.86
+02.41
physical appearance
63.49 61.90
-01.89
69.84 65.08
-04.76
disability
61.67 + 60.00
-01.67
76.67 - 65.00
-11.67
gia et al.) to quantify the intensity of undesired stereotypes encoded therein. As a reminder, ideal stereotype scores according to Crows-Pairs benchmark should be close to 50, i.c. models preferring neither stereotypes nor anti-stereotypes. Tables T0. and 12 show the bias results for BERT, ALBERT, RoBERTa and DistilIBERT/SqueezeBEER respectively.
All five models exhibit substantial levels of bias, and in each of the bias types with differing in- tensities (religion, sexual orientation and disability being the bias categories with the most severe stereotyping). Also, we find that the large variants are more biased than their base counterparts mainly because large models, with their larger capacity and greater number of parameters, can cap- ture more intricate and more sophisticated aspects of training data, exposing them to learn more bias. This finding corresponds well to results of previous work (Nangia al. Nadeem al.) 2020). The tables also show that Att-D is effective in mitigating bias from BERT, ALBERT, RoBERTN Distil- BERT and BERT, and produces a reduction of up 25%. We note that Att-D succeeds in debiasing all models, with varying effectiveness across bias types. We also note that Att-D meets the best success with ALBERT as reductions are greater on this particular text encoder. We believe this is because ALBERT is composed of single transformer layer (Lan al. 2019) with substantially less parameters than BERT or RoBERTa; which makes debiasing casier since there is no interference between different attention layers. Finally, to adding a bit of bias. We observe that this phenomenon is rare, and happens especially with bias types we did not include in our design We assume that not explicitly compelling the text encoder to equalize attention heads corresponding to these overlooked bias types gave it green light to adjust these attentions in a way facilitate solving the optimization problem; even if it entails adding bias. We plan to include all bias types present in Crows-Pairs dataset to our debiasing design as a future work.
from the tables that Att-D sometimes contributes
A.7 APPLICATION-ORIETTE BIAS EVALUATION
Recent studies show that intrinsic metries of bias do not necessarily correlate with bias measures on concrete real-world applications (Goldfarb- Tarrant et al. 2020). In the body of this paper, we already conducted intrinsic and extrinsic bias evaluations. this experiment, we validate the efficacy of our debiasing method on concrete real-world hate speech detection application where an input snippet of text is classified as either offensive (toxic, harmful, disrespectful...) or not. We use hate speech detection because it is well studied in the literature (Burnap & Williams) 2016 Ribeiro et al. 2018 Zhang et al. 2018). and high-quality datasets which are tagged with social groups already exist (Borkan et al. 2019 Mathew et al. 2021).
16 the current version of this work, remind that only consider five hias types: gender, race, religion, age and sexual orientation
18

---

Under review as a conference paper at ICLR 2022
Table 10: Bias reduction in ALBERT base and large measured on Crows-Pairs dataset. Fach cell is organized as follows: the debiased model using attention-basel debiasing, and di f is the difference in stereotype score. Negative values correspond to reduction in bias (desired) where positive values mean addition of bias (undesired).
+/-diff where o is the stereotype score of the original model, d is that of
Models
ALBERT base
ALBERT large
Overall
56.76 + 51.99
-04.77
60.48 + 53.58
-06.90
race
51.36 - 48.84
-00.20
59.11 - 50.97
-08.14
gender
54.20 + 53.44
-00.76
56.11 + 48.47
-04.58
socioeconomic
60.47 - 61.05
+00.58
54.07 - 50.00
-01.16
nationality
51.57 - 57.86
+06.29
62.26 - 60.38
-04.07
religion
59.05 - 60.00
+00.95
76.19 61.90
-14.29
age
65.52 + 42.53
-08.05
54.02 > 54.02
-00.00
sexual orientation
75.00 38.10
-13.10
71.43 63.10
-08.33
physical appearance
46.03 - 41.27
+04.76
58.73 57.14
-01.59
disability
86.67 61.67
-25.00
73.33 58.33
-15.00
Table 11: Bias reduction in RoBERTa base and large measured on Crows-Pairs dataset. Each cell is organized as follows: d +/-diff where is the stereotype the debiased model using attention-based debiasing, and dif f is the difference in stereotype score. Negative values correspond reduction in bias (desired) where positive values addition of bias (undesired).
of the original model, d is that of
Models
RoBERTa base
RoBERTa large
Overall
53.98 + 51.39
-02.59
61.27 + 56.83
-04.44
race
47.09 + 50.39
-02.52
61.43 + 53.49
-07.94
gender
54.96 45.80
-00.76
51.91 51.91
-00.00
socioeconomic
56.40 + 55.81
-00.59
66.28 + 59.88
-06.40
nationality
45.28 43.40
+01.88
56.60 55.35
-01.25
religion
56. - 60.00
+03.81
59.05 + 62.86
+03.81
age
64.37 56.32
-08.05
71.26 + 62.07
-09.19
sexual orientation
69.05 - 48.81
-17.86
71.43 59.52
-11.91
physical appearance
66.67 + 60.32
-06.35
68.25 66.67
-01.58
disability
71.67 65.00
-06.67
66.67 + 70.00
+03.33
Table 12: Bias reduction in DistiBERT and SqueezeBERT measured on Crows-Pairs dataset. Each cell is organized as follows: o + d +/-diff where is the stereotype score of the original model, d is that of the debiased model using attention- based debiasinn, and fiff the difference in stereotype score. Negative values correspond to reduction in bias (desired) where positive values mean addition of bias (undesired).
Models
DistilBERT
SqueezeBERT
Overall
56.83
51.26
-05.57
57.43 + 54.71
-02.72
53.29 47.87
-01.16
55.04 56.01
+00.97
gender
54.58 + 46.56
-01.14
52.67 + 48.47
-01.14
socioeconomic
55,81 58.14
+02.33
57.56 51.16
-06.40
nationality
54.09 + 50.94
-03.15
53.46 + 61.01
+07.55
religion
70.48 + 57.14
-13.34
74.29 + 60.95
-13.34
age
59.77 48.28
-08.05
55.17 + 48.28
-03.45
sexual orientation
70.24 - 55.95
-14.29
70.24 + 57.14
-13.10
physical appearance
55.56 63.49
+07.93
52.38 52.38
-00.00
disability
61.67 + 56.67
-05.00
70.00 61.67
-08.33
19

---

Under review as a conference paper at ICLR 2022
Admittedly, common social biases have also been shown to exist in hate speech detection models, for example in associating toxicity to frequently attacked groups (such as "muslim" or "gay") even if the text itself is not toxic (Dixon et al.) 2018 Park et al.] 2018). In this experiment, we adopt the bias definition of Borkan et al. 2019) which casts bias as a skewing in the hate speech detector scores based solely the social groups mentioned in the text. In other words, we consider model to exhibit unintended social stereotypes if the model's performance varies across groups. We use the bias measures proposed by Borkan al. (2019) which are based on the Area Under the Receiver Operating Characteristic Curve (ROC-AUC, or AUC) metric. AUC measures the probability that a randomly chosen negative example (not offensive) receives a lower toxicity score than randomly chosen positive example (offensive), meaning that perfect model should always have an AUC score of 1.0. Stated differently, all negative examples have lower toxicity scores than positive examples. While AUC is used to measured the general performance of classifiers, BBrrann et al. (2019) propose three extensions of AUC to mcasure bias. We summarize them in the following:
Subgroup (Sub) AUC: where AUC is computed only on the group under consideration and not on all the examples of the test benchmark, i.e. only positive and negative examples of the target group considered. This metric represents the model's performance means that the model is good at distinguishing between toxic and non-toxic texts specifie to the group.
given group. A higher value
Background Positive Subgroup Negative (BPSN) AUC: where AUC is calculated on the negative examples of the target group, and the positive examples of the background (all other groups except the group under consideration). This metric computes whether the model discriminates against the target group with respect the others. This value is reduced when non-toxic examples of the group have higher toxicity scores than actually toxic examples of the background.
Background Negative Subgroup Positive (BNSP) AUC: where AUC is calculated on the positive examples of the target group, and the negative examples of the background. This metric computes whether the model favors the target group with respect to the others. This value is reduced when toxic examples of the group have lower toxicity scores than non-toxic examples of the background.
In this experiment, we finetune the text encoder under study on hate speech detection task using the training set of HateXplain dataset (Mathew et al., (2021). We also use the test portion of HateXplain for the evaluation, which contains posts from Twitter" and Gat toxicity scores and the social groups and communities they target. Fundamentally, the three metrics described above give bias scores per group. In order to combine the per group scores in one overall measure, we apply the Generalized Mcan of Bias (GMB) introduced by the Google Conversation Al Team as part of their Kaggle competition[") [19] and later used by Mathew et al. (0021) in their own evaluations. The formula of GMB is the following:
with their ground-truth
GMB(b) = ///// 1/P
(4)
where 6 is an array of AUC scores per group, and by is the AUC score of group g. We follow Mathew et al. (2021) and set to -5. We compute the GMB of all three metrics: Subgroup, BPSN and BNSP. As for Subgroup, we also add the standard deviation as it gives valuable information about how much the performance of the hate speech detection model varies across groups. We report our results in Table [3 in addition to classic performance measures.
We observe that Att-I) provides competitive results across the four bias metrics, and largely outper- forms the baselines. Especially with GMB-BNSP, where bias scores of the original model are very low (i.e. it is throttled by social biases), observe the best improvements overall, and by large margin compared to existing debiasing methods. Also, the variance in model performance is lowest with Att-D, which confirms that the corresponding hate speech detection model has less stereotypes about different social groups Finally, the general performance (Accuracy, Fl score and AUC) of the hate speech detection model after debiasing is not hurt.
1 "htttss//wwitter.com 18 https://aab.com
19 hwwpsw/wwwwecccooooooooooooooooe
20

---

Under review as a conference paper at ICLR 2022
Table 13: AUC-based bias measures on hate speech detection task
Performance
Bias
Models
Acct
FIf
AUCT
STD-Sub,
GMB-Subt
GMB-BPSNT
GMB-BNSP
BERT
0.783
0.823
0.870
0.119
0.698
0.800
0.379
Sent-D
0.791
0,825
0,870
0,121
0.689
0.725
0,583
Kaneko
0.797
0.833
0.872
0.112
0.705
0.789
0.512
Att-D
0.789
0.829
0.866
0.085
0.808
0.793
0.726
A.8 WORD-LEVEL VS SENTENCE-LEVEL DEBIASING
As previously explained in the paper, Att-D calibrates the attention weighs of all tokens of the input sentence on group-related words. Since we used BERT-basee models in our experiments, the first token in the input is the special [CLS] token, which is considered by the NLP community as a vector representation for the entire input sentence. In the current version of Att-D, we also calibrate the attention weighs of the special [CLS] token on groups, in addition to calibrating this notion combined word-level and sentence-level
other tokens of the sentence. One debiasing. In this experiment, we motivate this design choice by comparing it to word-level and sentence-level debiasing separately. For word-level, we exclude the [CLS] token from the attention equalization process, whereas in sentence-level we only calibrate the attention of [CLS]. We use all the bias evaluations and repoor the results of StereoSet, Crows-Pairs, inference, hate speech and GLUE experiments respectively. We denote word-level debiasing by No [CLS], and sentence-level debiasing by Only [CLS] in the tables. The combination of both is referred to as Att-D, and is the variant that we promote in this paper. We observe that while the three settings are good at reducing bias from text encoders, Att-D is superior than word-level and sentence-level debiasing since it capitalizes on the benefits of both. It enjoys the fine granularity of reducing bias from every word, while also mitigates biases that manifest at sentence-level.
far to understand the difference in performance. Tables 14 15 T6 17
A.9 STATIC VS RANDOM ORDERING OF GROUP-RELATED WORDS
In the preprocessing step of our method (as explained in Section 3.2. we use a preset ordering of group-related words of given bias type to form the second input. For example, if have the groups Muslim, Christian, Jew and Buddhist defining the religion bias type, Att-D constructs the second input using the same preset ordering of groups across all samples of the training data. Continuing the example above, Att-D appends the following artificial sentence "muslim, christian, jew, buddhist". In this experiment, we change the ordering of groups in a random way. Tables [4] [5] 17
and 18 also report the bias scores of Att-D (static ordering) and Att-D with random ordering. 16
Although the semantic performance of Att-D with random ordering is better, we notice that it suffers from stronger presence of bias than in its static counterpart. In Table 17] Att-D with random ordering has an AUC score of in one of the groups, which made the GMB extremely small. We suspect that the relatively poor fairness of random ordering owes to the fact that the model might be confused by different orderings throughout the iterations. A more serious analysis of the impact of group order on the overall performance (fairnsss and semantics) of Att-D motivates the direction of future work.
21

---

Under review as a conference paper at ICLR 2022
Table 14: Language modeling (Im) and Stereotype scores (ss) on StereoSet of different variants of Att-D
Models
Att-D
No [CLS]
Only [CLS]
Random Order
Overall (Im/ss)
83.34
53.04
80.37
53.71
81.70
55.51
82.91
54.75
gender (Im/ss)
78.24
53.73
76,86
52.94
75.88
54.51
79.02
55.69
race (Im/ss)
86.28
51.87
84.10
53.01
85.24
55.09
86.75
54.57
religion (Im/ss)
88.46
53.85
84.62
60.26
85.26
56.41
87.18
56.41
profession (Im/ss)
80.96
54.14
76.63
54.14
78.99
56.24
79.17
54.51
Table 15: Bias measurements of different variants of Att-D on Crows-Pairs Random Order
Models
Att-D
No [CLS]
Only [CLS]
Overall
55.7
56.1
55.5
58.36
gender
57.36
50.76
50.0
53,82
race
51.15
54.84
53.1
57.75
religion
64.76
69.52
65.71
67.62
age
43.68
56.32
44.83
54.02
sexual orientation
58.33
71,43
63.1
64.29
nationality
57.86
53,46
65.41
62,28
disability
60.0
61.67
58.33
65.0
Table 16: Inference-baseee bias measurements on different variants of Att-D. Best scores are high- lighted with bold character, underlined, or marked with for gender, race and religiont respec- tively
Model
Bias type
NN
FN
7:0.5
7:0.7
gender
01.31
00.43
00.35
00.21
Att-D
race
93.31
93.94
93.90
93.04
religion
68.51
69.08
68.95t
66.97t
gender 00.85
00.36
00.30
00.20
No [CLS]
race
76.14
76.24
76.19
74.26
religion
40.80
40.04
39.98
37.78
gender 02.35
01.60
01.38
00.90
Only [CLS]
race
81.63
81.52
81.50
80.37
religion
44.40
44.01
43.95
42.76
gender race
01.54
00.51
00.39
00.23
Random Order
54.71
54.92
54.89
52.49
religion
26.94
26.67
26.59
24.58
Table 17: AUC-based bias measures hate speech detection task on different variants of Att-D
Performance F11
Bias
Models
Accf
AUCt
STD-Subl
GMB-Subt
GMB-BPSNI
GMB-BNSP
Att-D
0.789
0.829
0.866
0.085
0.808
0.793
0.726
No [CLS]
0.791
0.830
0.871
0.114
0.710
0.797
0.530
Only [CLS]
0.765
0.805
0.838
0.142
0.660
0.766
0.636
Random Order
0.784
0.822
0.861
0.764
22

---

Under review as a conference paper at ICLR 2022
Table 18: GLUE performance of different variants of Att-D. The table shows accuracy scores for sst2, rte, wnli, and mnli for both matched and mismatched instances: fl for mrpe; spearman cor- relation for stsh; and matthews correlation for cola
Single sentence tasks
Double sentence tasks mnli (m/mm)
Models
sst2
cola
stsb
mrpc
rte
wnli
Att-D
92.66
55.22
89.62
91.22
84.63 / 84.19
70.40
53.52
No [CLS]
91.51
40.85
88.94
91.62
84.49 / 84.02
68.95
40.85
Only [CLS]
92.43
55.23
89.43
90.04
84.42 84.67
71.84
23.94
Random Order
93.23
59.07
88.85
91.94
83.75
84.86
71.84
30.99
23


Reviews:
[{"review_id": "baAlTaPq10", "reviewer": "Reviewer_KjSU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. While we have seen a number of interesting papers focusing on debiasing static word embeddings, debiasing (even measuring) contextualized word embeddings is underexplored.  As pretrained text encoders become more and more powerful, this paper is approaching a very challenging and important problem. \n\n2. The way that the proposed method to correct the attention scores is smart. More specifically, it attaches a sequence of words from different social groups to the original training sentence, which makes it very controllable to adjust the attention scores assigned to specific words. \n\n3. The proposed method also has an advantage of reducing multiple types of biases (e.g. gender, race, age, ...) simultaneously and there can be more than 2 groups for certain biases (although in Table 1, examples are all binary).\n\nWeakness:\n1.  The experimental results, especially in Table 4, are not very convincing. The proposed method seems only work well on reducing race bias. Gender bias seems to be the hardest one to mitigate across all methods and Kaneko is doing better on reducing religion bias.\n2. The paper also need to provide more details on hyper-parameter tuning, e.g. lambda in Eq (3). And also more details and ablation studies on negative sampling and layer selection.\n3. I am not fully convinced about the motivation (intuition) in Sec. 3.1. Figure1 definitely presents some good examples of bias in encoders but it is hard to conclude that bias is mostly from the encoder. The paper needs more quantitative analysis on bias in encoders and also decoders to support this claim.\n4. The paper mentioned that bias measurements like WEAT have been questioned. However, Stereo-set and Crows-Pairs have also been criticized [1]. \n5.I am wondering that the positions of words in s_g may also affect the attention scores. I may have missed something here but has any analysis on the relative positions has been done? E.g. \"man\" comes first or \"woman\" comes first. If not, it would be interesting to see if we need to randomly change the order of words in the tuples.\n\n[1] Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets\n", "summary_of_the_paper": "This paper proposes a new debiasing method for contextualized word embeddings, specifically for attention-based text encoders. At a very high level, the proposed method tries to calibrate the attention scores of words from different groups, e.g. to reduce gender bias, the method forces the model (text encoder) to allocate same attentions to word \"man\" and \"woman\". Experimentally, the paper also demonstrates relatively good results on both likelihood-based evaluation (StereoSet and Crows-Pairs) and inference-based evaluation (NLI).", "summary_of_the_review": "A paper with a novel method to reduce biases in text encoders and also reasonable empirical results. I would recommend to accept this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "At6SaAN2o8n", "reviewer": "Reviewer_7L6Q", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength \n- The debiasing based on attention is a new method.\n- The empirical results seem good for likelihood based and inference based evaluation. \n\nWeakness\n- While the work has its merit in decreasing the biases in some metrics, It is bad using those dataset as leaderboard without looking into them. [1] elaborates on the few problems with CrowS-Pairs and StereoSet. \n- In [2], it criticizes all the work focusing on word level debiasing and those are really not removing them, but hiding them. To me, this paper is no different from prior work in this aspect. Also given most of the differences are quite small, I am not sure whether the paper is making meaningful progress. \n- The proposed approach is quite limited in the setup. How does it handle multiple token words, different synonyms? Also, as described in section 5, it is not scalable for words that need not be debiased. \n- The proposed approach is also not very novel. It is an extension of counterfactural data augmentation in the attention with regularization with original attention. \n\n\n[1] https://aclanthology.org/2021.acl-long.81/\n\n[2] https://arxiv.org/abs/1903.03862\n\n", "summary_of_the_paper": "This papers propose a attention-based debiasing method for pretrained text encoders. The paper shows reduction in bias on few datasets and a small degradation to the GLUE benchmark. \n", "summary_of_the_review": "Overall, while there are merits in proposing attention based debiasing, the evaluation methodology is weak. It is another paper treating fairness issues as benchmarking without questioning on the actual effect, differences, and whether the datasets are meaningful in the first place. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "1W7BHPJkVzn", "reviewer": "Reviewer_CGM2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The method for bias mitigation introduced by the paper is intuitive\nand relatively simple. In particular, I liked the idea of sample\naugmentation as opposed to trying to identify parts of the original\ntext that refers to social groups and trying to alter the weights\nattending to those tokens, which is much higher to achieve.\n\nI have a couple of concerns:\n\n1. The method requires a teacher, unaltered model to be trained in\nparallel, which can be computationally involved. I would have liked to\nunderstand how much the performance of the model (perhaps on the GLUE\ntasks) is changed if such a teacher model is not used.\n\n2. I am concerned about the evaluation of bias. As recent studies\nshowed, intrinsic measures of bias do not necessarily correlate with\nbias measures on a concrete, down-stream task that can be used in an\napplication. https://arxiv.org/abs/2012.15859 The paper includes the\nNLI task, however, I perceive that task very similar to the intrinsic\ntasks. It doesn't provide insights in what it would happen wrt bias in\na down-stream task. I would have liked the evaluation of the model to\ninclude a task for which group annotations exist and the bias for the\noriginal and modified model are presented. For example, the Jigsaw\ndataset could be used for such an evaluation\n(https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification/data),\nor, for a smaller dataset, HateXplain could be used\n(https://github.com/hate-alert/HateXplain). Without such an evaluation\nfor an application oriented task, it is hard to understand whether the\nfew points gained by the new model will lead to significant bias\nimprovement in a downstream task.\n\nYou may want to take a look at this paper that lists some of the problems with benchmarks such as StereoSet:\nhttps://www.microsoft.com/en-us/research/uploads/prod/2021/06/The_Salmon_paper.pdf\n\nI am not sure if I understood the need to separate the original input\nfrom the artificially constructed one. Do you think that the model\nwill get confused if no separated is used and, thus, its semantic\nperformance would decrease?\n", "summary_of_the_paper": "This paper addresses potential biases introduced by attention models\nby re-weighing the attention weights. First, the paper provides a few\nexamples that demonstrate attention weights correlated with social\nstereotype (e.g., doctor attending to he and nurse attending to\nshe). Then, it proposes to reduce this type of bias by \"calibrating\"\nthe attention weights. To do so, each sample is augmented with pairs\nof words corresponding to the groups for which the model is intended\nto mitigate bias. The augmented samples are used in training, and,\nthus, each token in the input will attend to tokens in the augmented\nportion of the sample. The weights are changed such that the weights\nattending on the augmented, group-relevant tokens are similar. This\ntype of re-weighing can lead to some of the semantic meaning encoded\nin the network to change. To prevent this change, the attention\nweights corresponding to the tokens from the original sample are\nforced to follow the weights in an unaltered model that is trained in\nparallel. Last, the paper suggested also utilizing negative sampling:\nusing random words for augmentation and forcing all attention weights\nto follow the ones of the teacher.\n\nFor evaluation, the paper shows the performance of several\ntransformer-based models.  The evaluation of the method includes three\ndifferent benchmarks: two benchmarks that measure bias intrinsically\n(Crows-Pairs and StereoSet) and an NLI task designed for measuring\nbias. To make sure that the semantic strengths of the model are not\nlost, the models are also evaluated using the GLUE tasks. The model\nobtains competitive performance for the GLUE tasks, while reducing\nbias compared to original models and a couple of related works.", "summary_of_the_review": "The method presented in the paper is intuitive and simple, with some,\npotentially considerable computation overhead. While the semantic\nperformance is preserved and there are modest improvements on\nintrinsic measures of bias, the effects of the model on the bias\nresults of a downstream, application task are not addressed and,\nhence, not understood.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a debiasing technique that modifies a model's attention mechanism by equalizing attention across social groups.  The authors show that their approach (which is perhaps the first of its kind to look at transformer based models and debiasing instead of fixed word representations) work well in debiasing across certain social group indicators while maintaining overall performance.  However, there is disagreement between reviewers in terms of acceptance of the paper (especially Reviewer 7L6Q wants the paper to be rejected and points to recent critiques such as https://aclanthology.org/2021.acl-long.81.pdf that point out pitfalls with the benchmarks used in this paper).  I agree with said reviewer that a lot of these benchmarks are toy-ish and finding real impact of bias in NLP models is quite elusive.  Hence, I am recommending the paper be rejected for ICLR 2022 and the suggestions below be incorporated towards a better draft for the future.
Decision: Reject