Should We Attend More or Less? Modulating Attention for Fairness
Authors: Abdelrahman Zayed
Goncalo Mordido
Samira Shabanian
Sarath Chandar
Venue: COLM (2024)
Keywords: attention modulation, bias, fairness, intra-processing, language models, attention, gender, race, religion, sexual orientation
TL;DR: This paper presents a novel approach to improve fairness in language models using attention modulation.

Abstract: The advances in natural language processing (NLP) pose both opportunities and challenges. While recent progress enables the development of high-performing models for a variety of tasks, it also poses the risk of models learning harmful biases from the data, such as gender stereotypes. In this work, we investigate the role of attention, a widely-used technique in current state-of-the-art NLP models, in the propagation of social biases. Specifically, we study the relationship between the entropy of the attention distribution and the model's performance and fairness. We then propose a novel method for modulating attention weights to improve model fairness after training. Since our method is only applied post-training and pre-inference, it is an intra-processing method and is, therefore, less computationally expensive than existing in-processing and pre-processing approaches. Our results show an increase in fairness and minimal performance loss on different text classification and generation tasks using language models of varying sizes.

Published as a conference paper at COLM 2024
Should We Attend More or Less? Modulating Attention for Fairness
Abdelrahman Zayed Gonçalo Mordido Samira Shabanian / Sarath Chandar 1,3,55
"Chanaar Research Lab
Mila Quebec Al Institute
"Polytechnique Montreal
'Independent Researcher
Canaaaa CIFAR Al Chair
(zayedabd, goncalo-filll filpppe torcato-mordido, sarath. chaptahhhacaaaraemamaa Ja la quebec, @gmail. camei.lccoommm com
(s. shabanian @
Abstract
The advances in natural language processing (NLP) pose both opportu- nities and challenges. While recent progress enables the development of high-performing models for variety of tasks, it also poses the risk of models learning harmful biases from the data, such as gender stereo- types. In this work, we investigate the role of attention, widely-used technique in current state-of-the-art NLP models, in the propagation of social biases. Specifically, we study the relationship between the entropy of the attention distribution and the model's performance and fairness. We then propose novel method for modulating attention weights to im- prove model fairness after training. Since our method is only applied post-training and pre-inference, is an intra-processing method and is, therefore, less computationally expensive than existing in-processing and pre-processing approaches. Our results show an increase in fairness and minimal performance loss on different text classification and generation tasks using language models of varying sizes. WARNING: This work uses language that is offensive.
1 Introduction
Advancements in transformer-based pre-trained language models, such as BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), GPT-2 (Radford et al., 2019), GPTT3 (Brown al., 2020) and XLNet (Yang et al., 2019), have led to the emergence of new state-of-the-art models in a variety of applications, including but not limited text summarization (Liu et al., 2022; Mordido & Meinel, 2020), sentence classification (Wang et al., 2018), question answering (Rajpurkar et al., 2018; 2016), and information extraction (Li et al., 2020a;b). Despite their success, recent studies (Nadeem et al., 2021; Meade et al., 2022; Zayed et al., 2024a) have demonstrated that these models also exhibit harmful biases based on factors such as gender, race, sexuallorientation, and religion. These biases pose significant challenge in deploying machine learning models in real-world applications, as they can result in discriminatory outcomes. For example, it would be ethically questionable to deploy machine learning model for resume filtering if it is known that the model would discriminate against certain applicants based on their gender.
Several techniques have been proposed to address gender bias in language models, which may be broadly classified into pre-processing (Lu et al., 2020; Hall Maudslay et al., 2019; De-Arteaga et al., 2019; Dixon et al., 2018), in-processing (Garg et al., 2019; Zhang et al., 2020; Attanasio et al., 2022; Kennedy et al., 2020), and post-processing methods (Wei al., 2020).
More recently, a new category of debiasing methods has been introduced: intra- methods (Savani et al., 2020). These methods involve modifying the model weights after
Work partially done during an internship Microsoft Research.
1

---

Published as a conference paper at COLM 2024
training but before inference and, therefore, significantly less costly than pre-processing and in-processing methods. In contrast to post-processing methods, which are primarily designed for tabular datasets, processnos methods have the advantage of not being dataset dependent. In this work, we propose a new intra-processing method to address gender bias in language models.
The work by Attanasio et al. (2022) proposed an in-processing bias mitigation method called entropy attention-based regularization (EAR), which improves model fairness by maximizing the entropy of the attention weights distribution during training. The authors argue that maximizing the entropy of the attention map distribution leads to the model attending to a broader context within the input sentence, preventing it from relying on few stereotypical tokens, which results in a fairer model. In this work, we study the effect of attention distribution entropy on both fairness and performance and find that the relationship between the model's bias and the entropy of its attention distribution is both dataset and centeecturrdeeenddent This suggests that some of the previous findings by Attanasio et al. (2022) may not be general.
Hence, we propose to modulate the attention distribution entropy after training, rather than maximize it during training. Our novel attention entropy modulation, called entropy-based attention temperature scaling (EAT), applies scaling factor to modulate the entropy of the attention map post-training. In the end, we are able to efficiently improve fairness with minimal performance loss. Our contributions may be summarized as follows:
1. We study the effect of modulating the entropy of the attention distribution on gender bias and performance in BERT and RoBERTa models fine-tuned on three text classification datasets.
2. We propose novel intra-processin bias mitigation method that modulates the at- tention distribution entropy in text classification with less than 3.5% degradation in performance. To the best of our knowledge, this is the first intra-processing method bias mitigation method in NLP, offering a computationally efficient alternative to existing methods.
3. Our method outperforms existing intra-processing debiasing methods originally proposed for image data and migrates such efforts to the NLP domain. We also outperform other pre/ in-processing methods, including EAR.
4. We combine our method and the previous intra-processing baselines with five popular
S in-processing and pre-processing bias mitigation methods and show that such method combinations always improve fairness.
5. We show that our method generalizes to different forms of social bias even when the hyperparameters are exclusively tuned to mitigate gender bias.
6. Finally, we show that our method extends beyond text classification to address social biases in text generation using GPT-Neo.
2 Related work
We will start by describing existing techniques for mitigating bias that are applied before, during, after model training. Additionally, we will delve into previous studies that have proposed modifying the attention map to improve performance.
2.1 Gender bias mitigation methods
Gender bias mitigation methods may be broadly classified into two categories: intrinsic and extrinsic approaches. While intrinsic methods (Adi et al., 2017; Hupkes et al., 2018; Conneau et al., 2018; Tenney et al., 2019; Belinkov & Glass, 2019) focus on analyzing the embedding representations assigned to gender tokens by the model, extrinsic methods (Sennrich, 2017; Isabelle et al., 2017; Naik et al., 2018) rely on the model's predictions to determine if different genders achieve similar predictions under the same context. In this paper, we focus on extrinsic bias mitigation methods, as they more accurately reflect the applicability and performance of the model in real-world situations.
2

---

Published as a conference paper at COLM 2024
Pre-processing methods for mitigating gender bias involve modifying the training data to improve model fairness. One common method is counterfactual data augmentation (CDA) (Lu et al., 2020), which adds counterfactual examples with flipped gender words to the training set. However, this leads to longer training times and meaningless examples. Hence, counterfactual data substitution (CDS) (Hall Maudslay et al., 2019) swaps the gender words in the training set with probability of 0.5, resulting in a dataset of the same size. More recently, Zayed et al. (2023) proposed a recipe that combines the counterfactual examples with the original examples while excluding the stereotypical ones. Moreover, gender blindness (De-Arteaga et al., 2019) removes all gender words from the dataset, preventing the model from associating any label with a specific gender. Lastly, data balancing (Dixon et al., 2018) adds new examples only for under-represented groups in the dataset.
In-processing bias mitigation methods aim to reduce bias during training by adding auxiliary loss terms to the model. One example is counterfactual logit pairing (Garg et al., 2019), which penalizes the model if it makes different predictions for the same input after altering sensitive attributes such as gender words. Another method by Kennedy et al. (2020) adds penalty term based the difference in output logits when sensitive attributes are present or absent. Instance weighting (Zhang et al., 2020) multiplies the loss by factor greater than 1 for stereotypical sentences to penalize the model more for misclassifying them, and attention- based regularization (Attanasio et al., 2022) maximizes the model's 's attention distribution entropy during training to improve fairness. MABEL (He et al., 2022) proposes adding an auxiliary loss to minimize the cosine similarity between the original and gender-swapped vector representation for each example.
While relatively less explored, post-processing bias mitigation methods modify the predic- tions of biased model and generate a new set of less biased predictions. For example, Wei et al. (2020) address this problem by formulating it as convex optimization problem with fairness constraints. processpogss debiasing methods have been introduced to reduce the biases of image processing models as a new category of techniques that lie between in-processing and post-processing methods. Examples include applying random perturba- tions to model weights, modifying the weights of a given layer, or performing adversarial fine-tuning to reduce model bias (Savani et al., 2020). Recent work prunes the attention heads with the highest contribution to bias intra-processing debiasing method (Zayed et al., 2024b).
2.2 Attention modulation
There has been an ongoing debate in the literature regarding the interpretability of the attention mechanism (Jain & Wallace, 2019; Wiegreffe & Pinter, 2019; Serrano & Smith, 2019; Moradi et al., 2019; Mohankumar et al., 2020). Despite this, several works have demonstrated that modulating the attention map values with prior knowledge improves model performance on downstream tasks. For example, in document summarization, Cao & Wang (2021) proposed content selection method that detects tokens that are irrelevant at inference time, masking them from the attention map. Moreover, Cao & Wang (2022) proposed to increase the attention weights between tokens that lie within the same section in the document. Furthermore, Zhang et al. (2022) applied temperature scaling during inference to the attention map, encouraging the model to focus on broader context to improve the quality of the summarization.
In the context of text classification, Li et al. (2021) proposed to use a local attention map, limiting the attention based on the dependency parse tree to make the model more syntax- aware. Additionally, in language generation, modulating the attention weights has been applied both during training and inference to enhance fluency and creativity (Dong et al., 2021) by updating the attention weights between tokens using a learned or pre-defined reweighting function. In machine translation, Yin et al. (2021) employed group of freelance translators to identify the words they used to translate each word in a given output sequence. The authors then added an auxiliary loss term to encourage the model to pay more attention to these words, resulting in a performance improvement. Moreover, Lu et al. (2021) proposed to increase the model's attention to essential words by measuring the performance drop
3

---

Published as a conference paper at COLM 2024
after the removal of such words, encouraging the model to attend more to the words that led to the most significant drop.
3 Self-attention
One of the key factors contributing to the success of transformer models (Vaswani et al., 2017) is the usage of self-attention (Bahdanau et al., 2015) to compute the representation of each token in various layers of the model. In particular, the representation of the 7th token is contingent upon its relevance to all other tokens within the sentence. This relevance between tokens and j is referred to as the attention from token to token j. Hence, if the maximum sentence length is T, the attention map will be T x T matrix representing the attention each token to all other tokens. The attention map is calculated as:
QKT du
Attention Q K, V) sootmax
V
(1)
where Q E I Txd,, K E RTx,, and V E RTxd, are the query, key, and value matrices with embedding dimensionalities dg. d2 and da respectively. The computation of these matrices is described as: Q = xwe; K = xww; V = xW where X E RTxd is the input vector of maximum length T and embedding dimentionality d, while w E Rdxd,, w e Rixd, and WV E Rdxda are three matrices of learnable parameters.
3.1 Attention entropy
The softmax function used to calculate the attention map in Eq.(1) ensures that the attention values from any token to all other tokens within the sentence are non-negative and sum to one, and therefore may be treated probabilities. The larger the attention values in this distribution between tokens and j, the greater the correlation between them. We follow the same procedure as Attanasio et al. (2022) to calculate the entropy of the attention distribution.
Considering the attention values At,bi, between tokens i and j in the head h of layer I, we first average the attention weights over all heads in the I-th layer: d'u En *1,iij- Subsequently, softmax function is applied to ensure that the resulting values form
4uj The entropy, first introduced by Shannon (1948), is E
probability distribution: ati
defined by H o R,, log atjj where T, represents the actual length of the sentence, excluding any padding tokens. We obtain the average entropy within a sentence in layer as H' + - H/,, with the overall attention entropy being computed by summing the entropy across all model layers: H = E
4 Attention entropy modulation
Attanasio et al. (2022) suggested that memorization encourages models to focus on stereo- typical tokens shortcut to solve the task, and proposed to maximize the entropy of the attention distribution. The intuition is to force the model to attend to a broader context, resulting in a less biased model. In our work, we challenge this hypothesis and demonstrate that model fairness may be improved not only by maximization but also by minimization of attention entropy. If there are stereotypical tokens in the narrower context, then attending to broader context is likely to improve fairness. However, if the narrower context is already devoid stereotypical tokens, then attending to a broader context could potentially expose the model to more bias. Hence, we posit that the relationship between attention entropy and bias both dataset and model dependent, and propose to perform attention entropy modulation, instead of maximization.
4

---

Published as a conference paper at COLM 2024
4.1 Entropy-based attention temperature scaling (EAT)
We propose a novel method for attention entropy modulation, which we term entropy-based attention temperature scaling (EAT). Our approach modulates the entropy of the model's attention maps by performing temperature scaling after training. The amount of scaling applied is controlled by hyperparameter that is chosen based on the validation set such that good trade-off between performance and fairness is achieved. A scaled attention map, i.e. after temperature scaling, is computed as (Hinton et al., 2015):
BOK T
Attention, (Q, V) softmax
V.
Note that this scaling is applied to all the attention layers of the model.
To gain a deeper understanding of the role of the temperature scaling factor & in modulating the attention map's s entropy, it is instructive to consider the cases where f is smaller and larger than 1. When
less than 1, the attention entropy increases since the attention map values are brought closer together prior to the application of the softmax function, resulting in a more uniform distribution. Indeed, as & reaches O, the values after the softmax closely resemble uniform distribution, which corresponds to the highest possible entropy. Conversely, when / is greater than 1, the attention entropy decreases since the difference between the largest value and the rest of the values in the attention map is amplified, resulting in less uniform distribution after the softmax. ß is significantly larger than 1, approach scenario where only the largest value will be attended to. These scenarios are illustrated in Figure
3 0
3
8 >>>1
Figure 1: An example showing the effect varying the temperature scaling factor / on the attention map's distribution. Note that & = 1 represents the unmodulated or original attention distribution.
Experiments
We will now provide detailed overview of the tasks, datasets, baselines, and evaluation metrics as well as the different setups used in our experiments.
5.1 Tasks and datasets
We performed our experiments on two distinct binary text classification tasks: sexism detec- tion and toxicity detection, where the objective is to train model to accurately differentiate between texts that are deemed sexist or toxic and those that are not, respectively. As defined by Dixon et al. (2018), toxic comment is one that prompts individual to disengage from discussion. All the datasets used in this study are English. We used the following datasets: Twitter dataset (Waseem & Hovy, 2016) with approximately 16, 000 tweets binarized into sexist and non-sexist, Wikipedia dataset (Dixon et al., 2018) with around 160, 000 comments categorized as toxic or non-toxic, and the Jigsaw dataset' with around 1.8 million examples binarized into toxic and non-toxic. Moreover, we evaluated the feasibility of extending our method to text generation using the bias in open-ended language generation dataset (BOLD) (Dhamala et al., 2021) with 23, 679 prompts referring to professions, genders, races, as well as religious and political groups.
Phttps://wwat http://wwwgll..../.ooo com/ htt.://sss..aaiiiieeaaaaaaaaaaaa

---

Published as a conference paper at COLM 2024
5.2 Baseline methods
We compare our method (EAT) with three other intra-processing methods originally pro- posed by Savani et al. (2020) for image data: random weight perturbation, layer-wise optimization, and adversarial fine-tuning. Moreover, we also use Six pre/ pre/in reeeii-processing gender bias mitigation methods: instance weighting (Zhang et al., 2020), data augmentation (Lu et al., 2020), data substitution (Hall Maudslay et al., 2019), gender blindness (De-Arteaga et al., 2019), MABEL (He et al., 2022), and entropy attention-based regularization (EAR) (Attanasio et al., 2022). We refer the reader to Section 2 for description of the methods.
5.3 Evaluation metrics
For text classification, we evaluate model performance using the area under the receiver operating characteristic curve (AUC). For text generation, perplexity (PPL. on Wikitext-2 is used to measure the language modeling ability (Merity et al., 2017). We also measure the performance on seven different downstream tasks: ARC (Chollet, 2019), HellaSwag (Zellers et al., 2019), MMLU (Hendrycks et al.), TruthfulQA (Lin et al., 2022), GSM8K (Cobbe et al., 2021), Winogrande (Sakaguchi et al., 2020), and DROP (Dua al., 2019).
To evaluate gender bias in text classification, we mainly use the demographic parity (DP) metric (Beutel al., 2017; Hardt et al., 2016; Reddy, 2022) calculated by:
DP 1 - lr( 1|z = 1) p(ÿ = 1/z = =0],
(3)
where ! represents the model's prediction and z E {0,1] denotes keeping or flipping the gender words in the sentence, respectively. The scale of demographic parity ranges from (least fair) (fairest). Our procedure for computing DP follows prior studies (Dixon et al., 2018; Park et al., 2018), where we use a synthetic dataset, the identity phrase templates test set (IPTTS), for measuring fairness. Additional fairness metrics are described in Appendix B. To evaluate other forms of social bias, we employ the pinned AUC equality difference metric (Dixon et al., 2018), which is defined as:
CIAUC- - -AAC-AUCI),
where AUC and AUC, refer to the model's AUC on the whole dataset and on examples referring to specific subgroup t (egg,, Christianity, Judaism, and Islam subgroups for religion bias), respectively. Lower values correspond to less bias.
5.4 Experimental details
For text classification, we trained BERT and RoBERTa base models for 15 epochs using cross-entropy loss on the Twitter and Wikipedia datasets and 4 epochs on the Jigsaw dataset. The performance, measured by the AUC, is in-line with the state-of-the-art results on the three datasets. The training, validation, and testing data were split in a ratio of 8:1:1, with the exception of the Wikipedia toxicity dataset, for which the split ratio used by Dixon al. (2018) was employed.
For text generation, we used GPT-Neo (Black et al., 2021) with 125M, 1.3B and 2.7B param- eters. All experiments were run five times using different random seeds. Our proposed method, EAT, uses a single hyperparameter, the temperature scaling factor B, which is selected based on the validation set. We adopted the same procedure to find the best hy- perparameters for our baselines. The criterion for determining the optimal value of B is to maximize the demographic parity (DP) while ensuring less than 3% degradation in valida- tion performance, through a search range of B E {0, 0.1, 10). Additional implementation details are provided in Appendix A. Our code will be made public?.
Improving fairness with attention entropy modulation. We will first investigate the relationship between attention entropy and gender bias by varying the tem- perature scaling coefficient B. Figure shows that as f decreases, attention entropy increases,
Experiment 1:
Phttps: //gi thub. com/ chandar- Lab/EEA

---

Published as a conference paper at COLM 2024
Attentonn entropyN)
Dataset Twitter
Wikipedla
Dataset
-25
-50
2
-10
-6
-30
Bert
Figure 2: Percentage of change in attention entropy, demographic parity (DP), and AUC of BERT and RoBERTa using different temperature scaling factors (ß) on three datasets, compared to the unmodulated model (i.e. 1). Higher DP values indicate fairer models.
The values of ß that are smaller or larger than correspond maximizing or minimizing the attention entropy, respectively. Best viewed in color.
Dataset
Model
Atten. entropy
Twitter
BERT
0.5
Maximization
RoBERTa
Minimization
Wikipedia
BERT
0.3
Maximization
RoBERTa
Minimization
Jigsaw
BERT
0.4
Maximization
RoBERTa
0.5
Maximization
Table 1: The & values yielding the highest improvement fairness, with less than 3% degrada- tion in validation AUC relative to the unmodulated model.
resulting in wider context being attended to. As f reaches 0, the models attend equally to all tokens, leading decrease performance and an increase in fairness. Conversely, increasing & decreases attention entropy, making the models attend to narrower context. Importantly, minimizing attention entropy leads to improvements in fairness compared to the baseline model, especially on Twitter and Wikipedia datasets using RoBERTa. A similar trend is also observed on the additional fairness metrics (see appendix B). Table presents the chosen based the validation dataset. We see that the choice of whether maximize or minimize attention entropy may vary depending on the dataset and model used.
Experiment 2: Comparing with EAR and other baselines, and generalizing other forms of social bias. Table 2 shows comparison between EAT and other pre /in-processing methods, showing its effectiveness in improving fairness on different models and datasets, with less than 3% degradation in AUC for all methods. More importantly, EAT outperforms EAR in all scenarios, which underlines the superiority of attention modulation attention maximization. Since intra-processing methods are only applied post-training, we also study their combination with different pre/ /ee/iinprocessnng baselines. In particular, we combine each intra-processing method with 5 distinct bias mitigation methods of the approaches outlined in Section 5.2 (we excluded MABLE for computational reasons), as well as no bias mitigation, on 3 datasets and 2 different models. In the end, this results in total of 36
7

---

Published as a conference paper at COLM 2024
Dataset
Model
Debiasing method
ADP (%)
BERT
Instance weighting (Zhang et al., 2020)
0.73+1.12
BERT
CDA (Lu et al., 2020)
1.8
BERT
CDS (Hall Maudslay et al., 2019)
1.70
BERT
Gender blindness (De-Arteaga et al., 2019)
0.75
BERT
MABEL (He et al., 2022)
1.15
BERT
EAR (Attanasio al., 2022)
3.30
Twitter
BERT
EAT (ours)
0.86
BERT
MABEL (He al., 2022) + EAT
1.10
RoBERTa
Instance weighting (Zhang et al., 2020)
0.32
RoBERTa
CDA (Lu et al., 2020)
1.96 1.37
RoBERTa
CDS (Hall Maudslay et al., 2019)
1.47
RoBERTa
Gender blindness (De-Arteaga et al., 2019)
1.57
RoBERTa
MABEL (He al., 2022)
RoBERTa
EAR (Attanasio al., 2022)
0.19
RoBERTa
EAT (ours)
0.94
RoBERTa
MABEL (He et al., 2022) + EAT
3.55 1.4
BERT
Instance weighting (Zhang et al., 2020)
0.44
BERT
CDA (Lu et al., 2020)
0
BERT
CDS (Hall Maudslay et al., 2019)
1.42
BERT
Gender blindness (De-Arteaga al., 2019)
0.58
BERT
MABEL (He et al., 2022)
1.3
BERT
EAR (Attanasio et al., 2022)
-1.46
Wikipedia
BERT
EAT (ours)
0.73
BERT
MABEL (He et al., 2022) EAT
0.70
RoBERTa
Instance weighting (Zhang et al., 2020)
1.84
RoBERIa
CDA (Lu et al., 2020)
1.92
RoBERTa
CDS (Hall Maudslay et al., 2019)
1.71
RoBERTa
Gender blindness (De-Arteaga et al., 2019)
0.27
RoBERTa
MABEL (He et al., 2022)
2.46 +.001
RoBERTa
EAR (Attanasio et al., 2022)
0.27 0.337
RoBERTa
EAT (ours)
0.77
RoBERTa
MABEL (He al., 2022) EAT
2.57 1.01
BERT
Instance weighting (Zhang et al., 2020)
0.05
BERT
CDA (Lu et al., 2020)
0.48
BERT
CDS (Hall Maudslay et al., 2019)
0.54
BERT
Gender blindness (De-Arteaga et al., 2019)
0.09
BERT
EAR (Attanasio al., 2022)
-0.88
Jigsaw
BERT
EAT (ours)
2.17
RoBERTa
Instance weighting (Zhang et al., 2020)
0.70
RoBERTa
CDA (Lu et al., 2020)
1.24
RoBERTa
CDS (Hall Maudslay et al., 2019)
1.00 1.79
RoBERTa
Gender blindness (De-Arteaga et al., 2019)
-0.02 + 0.76
RoBERTa
EAR (Attanasio et al., 2022)
0.41 1 0.93
RoBERIA
EAT (ours)
0.60 = 0.64
Table 2: A comparison between EAT and pre/in-processing methods in terms of percentage of change in demographic parity for the different models and datasets. Numbers are shown with one standard deviation. Our proposed EAT (i.e. attention modulation) outperforms EAR (i.e. attention maximization) in all scenarios. MABEL is a strong in in-processing baseline, so we show the result of its combination with our proposed EAT, which leads to more improvement in bias mitigation. We did not implement MABLE for the Jigsaw dataset for computational reasons.
scenarios per intra-processing method, and 900 different experiments. Figure illustrates the frequency with which each intra-processing method ranked first in terms of fairness across various forms of social bias, as measured by the pinned AUC equality difference (4). The results indicate that EAT outperforms the existing intra processin methods. The degradations in AUC performance of all methods were less than equal to 3,5%, except for random perturbation. Notably, the hyperparameter selection was conducted solely on gender bias, which shows EAT's ability generalize to other social biases.
8

---

Published as a conference paper at COLM 2024
Prompt: The United Nation of Islam African American...
Vanilla: terrorist group based the teachings of its founder, Mohammed. Random pert.: phenomenon that has infiltrated colleges campuses, racism TED ESV.
EAT: religious organization based in Los Angeles, California.
Prompt: Jane Cowl
American film and...
Vanilla: television actress best known for her roles Nurse Anne.
Random pert.: television actress and former beauty pageant titleholder who achieved prominence and name recogmition former model and actress.
EAT: television actress, known for her roles in television movies such Harsh Times.
Table 3: GPT-Neo continuations for BOLD prompts using the vanilla model, random perturbation, and our proposed EAT technique. Red indicates the model's bias.
in top fairness
20
10
EAT (ours)
Random
Layer-wise
Adv. fine-tuning
None
Age
Disability
Nationality
Religion
Figure 3: Comparing the social fairness of different cnntra-processingg methods in 36 scenarios by combining each method with various pre- pre-processing and in-processing methods using BERT and RoBERTa models three datasets.
Gmou - Gender
Group Political ideology
Group Professim
Gmou Pare
Group Religious idealogy
1.5
0.4
0.5
0.0
-5
-1.
07886910111123111
0708031013121311
077000910111 0708091011121314
0738091011121311
9708091011122131
1.88
2.78
Figure 4: Percentage of change in toxicity on BOLD dataset for different GPT-Neo sizes using EAT for different relative the unmodulated baseline model with B = 1.
40
Perplexit
20
10
0.8
1.0
1.2
1.4
1.38
2.7B
Figure Perplexity of EAT (solid) and random perturbation (dashed) on Wikitext-2 against using GPT-Neo with 1.3 and 2.7 billion parameters.
Experiment Extending EAT to text generation. We explore the applicability EAT to address various forms of social bias in text generation using BOLD framework. BOLD con- sists of thousands of prompts pertaining to five distinct groups. Following previous work (Dhamala et al., 2021), use the toxicity of both the prompt and the model's continuation as a proxy for its bias towards any particular group. Figure shows the percentage change in toxicity using EAT GPT-Neo for different groups, relative to the vanilla model. A

---

Published as a conference paper at COLM 2024
0.5
0.4
Performance
0.3
0.1
0.0
0.7
0.8
0.9
1.0
1.1
1.2
1.3
ARC
HellaSwag
MMLU
TruthfuiQA
Winogrande
GSMBK
DROP
Figure 6: Accuracy of EAT on 7 different downstream tasks using GPT-Neo with 125M parameters. The values for all tasks refer to accuracy scores, except for the DROP task which uses F1 score. We evaluate (-shot accuracy in all tasks, except for GSM8K where we compute the 5-shot accuracy. The table shows that attention entropy modulation (iee. maximization or minimization) in text generation does not result in performance degradation on downstream tasks.
qualitative comparison is shown in Table 3. Figures and illustrate the same comparison for performance. EAT reduces toxicity without sacrificing language modeling ability. Con- versely, random perturbation increases toxicity for all groups, with values falling outside the specified range, while substantially increasing perplexity. The percentages of increase in toxicity using random perturbation for gender, political, profession, race, and religion biases using GPT-Neo with 1.3 and 2.7 billion parameters are as follows: 3.2, 4.7, 4.2, 3.7, 4.8 and 1.6, 8, 2.2, 4.2, 0.4, respectively. We did not implement other intra-procsssing baselines due to resource constraints. In addition, EAT does not lead to substantial degradation on the downstream tasks used, while improving the performance in certain cases. More specifically, attention entropy minimization (B 1) leads to better performance ARC, HellaSwag, TruthfulQA, GSMSK, and Winogrande; while attention entropy maximization (B 1) improves performance on GSM8K and Winogrande.
Conclusion
In this work, we examined the impact entropy in the attention distribution on fairness and performance in different language models. Our results indicate that, in contrast with previous research (Attanasio et al., 2022), both attention entropy maximization and mini- mization may enhance fairness depending on the model and task at hand. With this in mind, we propose à computationally efficient and novel bias mitigation technique that modulates the entropy of the attention distribution after training and prior to inference. Our extensive results on both text classification and generation datasets show that we are able to improve fairness while maintaining most of the performance of the original biased model.
Acknowledgements
Sarath Chandar is supported by Canada CIFAR AI Chair and an NSERC Discovery Grant. This project is funded by the Microsoft Research and Mila partnership. The authors acknowledge the computational resources provided by the Digital Research Alliance of Canada. We thank Dhanya Sridhar, Sarthak Mittal, and Jules helpful feedback this project.
for their
10

---

Published as a conference paper at COLM 2024
Ethics statement
To determine the level of gender bias present within a model, we employed the widely-used IPTTS template, egg. Dixon et al. (2018); Park et al. (2018); Sun al. (2019); Kiritchenko & Mohammad (2018), which uses identical examples for different genders to measure the deviation in the model's predictions when gender-specific words are altered, as outlined in our experimental section. However, it is important to note that our approach is limited by the simplicity of the template, which may only accurately assess bias within the context of the examples provided in the template. Additionally, the template does not take into account non-binary gender identities. Furthermore, our use of demographic parity for fairness assessment is also limitation, as it quantifies bias based solely on the difference in means of the predictions for examples referring to different genders, and does not take into account the distribution of the predictions. Finally, it is also worth mentioning that while our work is intended to improve the fairness of NLP models, the proposed technique may also be used in the opposite manner. In other words, the principle of attention modulation used by our method could be used to increase model bias instead.
References
Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. In International Confer- ence on Learning Representations, 2017.
Giuseppe Attanasio, Debora Nozza, Dirk Hovy, and Elena Baralis. Entropy-based attention regularization frees unintended bias mitigation from lists. In Findings of the Association for Computational Linguistics: ACL 2022. Association for Computational Linguistics, 2022.
Dzmitry Bahdanau, Kyung Hyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, 2015.
Yonatan Belinkov and James Glass. Analysis methods in neural language processing: A survey. Transactions of the Association for Computational Linguistics, 2019.
Alex Beutel, Jilin Chen, Zhe Zhao, and Ed H Chi. Data decisions and theoretical implications when adversarially learning fair representations. Workshop on Fairness, Accountability, and Transparency in Machine Learning, 2017.
Sid Black, Gao Leo, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, March 2021. URL https://doi. org/10. /0/ 5081//22100/0/ zenodo. 5297715 If you use this software, please cite using these metadata
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvínd Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 2020.
Shuyang Cao and Lu Wang. Attention head masking for inference time content selection in abstractive summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp- 5008-5016, Online, June 2021. Association for Computational Linguistics.
Shuyang Cao and Lu Wang. Hibrids: Attention with hierarchical biases for structure-aware long document summarization. In Proceedings of the 60th Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Papers), pp. 786-807, 2022.
François Chollet. On the measure of intelligence. Xrv preprint ar xxXio:1911.01477, 2019.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiihhro Nakano, et al. Training verifiers to solve math word problems. Xix preprint ar arxi::2110.141 14168,,,, 2021.
11

---

Published as a conference paper at COLM 2024
Alexis Conneau, German Kruszewski, Guillaume Lample, Loic Barrault, and Marco Baroni. What you can cram into a single $&!#" vector: Probing sentence embeddings for linguistic properties. In Annual Meeting of the Association for Computational Linguistics, 2018.
Maria De-Artaaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. Bias in bios: A case study of semantic representation bias in high-stakes setting. In Conference on Fairness, Accountability, and Transparency, 2019.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL, pp. 4171-4186, 2019.
Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gupta. Bold: Dataset and metrics for measuring biases in open-ended language generation. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, Pp. 862-872, 2021.
Lucas Dixon, Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Measuring and mitigating unintended bias in text classification. In Conference on AI, Ethics, and Society, 2018.
Yue Dong, Chandra Bhagavatula, Ximing Lu, Jena D. Hwang, Antoine Bosselut, Jackie Chi Kit Cheung, and Yejin Choi. On-the-fly attention modulation for neural generation. In Findings of the Association for Compulational Linguistics: ACL-IJCNLP 2021, pp. 1261-1274, Online, August 2021. Association for Computational Linguistics.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and 2019.
Short Papers), pp.
Sahaj Garg, Vincent Perot, Nicole Limtiaco, Ankur Taly, Ed H Chi, and Alex Beutel. Coun- terfactual fairness in text classification through robustness. In Conference on Al, Ethics, and Society, 2019.
Rowan Hall Maudslay, Hila Gonen, Ryan Cotterell, and Simone Teufel. It's all in the name: Mitigating gender bias with name-based counterfactual data substitution. In Conference on Empirical Methods Natural Language Processing and International Joint Conference on Natural Language Processing, 2019.
Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. Advances in Neural Information Processing Systems, 2016.
Jacqueline He, Mengzhou Xia, Christiane Fellbaum, and Danqi Chen. Mabel: Attenuating gender bias using textual entailment data. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 9681-9702, 2022.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arxiv preprint arXivo1503.00551, 2015.
Dieuwke Hupkes, Sara Veldhoen, and Willem Zuidema. Visualisation and 'diagnostic classifiers' reveal how recurrent and recursive neural networks process hierarchical structure. Journal of Artificial Intelligence Research, 2018
Pierre Isabelle, Colin Cherry, and George Foster. A challenge set approach to evaluating machine translation. In Conference on Empirical Methods in Natural Language Processing, 2017.
12

---

Published as a conference paper at COLM 2024
Sarthak Jain and Byron C Wallace. Attention is not explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 3543-3556, 2019.
Brendan Kennedy, Xisen Jin, Aida Mostafazadeh Davani, Morteza Dehghani, and Xiang Ren. Contextualizing hate speech classifiers with post-hoc explanation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 544,, Online, July 2020. Association for Computational Linguistics.
Svetlana Kiritchenko and Saif Mohammad. Examining gender and race bias in two hundred sentiment analysis systems. In Joint Conference on Lexical and Computational Semantics, 2018.
Xiaoya Li, Jingrong Feng, Yuxian Meng, Qinghong Han, Fei Wu, and Jiwei Li. A unified MRC framework for named entity recognition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5849-5859, Online, July 2020a. Association for Computational Linguistics.
Xiaoya Li, Xiaofei Sun, Yuxian Meng, Junjun Liang, Fei Wu, and Jiwei Li. Dice loss for data-imbalanced NLP tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 465-476, Online, July 2020b. Association for Computational Linguistics.
Zhongli Li, Qingyu Zhou, Chao Li, Ke Xu, and Yunbo Cao. Improving bert with syntax- aware local attention. In Findings of the Association for Compulational Linguistics: ACL- IJCNLP 2021, PP. 645-653, 2021.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computa- fional Linguistics (Volume 1: Long Papers), pp. 3214-3252, 2022.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint xrXio:1907.11692, 2019.
Yixin Liu, Pengfei Liu, Dragomir Radev, and Graham Neubig, BRIO: Bringing order to abstractive summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Pp. 1890-2903, Dublin, Ireland, May 2022. Association for Computational Linguistics.
Kaiji Lu, Piotr Mardziel, Fangjig Wu, Preetam Amancharla, and Anupam Datta. Gender bias in neural natural language processing. In Logic, Language, and Security, 2020.
Yu Lu, Jiali Zeng, Jiajun Zhang, Shuangzhi Wu, and Mu Li. Attention calibration for transformer in neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1288-1298, 2021.
Nicholas Meade, Elinor Poole-Dayan, and Siva Reddy. An empirical survey of the effective- ness of debiasing techniques for pre-trained language models. In Annual Meeting of the Association for Computational Linguistics, 2022.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference Learning Representations, 2017.
Akash Kumar Mohankumar, Preksha Nema, Sharan Narasimhan, Mitesh M Khapra, Bal- aji Vasan Srinivasan, and Balaraman Ravindran. Towards transparent and explainable attention models. In Proceedings of the 58th Annual Meeting of the Association for Computa- fional Linguistics, pp. 2206-4216 2020.
Pooya Moradi, Nishant Kambhatla, and Anoop Sarkar. Interrogating the explanatory power of attention in neural machine translation. In Proceedings of the 3rd Workshop on Neural Generation and Translation, pp. 221-230, 2019.
13

---

Published as a conference paper at COLM 2024
Gonçalo Mordido and Christoph Meinel. Mark-evaluate: Assessing language generation ing population estimation methods. In International Conference on Computational Linguistics, December 2020.
Moin Nadeem, Anna Bethke, and Siva Reddy. StereoSet: Measuring stereotypical bias in pretrained language models. In Annual Meeting of the Association for Computational Linguistics and International Joint Conference on Natural Language Processing, 2021.
Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. Stress test evaluation for natural language inference. In International Conference on Computational Linguistics, 2018
Ji Ho Park, Jamin Shin, and Pascale Fung. Reducing gender bias in abusive language detection. In Conference on Empirical Methods in Natural Language Processing, 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language Models are Unsupervised Multitask Learners. OpenAl Blog, 1(8)9, 2019.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Conference Empirical Methods in Natural Language Processing, 2016.
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeling of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 784-789, Melbourne, Australia, July 2018. Association for Computational Linguistics.
Charan Reddy. Benchmarking bias mitigation algorithms in representation learning through fairness metrics. 2022.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 8732-8740. Association for the Advancement of Artificial Intelligence (AAAI), 2020.
Yash Savani, Colin White, and Naveen Sundar Govindarajulu. Intra-processing methods for debiasing neural networks. Advances in Neural Information Processing Systems, 33: 2798-2818 2020.
Rico Sennrich. How grammatical is character-level neural machine translation? Assessing MT quality with contrastive translation pairs. In European Chapter of the Association for Computational Linguistics, 2017.
Sofia Serrano and Noah A Smith. Is attention interpretable? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2931-2951,, 2019.
Claude Elwood Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3):379-422 423, 1948.
Tony Sun, Andrew Gaut, Shirlyn Tang, Yuxin Huang, Mai ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth Belding, Kai-Wei Chang, and William Yang Wang. Mitigating gender bias in natural language laturall processing: Literature review. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1630-1640, Florence, Italy, July 2019. Association for Computational Linguistics.
gender
Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT rediscovers the classical NLP pipeline. In Anmual Meeting of the Association for Computational Linguistics, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is All You Need. In Proc. of the Advances in Neural Information Processing Systems (Neurips), pp. 1998-6008, 2017.
14

---

Published as a conference paper at COLM 2024
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding, In EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 2018.
Zeerak Waseem and Dirk Hovy. Hateful symbols or hateful people? Predictive features for hate speech detection on Twitter. In NAACL Student Research Workshop, 2016.
Dennis Wei, Karthikeyan Natesan Ramamurthy, and Flavio P Calmon. Optimized score transformation for fair classification. Proceedings of Machine Learning Research, 108, 2020.
Sarah Wiegreffe and Yuval Pinter. Attention is not not explanation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 11-20, 2019.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. XLNet: Generalized autoregressive pretraining for language understanding. Advances in Neural Information Processing Systems, 2019.
Kayo Yin, Patrick Fernandes, Danish Pruthi, Aditi Chaudhary, André FT Martins, and Graham Neubig. Do context-aware translation models pay the right attention? In Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), PP. 788-801, 2021.
Abdelrahman Zayed, Prasanna Parthasarathi, Goncalo Mordido, Hamid Palangi, Samira Shabanian, and Sarath Chandar. Deep learning on healthy data diet: Finding important examples for fairness. In AAAI Conference on Artificial Intelligence, 2023.
Abdelrahman Zayed, Goncalo Mordido, Ioana Baldini, and Sarath Chandar. Why don't prompt-based fairness metrics correlate? arXiv preprint xrXio:2466005911 2024a.
Abdelrahman Zayed, Goncalo Mordido, Samira Shabanian, Toana Baldini, and Sarath Chandar. Fairness-aware structured pruning in transformers. In AAAI Conference on Artificial Intelligence, 2024b.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791-4800, 2019.
Guanhua Zhang, Bing Bai, Junqi Zhang, Kun Bai, Conghui Zhu, and Tiejun Zhao. Demo- graphics should not be the reason of toxicity: Mitigating discrimination in text classifi- cations with instance weighting. In Annual Meeting of the Association for Computational 2020.
Linguistics, 2020.
Shengqiang Zhang, Xingxing Zhang, Hangbo Bao, and Furu Wei. Attention temperature matters in abstractive summarization distillation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 127-141, 2022.
A Implementation details
This section provides the implementation details regarding the selection of hyperparraeeters, computational time, infrastructure used, dataset imbalance, number of model parameters, text generation configurations, and the packages employed for the baselines described in the paper.
15

---

Published as a conference paper at COLM 2024
A.1 Hyperparameter selection
The entropy attention regularization method (EAR) (Attanasio et al., 2022) has one hyperparameter a, which regulates the trade-off between the cross-entropy loss and the entropy maximization loss. We adopt the same pattern for identifying the optimal values of B, but with wider search space: {10 6, 6,10-5, ,.., 1, 10, 1003. We note that the used a range is also wider than the one used in the original work by Attanasio et al. (2022) to ensure a fair comparison.
Table 4 presents the optimal values of ß selected in Experiment& 2. A thorough analysis of the results reveals that the fairness of the model is depends on the dataset and architecture used, with the model's fairness improving through either maximization attention entropy. Furthermore, the results indicate that the combination of pre-processing and in-processing techniques with EAT also plays a role in determining the optimal value of p. Specifically, when using the entropy maximization baseline (EAR), the optimal value of B was found to be 10 for RoBERTa on the Wikipedia dataset. Given that values of β that are larger than 1 minimize the attention entropy, this result supports the conclusion that attention maximization was not an appropriate choice for this specific model and dataset. Our proposed method, EAT, effectively modulates attention to improve fairness.
minimization of
Dataset
Model
Pre/In-processing method
B
Attention entropy
BERT
Vanilla
0.5
Maximization
BERT
Instance weighting (Zhang et al., 2020)
0.5
Maximization
BERT
CDA (Lu al,, 2020)
0.5
Maximization
BERT
CDS (Hall Maudslay et al., 2019)
0.5
Maximization
BERT
Gender blindness (De-Arteaga et al., 2019)
0.5
Maximization
Twitter
BERT
EAR (Attanasio et al., 2022)
0.4
Maximization
ROBERTA
Vanilla
4
Minimization
RoBERTa
Instance weighting (Zhang et al., 2020)
4
Minimization
RoBERTa
CDA (Lu et al., 2020)
4
Minimization
RoBERTa
CDS (Hall Maudslay et al., 2019)
0.9
Maximization
RoBERTa
Gender blindness (De-Arteaga et al., 2019)
4
Minimization
RoBERTa EAR (Attanasio et al., 2022)
4
Minimization
BERT
Vanilla
0.3
Maximization
BERT
Instance weighting (Zhang et al., 2020)
0.3
Maximization
BERT
CDA (Lu et al., 2020)
0.3
Maximization
BERT
CDS (Hall Maudslay et al., 2019)
0.3
Maximization
BERT
Gender blindness (De-Artcaga et al., 2019)
0.3
Maximization
Wikipedia
BERT
EAR (Attanasio et al., 2022)
0.3
Maximization
RoBERTa
Vanilla
Minimization
RoBERTa
Instance weighting (Zhang et al., 2020)
Minimization
RoBERTa
CDA (Lu et al., 2020)
Minimization
RoBERTa
CDS (Hall Maudslay et al., 2019)
Minimization
RoBERTa
Gender blindness (De-Arteaga et al., 2019)
Minimization
RoBERTa
EAR (Attanasio et al., 2022)
10
Minimization
BERT
Vanilla
0.4
Maximization
BERT
Instance weighting (Zhang et al,, 2020)
0.4
Maximization
BERT
CDA (Lu et al., 2020)
0.4
Maximization
BERT
CDS (Hall Maudslay et al., 2019)
0.4
Maximization
BERT
Gender blindness (De-Arteaga et al., 2019)
0.4
Maximization
Jigsaw
BERT
EAR (Attanasio et al., 2022)
0.5
Maximization
ROBERTA
Vanilla
0.5
Maximization
RoBERTa
Instance weighting (Zhang et al., 2020)
1
None
RoBERTa
CDA (Lu et al., 2020)
0.5
Maximization
RoBERTa
CDS (Hall Maudslay et al., 2019)
0.8
Maximization
RoBERTa
Gender blindness (De-Arteaga et al., 2019)
0.5
Maximization
RoBERTa
EAR (Attanasio et al., 2022)
0.5
Maximization
Table 4: The values for the different models and datasets that yield the most substantial improvement in terms of demographic parity, while ensuring that the degradation in the validation AUC does not exceed 3% in comparison to the original biased model. The results were obtained by combining our method, EAT, with 5 different in-processing and pre-processing methods and no bias mitigation efforts (vanilla) on 3 datasets and 2 models, resulting in 36 different scenarios.
16

---

Published as a conference paper at COLM 2024
Dataset
Model
Debiasing method
Running time
BERT
Instance weighting (Zhang et al., 2020)
197%
BERT
CDA (Tuu et al., 2020)
285%
BERT
CDS (Hall Maudslay et al., 2019)
200%
BERT
Gender blindness (De-Arteaga al., 2019)
206%
BERT
EAR (Attanasio et al., 2022)
208%
Twitter
BERT
EAT (ours)
100%
RoBERTa
Instance weighting (Zhang et al., 2020)
197%
RoBERTa
CDA (Lu et al., 2020)
273%
RoBERTa
CDS (Hall Maudslay et al., 2019)
200%
RoBERTa
Gender blindness (De-Arteaga al., 2019)
198%
RoBERTa
EAR (Attanasio et al., 2022)
200%
RoBERTa
EAT (ours)
100%
BERT
Instance weighting (Zhang et al., 2020)
198%
BERT
CDA (Lu et al., 2020)
273%
BERT
CDS (Hall Maudslay al., 2019)
200%
BERT
Gender blindness (De-Arteaga et al,, 2019)
198%
BERT
EAR (Attanasio et al., 2022)
203%
Wikipedia
BERT
EAT (ours)
100%
RoBERTa
Instance weighting (Zhang al., 2020)
199%
RoBERTa
CDA (Lu et al., 2020)
280%
RoBERTa
CDS (Hall Maudslay et al., 2019)
200%
RoBERTa
Gender blindness (De-Arteaga et al., 2019)
203%
RoBERTa
EAR (Attanasio et al., 2022)
203%
RoBERTa
EAT (ours)
100%
BERT
Instance weighting (Zhang et al., 2020)
198%
BERT
CDA (Lu al., 2020)
255%
BERT
CDS (Hall Maudslay et al., 2019)
200%
BERT
Gender blindness (De-Arteaga al., 2019)
198%
BERT
EAR (Attanasio et al., 2022)
202%
Jigsaw
BERT
EAT (ours)
100%
RoBERTa
Instance weighting (Zhang et al., 2020)
197%
RoBERTa
CDA et al., 2020)
237%
RoBERTa
CDS (Hall Maudslay et al., 2019)
200%
RoBERTa
Gender blindness (De-Arteaga al., 2019)
198%
RoBERTa
EAR (Attanasio et al., 2022)
200%
RoBERTa
EAT (ours)
100%
Table 5: A comparison between the running time of EAT and other pre-processing and in-processing methods relative to the vanilla model with debiasing. The total running time for any method is calculated as the time to train the biased model plus the time to train the debiased model, to compute the percentage of change performance and fairness. EAT's running time the same as the vanilla model without debiasing since the temperature scaling does not introduce additional time overhead.
A.2 Packages used
To implement the baselines for counterfactual data augmentation (CDA) (Lu et al., 2020), counterfactual data substitution (CDS) (Hall Maudslay et al., 2019), and gender blindness (De-Arteaga et al., 2019), it is essential to accurately detect gender-specific words for modifi- cation removal. We used the publicly available gender-bender Python package² for this purpose. This package provides a comprehensive list of gender-specific words and their corresponding alternatives, which enabled us to effectively implement the aforementioned methods. We used the detoxify library* for measuring toxicity.
9https://ww.iithhe www.b..ttttuu.....m comm/arreettRRggnnenee
17

---

Published as a conference paper at COLM 2024
A.3 Number of trainable parameters
In text classification, our experiments were conducted on BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) base models, which possess 110 and 125 million trainable param- eters, respectively. As for text generation, we used GPTT-Neo (Black et al., 2021) with 1.3 and 2.7 billion parameters.
A.4 Infrastructure used
We used a single NVIDIA A100-SXM4-40GB GPU for our experiments.
A.5 Running time
The computational time for each experiment is proportional to the size of the corresponding dataset. Using a single GPU, the running time for the vanilla model without debiasing was approximately 4 hours for Twitter and BOLD frameworks, whereas was 12 and 24 hours for the Wikipedia and Jigsaw datasets, respectively.
We also report the computational time for different debiasing methods. EAT reduces bias using temperature scaling factor in the attention map after the model is trained. This means that our method does not require the model to be re-trained find new of weights. In contrast, pre-processing and in-processing debiasing methods require training the model from scratch with alterations either in the training data (for pre-processoe pre-processing) or the objective function (for in-processing). Table 5 shows how much extra time each debiasing method takes compared to the vanilla model. Specifically, EAT's extra time is negligible because it simply involves adding temperature scaling factor to the attention map, hence be reported as 100% of the vanilla model's 'ss time. On the other hand, the rest the baselines have over 100% values.
A.6 Dataset imbalance
The percentage of examples with positive labels in the Twitter, Wikipedia, and Jigsaw datasets are 20.29%, 9.62%, and 5.98%, respectively.
A.7 Decoding configurations for text generation
We applied the following configurations for the text generation using GPT-Neo used in the BOLD experiments:
The maximum allowed tokens for generation, excluding the prompt tokens is 50 tokens.
The minimum required tokens for generation, without considering the prompt tokens is 0 tokens.
We employed sampling, instead of using greedy decoding.
No beam search was utilized.
B Results on additional fairness metrics
We present supplementary results that demonstrate the effectiveness of our proposed method, using two additional fairness metrics: equality of odds (EqOdd) and equality of opportunity (Beutel et al., 2017; Hardt et al., 2016). EqOdd is computed by first calculating the equality of opportunity for y (EqOppl):
EqOpp1 = 1 1 1pl(pp = 1/2 1,y = 1)
55)
- -p( = 1/2 = 0,y 1)/,
18

---

Published as a conference paper at COLM 2024
and y 0 (EqOppO):
EqOppO = 1 - 1p(0 = 1|2 1, -p 1/2 y,y 00],
(6)
and computing the average:
EqOdd 0.5 x (EqOpp1 + EqOpp0)
Wikipedia
Josaw
0.00
A Ea0ppatN
-0.01
0.25
-0.02
-0.50
2
Eqpo0199
Beft
Roborta
Figure 7: Equality of opportunity (5) (6) and equality odds (7), using different temperature scaling factors (B) on the different models and datasets. The values of ß that are smaller or larger than correspond maximizing or minimizing the attention entropy, respectively.
noteworthy that EqOpp1 and EqOppO assess the model's fairness on distinct sets of examples; specifically, EqOpp1 measures fairness on examples with y = 1, while EqOpp0 measures fairness on examples with y = 0. EqOdd is simply the average of these two metrics. Figure 7 illustrates that both EqOpp1 and EqOdd concur with the DP metric and exhibit similar trends when attention modulation is applied. However, the EqOpp0 fails to improve the Jigsaw dataset which suggests that the model's bias is primarily concentrated in the = examples (which are labeled as toxic), and thus modulating the attention map improves fairness in this group of examples, but not in the y y= 0 examples. Our experiments in the main paper focus on the DP metric as it reflects the model's bias in both the y 0 and y examples, which is also in agreement with the EqOdd metric.
C Additional results
In this section, we provide additional results on the bias text classification and generation models.
C.1 Qualitative results
Table shows the tokens with the highest attention weights in BERT and RoBERTa models the Wikipedia toxicity detection task using different debiasing methods. EAT puts the highest attention weights on the tokens that are more relevant to the task, as opposed to the vanilla model and EAR, which focus more on gender tokens.
19

---

Published as a conference paper at COLM 2024
C.2 Quantitative results
Our bias assessment uses the toxicity in the model's continuation as a proxy for its bias towards different groups. This is in line with the original BOLD paper (Dhamala et al., 2021). Figures 8 and 9 show a detailed comparison for the percentage of change in toxicity when using EAT with & 0.9 and random perturbation using GPI-Neo with 1.3 and 2.7 billion parameters, respectively. We used the detoxify library for measuring toxicity. The results show that EAT is more effective in producing less toxic output with an almost negligible decrease in the language modeling ability, shown in Figure 5.
Input
Model
Method
Token with highest att.
Hello Dan, please fix LanguageRo phpp first, it has explicit references to Wikipedia.
Vanilla
Dan
BERT
EAR
Dan
EAT
Wikipedia
Emilia, do you find more inform- -ation about this Swedish diva? If you do so, help complete this article!
Vanilla
Emilia
RoBERTa
EAR
Emilia
EAT
article
Table 6: Tokens with the highest attention using BERT and RoBERTa on the toxicity detection
task using Wikipedia dataset for different bias mitigation methods.
20

---

Published as a conference paper at COLM 2024
Amerkonnacco accrs
Amerean, umereenyetssree yerwore
wightwing
drmorary
Ierd
metilarring pocupetore
SeARO occupetior
conoucer accupetiors
Loccpptarrs
enci occupedora
Supg
amtetaier porupatfore
pocupedirn
nure no crecabiee
portecien, frier yooe
orginasing archcses
occupattors
treain pesoond
thee
pecupation
rs kayy Induttyy pccupetore
Mixun Ametcars
Exppoin_Ameion Americrs
Jndsen
dhiiiiaity
tudii
hadu pm
Inaishys
Figure 8: The percentage of change in toxicity when using random perturbation and EAT with B 0.9 for GPT-Neo with 1.3 billion parameters, compared to the vanilla model. The comparison is on different subgroups that belong to professions, genders, races, as well as religious and political groups.
21

---

Published as a conference paper at COLM 2024
Amerkonnacce
Amer uesreee yerwore
wighiwing
drmorary
Ierd
metilarting pocupetore
SeARn occupetior
coroucer accupetiors
Loccuplarrs
Supg
amtetaier porupattore
enci occupedors
pocupedirs
nure crecatiee
portecien, frier yooe
orginasing archcses
occupattors
Ireuin pesoond
thes
pecupation
rs kaay Induttyy pocupetore
Ametcars
Judsee
diiiitaity
tudiisn
haduan am
Bavg taalshyli
Figure 9: The percentage of change in toxicity when using random perturbation and EAT with B 0.9 for GPT-Neo with 2.7 billion parameters, compared to the vanilla model. The comparison is on different subgroups that belong to professions, genders, races, as well as religious and political groups
22


Reviews:
[{"review_id": "dPdf6sq6UW", "reviewer": "Reviewer_q9VN", "summary": "This paper focuses on minimizing social bias in language models. Specifically, it builds on prior work by Attansio et al., (2022) and attempts to provide an alternative solution to building an accurate language model that also improves fairness metrics. Attansio et al. posit that maximizing the entropy of the attention weights distribution during fine-tuning leads to less biased models (called EAR). This paper argues that this is not necessarily always true; instead it proposes to modulate the attention entropy using a hyperparameter (temperature scaling factor), which could lead to either an increase of decrease in entropy, arguing that the modulation is a factor of the dataset and architecture (henceforth, EAT). \n\nThis hypothesis is experimentally validated on three classification datasets (Wikipedia, toxic v/s non-toxic classification; Jigsaw, toxic v/s non-toxic classification; Twitter, sexist v/s non-sexist comment classification) and one open ended generation dataset (BOLD). A variety of alternative and previously proposed approaches are considered as baselines. The quality of the models are measured by AUC for classification and perplexity for open-ended generation, while fairness is measured using Demographic Parity for gender bias and pinned AUC equality difference for other biases (e.g. religion). Findings reveal that the choice of the temperature scaling factor can indeed affect fairness metrics on classification tasks, with EAT consistently improving these metrics compared to EAR. EAT more frequently leads to improved fairness metrics compared to other baselines as well.  On text generation, EAT outperforms the random perturbation baseline on both quality and fairness metrics.\n\nOverall, I like that this work proposes a post-training (or what is called as *intra-processing* in this paper) approach to mitigating fairness issues, compared to Attansio et al., which requires fine-tuning the underlying model. This makes the approach fairly easy to try. I also appreciate that this work does not limit itself to classification tasks. With the direction the community is heading in, it is paramount to measure and mitigate biases in language modeling tasks as well. Finally, I also appreciate that this work attempts to provide a generalization to a prior, well-recognized solution to the problem (i.e. entropy maximization is not sufficient, and modulating the entropy is a more general solution)\n\nHowever, I do think this work is not technically exciting as it proposes a straightforward application of temperature scaling, which has been used in knowledge distillation, calibration of neural networks [1], etc. Another drawback of this approach is that it requires a per dataset approach to mitigating bias (by identifying the right temperature scaling parameter). Finally, I also think that this paper is missing some side-by-side comparison of examples where EAT improves upon EAR. In the introduction to section 4, the authors argue that *\"If there are stereotypical tokens in the narrower context, then attending to a broader context is likely to improve fairness. However, if the narrower context is already devoid of stereotypical tokens, then attending to a broader context could potentially expose the model to more bias\".* I think having some examples that actually validate this will strengthen the paper. \n\n\n[1] - https://arxiv.org/pdf/1706.04599", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thank you for your time in providing a thorough and complete review. We appreciate your interest in understanding all the details in our work and address your main points below.**\n\n\n>The temperature scaling hyperparameter indicates that this approach needs to be configured for different data distributions/architectures.\n\nWe agree that one of the limitations of our method is that it assumes having access to validation data from the same distribution as the test data to choose the optimal temperature scaling.\n\n>The technical novelty in this paper is limited as the approach is an application of temperature scaling\n\nWe agree that the idea of temperature scaling exists in the literature, as the reviewer rightly pointed out. The main contribution of our work is challenging the existing claim that maximizing attention distribution entropy always improves fairness, by showing empirically and conceptually that this is not a general observation. \n\n>Qualitative insights are completely missing.\n\nThanks for pointing this out. Table 3 in Section 5.4 provides some qualitative results, but we will add more examples to strengthen the paper, as shown below.\n\n\nFor toxicity detection using BERT and Input *“Hello Dan, please fix LanguageRo.php first, it has explicit references to Wikipedia.”*:\n\n| **Method** | **Token with highest attention** |\n|:---:|:---:|\n| Vanilla | Dan |\n| EAR | Dan |\n| EAT | Wikipedia |\n\nUsing RoBERTa and Input *“Emilia, do you find more information about this Swedish diva? If you do so, help complete this article!”*:\n\n| **Method** | **Token with highest attention** |\n|:---:|:---:|\n| Vanilla | Emilia|\n| EAR | Emilia|\n| EAT | article|\n\n>How confident are you that a single bias/fairness metrics supports the findings in this paper\n\nIn line with related works [1-3], we use four bias assessment metrics for text classification. For gender bias, we use demographic parity, equality of odds, and equality of opportunity; and for general social biases, we use pinned AUC. Due to space limitations, some results are shown in Appendix B. We also use BOLD benchmark for bias assessment in text generation.\n \n[1] Reddy et al. Benchmarking bias mitigation algorithms in representation learning through fairness metrics. NeurIPS Datasets and Benchmarks Track, 2021.\n\n[2] Shen et al. Optimising Equal Opportunity Fairness in Model Training. NAACL, 2022\n\n[3] Dixon et al. Measuring and mitigating unintended bias in text classification. AIES, 2018.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* This paper proposes an approach to mitigating bias that does not require fine-tuning of the underlying model\n* Experiments are not restricted to classification; open-ended text generation results indicate that this method is broadly applicable to other tasks as well.\n* Provides an argument that improves upon prior work by making it more general.", "reasons_to_reject": "* The temperature scaling hyperparameter indicates that this approach needs to be configured for different data distributions/architectures, which makes it less broadly useful, when test data distribution is unknown. \n* The technical novelty in this paper is limited as the approach is an application of temperature scaling\n* Qualitative insights to back up quantitative results are completely missing.", "questions_to_authors": "* Citations are incorrectly parenthesized throughout the paper, and this needs fixing [e.g. intra-processing methods Savani et al. (2020) --> intra-processing methods (Savani et al., 2020)].\n* Why did you choose the bias metrics that you did in section 5.3? How confident are you that a single bias/fairness metrics supports the findings in this paper? If other fairness metrics are considered, do you expect the findings in this work to generalize?", "extra_scores": {}}, {"review_id": "PDrRhlG5ai", "reviewer": "Reviewer_czv4", "summary": "This paper proposes a novel intra-processing method, Entropy-based Attention Temperature Scaling (EAT), to modulate attention entropy for improving fairness in NLP models. The problem is that transformer models memorize stereotypical tokens. EAT does temperature scaling after training. It adds noise to the softmax in the attention mechanism. \n\nThe study uses three datasets: Twitter, Wikipedia, and Jigsaw, all in English, for two text classification tasks,  sexism and toxicity detection. Performance goes down in the overall performance, but it reduces disparities in AUC between messages that refer to various groups.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thank you for your detailed review and comments. We are encouraged that you find our work interesting, empirically useful, and computationally efficient. We address your main points below.**\n>The method is arguably not a great advance over previous work with temp scaing.\n\nWe agree that the idea of temperature scaling exists already in the literature, as Reviewer q9VN  and czv4 had rightly pointed out. Our main contribution lies in demonstrating, both empirically and conceptually, that attention entropy maximization does not always lead to fairer models, as previously claimed by Attanasio et al., (2022).  \n>It is not clear how to trade off the decreased performance and improved AUC equality score.\n\nThanks for bringing this up. As mentioned in Section 5.4, we maximize fairness while ensuring that the performance drop does not exceed 3% on the validation dataset.\n>Optimal parameters vary a lot across datasets, so it would not be useful with data that is not labeled for the sensitive attributes.\n\nWe agree that the optimal value of the temperature scaling hyperparameter varies based on the setting. For different architectures and datasets, models memorize different shortcuts to solve the task by focusing on an overly narrow/wide context, which leads to bias. The optimal beta mitigates this bias by modulating the model’s attention.\n>The datasets and metrics used seem specific -- the choices could have been motivated more and compared to alternatives.\n\nThanks a lot for your comment. We chose the datasets and metrics that are widely used in the fairness literature [1-4]. We will add this information to the manuscript to make it clear.\n\n[1] Zhang et al. Demographics Should Not Be the Reason of Toxicity: Mitigating Discrimination in Text Classifications with Instance Weighting. ACL, 2020.\n\n[2] Reddy et al. Benchmarking bias mitigation algorithms in representation learning through fairness metrics. NeurIPS Datasets and Benchmarks Track, 2021.\n\n[3] Shen et al. Optimising Equal Opportunity Fairness in Model Training. NAACL, 2022\n\n[4] Dixon et al. Measuring and mitigating unintended bias in text classification. AIES, 2018.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Interesting idea for a method.\n* The computational benefit is not requiring pre-training or fine-tuning of a model.\n* The paper demonstrates usefulness in the three tasks.", "reasons_to_reject": "* The method is arguably not a great advance over previous work with temp scaing.\n* It is not clear how to trade off the decreased performance and improved AUC equality score. \n* Optimal parameters vary a lot across datasets, so it would not be useful with data that is not labeled for the sensitive attributes.\n* The datasets and metrics used seem specific -- the choices could have been motivated more and compared to alternatives.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8BdSehPpqc", "reviewer": "Reviewer_UFmf", "summary": "This paper proposes an intra-processing social bias mitigation method that modulates the attention matrix through a validation set. The proposed method is task/model-agnostic and is supposed to be a light-weight solution for the social bias area. However, there are some concerns that should be clarified to improve the paper (please see Weaknesses and Questions below).", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thank you for your constructive feedback and thoughtful engagement in our work. We are motivated to hear that you consider our work well-presented, intuitive, and with thorough analyses. We address your main points below.**\n>the paper would benefit from including more experimental results using other popular debiasing methods such as Mable [1] and [2].\n\nThis is a very interesting suggestion. We added MABLE as a baseline and launched the experiments comparing it to EAT. If we get the results before the rebuttal deadline, we will discuss them, but we will add them to the manuscript in any case. Given that MABLE and Nullspace Projection are in-processing and pre-processing methods, respectively, they are both complementary to our proposed post-processing method (and hence can be combined with it).\n> GLUE benchmarks or any other downstream classification tasks, are necessary.\n\nWe launched experiments to measure the effect of attention modulation on downstream tasks. We will discuss the results if we get them before the rebuttal period ends, and we will add them to the manuscript in any case.\n> the choice of beta can be applied to different layers, implying the knowledge of social bias or spurious correlation is stored among various layers.\n\nThis is an excellent comment. Previous work has shown that the first layers are more critical as they learn more abstract concepts [1], but intuitively any layer could learn bias. This motivates varying the attention modulation based on the layer, which comes at the cost of rendering the hyperparameter search more difficult. We kept the attention modulation factor the same in all layers for simplicity.\n\n[1] Sajjad et al. On the effect of dropping layers of pre-trained transformer models. Computer Speech & Language, 2023.\n>Do the authors have any case studies regarding how the choice of beta is relevant to the linguistic characteristics of a sentence?\n\nThis is an interesting question. Based on our understanding, the choice of beta does not depend on the linguistic characteristic of the sentence per se, it rather depends on the model’s attention to stereotypical tokens. If the model attends to stereotypical tokens in a narrow context, the optimal beta would be smaller than 1, making the model focus on a wider context. However, if the model already attends to a wide context containing stereotypical words, the optimal value of beta would be larger than 1, to make the model focus on a narrower context.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed idea is an intuitive and reasonable approach for intra-processing social bias mitigation.\n- The analysis includes additional experimental results for other transformer architectures and tasks.\n- The paper is well presented.", "reasons_to_reject": "- I really appreciate the authors including testing for other transformer architectures and language generation tasks in the Appendix. However, the paper would benefit from including more experimental results using other popular debiasing methods such as Mable [1] and [2]. I do not expect the performance of the proposed method to be better than these two methods, but it is important to include these comparisons so that we can discern any real differences among these different methods.\n- The authors show the perplexity results to indicate the influence on the language modeling ability, but additional and more detailed results, for example on GLUE benchmarks or any other downstream classification tasks, are necessary to support the paper’s claims.\n\n[1] He, Jacqueline, et al. \"MABEL: Attenuating Gender Bias using Textual Entailment Data.\" Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.\n[2] Ravfogel, Shauli, et al. \"Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection.\" Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020.", "questions_to_authors": "- It is rather straightforward that the choice of beta can be applied to different layers, implying the knowledge of social bias or spurious correlation is stored among various layers. Do the authors have any insights or empirical observations in this respect?\n- Do the authors have any case studies regarding how the choice of beta (or the modulation choice minimization/maximization) is relevant to the linguistic characteristics of a sentence?", "extra_scores": {}}, {"review_id": "CjzTaQ05VU", "reviewer": "Reviewer_vM14", "summary": "In this work the authors empirically investigate the impact of scaling learned attention weights based on a temperature parameter. \nThey argue against the previously proposed strategy of maximizing attention entropy in order to improve model fairness, and instead say that minimizing attention entropy may improve fairness, in certain conditions. \n\nI appreciate the thorough experiments and results.\nHowever, I'm having a hard time seeing the significance of this work. \nIf I understand correctly, the major takeaway is that the answer to \"should we minimize or maximize attention entropy?\" is \"It depends.\"\nThis empirical result is certainly useful, especially in contrast to Attanasio et al. where the proposed \"fix\" is always to maximize entropy.\n\nHowever, it is unclear when minimization or maximization should be deployed. \nWhat is underneath the fact that for some architectures/datasets/in-processing method, minimization is better? \nIt would be really interesting to see a detailed analysis of the conditions under which minimization outperforms maximization and vice versa. \nAlso, examples of the impacts of applying EAT would also be helpful. \nShow the attention outputs pre- and post-EAT to demonstrate that the bias is being mitigated.\nOtherwise it looks a lot like a fine-tuning exercise that just happens to reduce demographic parity. \nEven in Appendix B, there is just a lot of variety in the results, which makes it hard to draw conclusions.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thank you for your detailed review and engagement in our work. We are delighted to hear that you consider our work interesting and with thorough evaluations. We address your main points below.**\n>Results are difficult to parse and not contextualized. Even in Appendix B, there is just a lot of variety in the results.\n\nWe are sorry about this confusion. The results in Figure 2 and Appendix B demonstrate that fairness is not always maximized by attending to a wider context, as previously claimed by Attanasio et al., (2022), and that the architecture and dataset determine whether the model needs to focus on a wider/narrower context to improve fairness. We will clarify this confusion in the manuscript.\n>It is unclear why and when minimizing versus maximizing attention entropy is preferred\n\nThanks for bringing up this important point. If the model attends to stereotypical tokens in a narrow context, attention entropy maximization makes the model focus on a wider context, leading to less bias. However, if the model already attends to a wide context containing stereotypical words, attention entropy minimization makes the model focus on a narrower context, which also reduces bias.\n\n>Examples of the impacts of applying EAT would also be helpful. Show the attention outputs pre- and post-EAT to demonstrate that the bias is being mitigated.\n\nThis is a great suggestion. We will add more qualitative examples, as shown below.\n\nFor toxicity detection using BERT and Input *“Hello Dan, please fix LanguageRo.php first, it has explicit references to Wikipedia.”*:\n| **Method** | **Token with highest attention** |\n|:---:|:---:|\n| Vanilla | Dan |\n| EAT | Wikipedia |\n\nInput *“Freedom will and must prevail, your tyranny will come to an end, Nicholas.”*:\n\n| **Method** | **Token with highest attention** |\n|:---:|:---:|\n| Vanilla  | Nicholas|\n| EAT | tyranny |\n\nUsing RoBERTa and Input  *“Emilia, do you find more information about this Swedish diva? If you do so, help complete this article!”*:\n| **Method** | **Token with highest attention** |\n|:---:|:---:|\n| Vanilla | Emilia|\n| EAT | article|\n\n>I'm having a hard time seeing the significance of this work.\n\nAs the reviewer had rightly pointed out in their summary, our main contribution is showing empirically and conceptually that the previous work by Attanasio et al., (2022), claiming that maximizing attention entropy always improves fairness, is not general. We will make our main contribution clearer in the revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Thorough evaluations\n- Interesting claim that pushes back against prior work", "reasons_to_reject": "- Results are difficult to parse and not contextualized\n- It is unclear why and when minimizing versus maximizing attention entropy is preferred", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper reevaluates a previously recognized solution to bias-mitigation, resulting in a very different conclusion (relating to attention entropy, namely that increasing or decreasing entropy can improve fairness, depending on task and dataset). This is an interesting, and the approach taken simple and easily applied without fine-tuning; moreover the experiments were not limited to classification only, which is common in fairness papers. However the paper is lacking any deep insights to explain the key finding - that is, what is the underlying cause of the need to sharpen or smooth the attention, in terms of the data/task/linguistic context/etc. The author response is not satisfying, just summarizing the observed model behavior (see response to vM14). In other respects, the paper is strong: well written, evaluation sound, and the authors added new evaluations in their response.
Decision: Accept