Should We Attend More or Less? Modulating Attention for Fairness

Authors: abdelrahman  zayed
gonçalo  mordido
samira  shabanian
sarath  chandar
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-20 15:53:34

Abstract: The advances in natural language processing (NLP) pose both opportunities and challenges. While recent progress enables the development of high-performing models for a variety of tasks, it also poses the risk of models learning harmful biases from the data, such as gender stereotypes. In this work, we investigate the role of attention, a widely-used technique in current state-of-the-art NLP models, in the propagation of social biases. Specifically, we study the relationship between the entropy of the attention distribution and the model's performance and fairness. We then propose a novel method for modulating attention weights to improve model fairness after training. Since our method is only applied post-training and pre-inference, it is an intra-processing method and is, therefore, less computationally expensive than existing in-processing and pre-processing approaches. Our results show an increase in fairness and minimal performance loss on different text classification and generation tasks using language models of varying sizes.

Main paper: 
 introduction: 
 Advancements in transformer-based pre-trained language models, such as BERT (Devlin et al., 2019) , RoBERTa (Liu et al., 2019) , GPT-2 (Radford et al., 2019) , GPT-3 (Brown et al., 2020) and XLNet (Yang et al., 2019) , have led to the emergence of new state-of-the-art models in a variety of applications, including but not limited to text summarization (Liu et al., 2022; Mordido & Meinel, 2020) , sentence classification (Wang et al., 2018) , question answering (Rajpurkar et al., 2018; 2016) , and information extraction (Li et al., 2020a; b) . Despite their success, recent studies (Nadeem et al., 2021; Meade et al., 2022; Zayed et al., 2024a) have demonstrated that these models also exhibit harmful biases based on factors such as gender, race, sexual-orientation, and religion. These biases pose a significant challenge in deploying machine learning models in real-world applications, as they can result in discriminatory outcomes. For example, it would be ethically questionable to deploy a machine learning model for resume filtering if it is known that the model would discriminate against certain applicants based on their gender.Several techniques have been proposed to address gender bias in language models, which may be broadly classified into pre-processing (Lu et al., 2020; Hall Maudslay et al., 2019; De-Arteaga et al., 2019; Dixon et al., 2018) , in-processing (Garg et al., 2019; Zhang et al., 2020; Attanasio et al., 2022; Kennedy et al., 2020) , and post-processing methods (Wei et al., 2020) . More recently, a new category of debiasing methods has been introduced: intra-processing methods (Savani et al., 2020) . These methods involve modifying the model weights after training but before inference and, therefore, are significantly less costly than pre-processing and in-processing methods. In contrast to post-processing methods, which are primarily designed for tabular datasets, intra-processing methods have the advantage of not being dataset dependent. In this work, we propose a new intra-processing method to address gender bias in language models. The work by Attanasio et al. (2022) proposed an in-processing bias mitigation method called entropy attention-based regularization (EAR), which improves model fairness by maximizing the entropy of the attention weights distribution during training. The authors argue that maximizing the entropy of the attention map distribution leads to the model attending to a broader context within the input sentence, preventing it from relying on a few stereotypical tokens, which results in a fairer model. In this work, we study the effect of attention distribution entropy on both fairness and performance and find that the relationship between the model's bias and the entropy of its attention distribution is both dataset and architecture-dependent. This suggests that some of the previous findings by Attanasio et al. (2022) may not be general.Hence, we propose to modulate the attention distribution entropy after training, rather than maximize it during training. Our novel attention entropy modulation, called entropy-based attention temperature scaling (EAT), applies a scaling factor to modulate the entropy of the attention map post-training. In the end, we are able to efficiently improve fairness with minimal performance loss. Our contributions may be summarized as follows:1. We study the effect of modulating the entropy of the attention distribution on gender bias and performance in BERT and RoBERTa models fine-tuned on three text classification datasets. 2. We propose a novel intra-processing bias mitigation method that modulates the attention distribution entropy in text classification with less than 3.5% degradation in performance. To the best of our knowledge, this is the first intra-processing method bias mitigation method in NLP, offering a computationally efficient alternative to existing methods. 3. Our method outperforms existing intra-processing debiasing methods originally proposed for image data and migrates such efforts to the NLP domain. We also outperform other pre/in-processing methods, including EAR. 4. We combine our method and the previous intra-processing baselines with five popular in-processing and pre-processing bias mitigation methods and show that such method combinations always improve fairness. 5. We show that our method generalizes to different forms of social bias even when the hyperparameters are exclusively tuned to mitigate gender bias. 6. Finally, we show that our method extends beyond text classification to address social biases in text generation using GPT-Neo. 
related work: 
 We will start by describing existing techniques for mitigating bias that are applied before, during, or after model training. Additionally, we will delve into previous studies that have proposed modifying the attention map to improve performance.Gender bias mitigation methods may be broadly classified into two categories: intrinsic and extrinsic approaches. While intrinsic methods (Adi et al., 2017; Hupkes et al., 2018; Conneau et al., 2018; Tenney et al., 2019; Belinkov & Glass, 2019) focus on analyzing the embedding representations assigned to gender tokens by the model, extrinsic methods (Sennrich, 2017; Isabelle et al., 2017; Naik et al., 2018) rely on the model's predictions to determine if different genders achieve similar predictions under the same context. In this paper, we focus on extrinsic bias mitigation methods, as they more accurately reflect the applicability and performance of the model in real-world situations.Pre-processing methods for mitigating gender bias involve modifying the training data to improve model fairness. One common method is counterfactual data augmentation (CDA) (Lu et al., 2020) -Arteaga et al., 2019) removes all gender words from the dataset, preventing the model from associating any label with a specific gender. Lastly, data balancing (Dixon et al., 2018) adds new examples only for under-represented groups in the dataset.In-processing bias mitigation methods aim to reduce bias during training by adding auxiliary loss terms to the model. One example is counterfactual logit pairing (Garg et al., 2019) , which penalizes the model if it makes different predictions for the same input after altering sensitive attributes such as gender words. Another method by Kennedy et al. (2020) adds a penalty term based on the difference in output logits when sensitive attributes are present or absent. Instance weighting (Zhang et al., 2020) multiplies the loss by a factor greater than 1 for stereotypical sentences to penalize the model more for misclassifying them, and attentionbased regularization (Attanasio et al., 2022) maximizes the model's attention distribution entropy during training to improve fairness. MABEL (He et al., 2022) proposes adding an auxiliary loss to minimize the cosine similarity between the original and gender-swapped vector representation for each example.While relatively less explored, post-processing bias mitigation methods modify the predictions of a biased model and generate a new set of less biased predictions. For example, Wei et al. (2020) address this problem by formulating it as a convex optimization problem with fairness constraints. Intra-processing debiasing methods have been introduced to reduce the biases of image processing models as a new category of techniques that lie between in-processing and post-processing methods. Examples include applying random perturbations to model weights, modifying the weights of a given layer, or performing adversarial fine-tuning to reduce model bias (Savani et al., 2020) . Recent work prunes the attention heads with the highest contribution to bias as an intra-processing debiasing method (Zayed et al., 2024b) .There has been an ongoing debate in the literature regarding the interpretability of the attention mechanism (Jain & Wallace, 2019; Wiegreffe & Pinter, 2019; Serrano & Smith, 2019; Moradi et al., 2019; Mohankumar et al., 2020) . Despite this, several works have demonstrated that modulating the attention map values with prior knowledge improves model performance on downstream tasks. For example, in document summarization, Cao & Wang (2021) proposed a content selection method that detects tokens that are irrelevant at inference time, masking them from the attention map. Moreover, Cao & Wang (2022) proposed to increase the attention weights between tokens that lie within the same section in the document. Furthermore, Zhang et al. (2022) applied temperature scaling during inference to the attention map, encouraging the model to focus on a broader context to improve the quality of the summarization.In the context of text classification, Li et al. (2021) proposed to use a local attention map, limiting the attention based on the dependency parse tree to make the model more syntaxaware. Additionally, in language generation, modulating the attention weights has been applied both during training and inference to enhance fluency and creativity (Dong et al., 2021) by updating the attention weights between tokens using a learned or pre-defined reweighting function. In machine translation, Yin et al. (2021) employed a group of freelance translators to identify the words they used to translate each word in a given output sequence.The authors then added an auxiliary loss term to encourage the model to pay more attention to these words, resulting in a performance improvement. Moreover, Lu et al. (2021) proposed to increase the model's attention to essential words by measuring the performance drop after the removal of such words, encouraging the model to attend more to the words that led to the most significant drop. 
self-attention: 
 One of the key factors contributing to the success of transformer models (Vaswani et al., 2017) is the usage of self-attention (Bahdanau et al., 2015) to compute the representation of each token in various layers of the model. In particular, the representation of the i th token is contingent upon its relevance to all other tokens within the sentence. This relevance between tokens i and j is referred to as the attention from token i to token j. Hence, if the maximum sentence length is T, the attention map will be a T × T matrix representing the attention of each token to all other tokens. The attention map is calculated as:EQUATIONwhere Q ∈ R T×d q , K ∈ R T×d k , and V ∈ R T×d v are the query, key, and value matrices with embedding dimensionalities d q , d k , and d v respectively. The computation of these matrices is described as:Q = XW Q ; K = XW K ; V = XW V, where X ∈ R T×d is the input vector of maximum length T and embedding dimentionality d, whileW Q ∈ R d×d q , W K ∈ R d×d k ,and W V ∈ R d×d v are three matrices of learnable parameters.The softmax function used to calculate the attention map in Eq.( 1) ensures that the attention values from any token to all other tokens within the sentence are non-negative and sum to one, and therefore may be treated as probabilities. The larger the attention values in this distribution between tokens i and j, the greater the correlation between them. We follow the same procedure as Attanasio et al. (2022) to calculate the entropy of the attention distribution.Considering the attention values a l,h,i,j between tokens i and j in the head h of layer l, we first average the attention weights over all heads in the l-th layer: a ′ l,i,j = 1 h ∑ h a l,h,i,j . Subsequently, a softmax function is applied to ensure that the resulting values form a probability distribution: a l,i,j = e a ′ l,i,j∑ j e a ′ l,i,j. The entropy, first introduced by Shannon (1948), is defined byH l i = -∑ T sj=0 a l,i,j log a l,i,j , where T s represents the actual length of the sentence, excluding any padding tokens. We obtain the average entropy within a sentence in layer l asH l = 1 T s ∑ T s i=0 H l i, with the overall attention entropy being computed by summing the entropy across all model layers:H = ∑ l H l .4 Attention entropy modulation Attanasio et al. (2022) suggested that memorization encourages models to focus on stereotypical tokens as a shortcut to solve the task, and proposed to maximize the entropy of the attention distribution. The intuition is to force the model to attend to a broader context, resulting in a less biased model. In our work, we challenge this hypothesis and demonstrate that model fairness may be improved not only by maximization but also by minimization of attention entropy. If there are stereotypical tokens in the narrower context, then attending to a broader context is likely to improve fairness. However, if the narrower context is already devoid of stereotypical tokens, then attending to a broader context could potentially expose the model to more bias. Hence, we posit that the relationship between attention entropy and bias is both dataset and model dependent, and propose to perform attention entropy modulation, instead of maximization. 
entropy-based attention temperature scaling (eat): 
 We propose a novel method for attention entropy modulation, which we term entropy-based attention temperature scaling (EAT). Our approach modulates the entropy of the model's attention maps by performing temperature scaling after training. The amount of scaling applied is controlled by a hyperparameter β that is chosen based on the validation set such that a good trade-off between performance and fairness is achieved. A scaled attention map, i.e. after temperature scaling, is computed as (Hinton et al., 2015) :Attention s (Q, K, V) = softmax βQK T √ d k V.(2)Note that this scaling is applied to all the attention layers of the model.To gain a deeper understanding of the role of the temperature scaling factor β in modulating the attention map's entropy, it is instructive to consider the cases where β is smaller and larger than 1. When β is less than 1, the attention entropy increases since the attention map values are brought closer together prior to the application of the softmax function, resulting in a more uniform distribution. Indeed, as β reaches 0, the values after the softmax closely resemble a uniform distribution, which corresponds to the highest possible entropy.Conversely, when β is greater than 1, the attention entropy decreases since the difference between the largest value and the rest of the values in the attention map is amplified, resulting in a less uniform distribution after the softmax. If β is significantly larger than 1, we approach a scenario where only the largest value will be attended to. These scenarios are illustrated in Figure 1 .  
experiments: 
 We will now provide a detailed overview of the tasks, datasets, baselines, and evaluation metrics as well as the different setups used in our experiments.We performed our experiments on two distinct binary text classification tasks: sexism detection and toxicity detection, where the objective is to train a model to accurately differentiate between texts that are deemed sexist or toxic and those that are not, respectively. As defined by Dixon et al. (2018) , a toxic comment is one that prompts an individual to disengage from a discussion. All the datasets used in this study are in English. We used the following datasets: Twitter dataset (Waseem & Hovy, 2016) with approximately 16, 000 tweets binarized into sexist and non-sexist, Wikipedia dataset (Dixon et al., 2018) with around 160, 000 comments categorized as toxic or non-toxic, and the Jigsaw dataset1 with around 1.8 million examples binarized into toxic and non-toxic. Moreover, we evaluated the feasibility of extending our method to text generation using the bias in open-ended language generation dataset (BOLD) (Dhamala et al., 2021) with 23, 679 prompts referring to professions, genders, races, as well as religious and political groups.We compare our method (EAT) with three other intra-processing methods originally proposed by Savani et al. (2020) for image data: random weight perturbation, layer-wise optimization, and adversarial fine-tuning. Moreover, we also use six pre/in-processing gender bias mitigation methods: instance weighting (Zhang et al., 2020) , data augmentation (Lu et al., 2020) , data substitution (Hall Maudslay et al., 2019) , gender blindness (De-Arteaga et al., 2019), MABEL (He et al., 2022) , and entropy attention-based regularization (EAR) (Attanasio et al., 2022) . We refer the reader to Section 2 for a description of the methods.For text classification, we evaluate model performance using the area under the receiver operating characteristic curve (AUC). For text generation, perplexity (PPL) on Wikitext-2 is used to measure the language modeling ability (Merity et al., 2017) . We also measure the performance on seven different downstream tasks: ARC (Chollet, 2019), HellaSwag (Zellers et al., 2019) , MMLU (Hendrycks et al.) , TruthfulQA (Lin et al., 2022) , GSM8K (Cobbe et al., 2021) , Winogrande (Sakaguchi et al., 2020) , and DROP (Dua et al., 2019) .To evaluate gender bias in text classification, we mainly use the demographic parity (DP) metric (Beutel et al., 2017; Hardt et al., 2016; Reddy, 2022) calculated by: DPEQUATIONwhere ŷ represents the model's prediction and z ∈ {0, 1} denotes keeping or flipping the gender words in the sentence, respectively. The scale of demographic parity ranges from 0 (least fair) to 1 (fairest). Our procedure for computing DP follows prior studies (Dixon et al., 2018; Park et al., 2018) , where we use a synthetic dataset, the identity phrase templates test set (IPTTS), for measuring fairness. Additional fairness metrics are described in Appendix B. To evaluate other forms of social bias, we employ the pinned AUC equality difference metric (Dixon et al., 2018) , which is defined as:EQUATIONwhere AUC and AUC t refer to the model's AUC on the whole dataset and on examples referring to a specific subgroup t (e.g., Christianity, Judaism, and Islam subgroups for religion bias), respectively. Lower values correspond to less bias.For text classification, we trained BERT and RoBERTa base models for 15 epochs using cross-entropy loss on the Twitter and Wikipedia datasets and 4 epochs on the Jigsaw dataset. The performance, measured by the AUC, is in-line with the state-of-the-art results on the three datasets. The training, validation, and testing data were split in a ratio of 8:1:1, with the exception of the Wikipedia toxicity dataset, for which the split ratio used by Dixon et al. (2018) was employed.For text generation, we used GPT-Neo (Black et al., 2021) with 125M, 1.3B and 2.7B parameters. All experiments were run five times using different random seeds. Our proposed method, EAT, uses a single hyperparameter, the temperature scaling factor β, which is selected based on the validation set. We adopted the same procedure to find the best hyperparameters for our baselines. The criterion for determining the optimal value of β is to maximize the demographic parity (DP) while ensuring less than 3% degradation in validation performance, through a search range of β ∈ {0, 0.1, .., 10}. Additional implementation details are provided in Appendix A. Our code will be made public2 .Experiment 1: Improving fairness with attention entropy modulation. We will first investigate the relationship between attention entropy and gender bias by varying the temperature scaling coefficient β. Figure 2 shows that as β decreases, attention entropy increases, The values of β that are smaller or larger than 1 correspond to maximizing or minimizing the attention entropy, respectively. Best viewed in color. resulting in a wider context being attended to. As β reaches 0, the models attend equally to all tokens, leading to a decrease in performance and an increase in fairness. Conversely, increasing β decreases attention entropy, making the models attend to a narrower context. Importantly, minimizing attention entropy leads to improvements in fairness compared to the baseline model, especially on Twitter and Wikipedia datasets using RoBERTa. A similar trend is also observed on the additional fairness metrics (see appendix B). Table 1 presents the β chosen based on the validation dataset. We see that the choice of whether to maximize or minimize attention entropy may vary depending on the dataset and model used.Experiment 2: Comparing with EAR and other baselines, and generalizing to other forms of social bias. Table 2 shows a comparison between EAT and other pre/in-processing methods, showing its effectiveness in improving fairness on different models and datasets, with less than 3% degradation in AUC for all methods. More importantly, EAT outperforms EAR in all scenarios, which underlines the superiority of attention modulation to attention maximization. Since intra-processing methods are only applied post-training, we also study their combination with different pre/in-processing baselines. In particular, we combine each intra-processing method with 5 distinct bias mitigation methods of the 6 approaches outlined in Section 5.2 (we excluded MABLE for computational reasons), as well as no bias mitigation, on 3 datasets and 2 different models. In the end, this results in a total of 36 (Zhang et al., 2020) -0.32 ± 3.07 RoBERTa CDA (Lu et al., 2020) 1.96 ± 1.37 RoBERTa CDS (Hall Maudslay et al., 2019) 1.47 ± 1.38 RoBERTa Gender blindness (De-Arteaga et al., 2019) -1.97 ± 1.51 RoBERTa MABEL (He et al., 2022) 2.71 ± 1.4 RoBERTa EAR (Attanasio et al., 2022) -0. (Zhang et al., 2020) 1.08 ± 0.74 RoBERTa CDA (Lu et al., 2020) 1.12 ± 0.94 RoBERTa CDS (Hall Maudslay et al., 2019) 1.71 ± 1.02 RoBERTa Gender blindness (De-Arteaga et al., 2019) 0.22 ± 1.07 RoBERTa MABEL (He et al., 2022) 2.46 ± 1.01 RoBERTa EAR (Attanasio et al., 2022) 0.27 ± 0.37 RoBERTa EAT (ours) 2.32 ± 0.77 RoBERTa MABEL (He et al., 2022) (Zhang et al., 2020) 0.70 ± 0.69 RoBERTa CDA (Lu et al., 2020) 1.24 ± 0.73 RoBERTa CDS (Hall Maudslay et al., 2019) 1.00 ± 0.79 RoBERTa Gender blindness (De-Arteaga et al., 2019) -0.02 ± 0.76 RoBERTa EAR (Attanasio et al., 2022) -0.41 ± 0.93 RoBERTa EAT (ours) 0.60 ± 0.64Table 2 : A comparison between EAT and pre/in-processing methods in terms of percentage of change in demographic parity for the different models and datasets. Numbers are shown with one standard deviation. Our proposed EAT (i.e. attention modulation) outperforms EAR (i.e. attention maximization) in all scenarios. MABEL is a strong in-processing baseline, so we show the result of its combination with our proposed EAT, which leads to more improvement in bias mitigation. We did not implement MABLE for the Jigsaw dataset for computational reasons.scenarios per intra-processing method, and 900 different experiments. Figure 3 illustrates the frequency with which each intra-processing method ranked first in terms of fairness across various forms of social bias, as measured by the pinned AUC equality difference (Eq.( 4)). The results indicate that EAT outperforms the existing intra-processing methods.The degradations in AUC performance of all methods were less than or equal to 3.5%, except for random perturbation. Notably, the hyperparameter selection was conducted solely on gender bias, which shows EAT's ability to generalize to other social biases.Prompt: The United Nation of Islam is an African American... Vanilla: terrorist group based on the teachings of its founder, Mohammed. Random pert.: phenomenon that has infiltrated colleges campuses, racism TED ESV. EAT: religious organization based in Los Angeles, California. Prompt: Jane Cowl was an American film and... Vanilla: television actress best known for her roles as Nurse Anne. Random pert.: television actress and former beauty pageant titleholder who achieved prominence and name recognition as a former model and actress. EAT: television actress, known for her roles in television movies such as Harsh Times. Figure 5 : Perplexity of EAT (solid) and random perturbation (dashed) on Wikitext-2 against β using GPT-Neo with 1.3 and 2.7 billion parameters.Experiment 3: Extending EAT to text generation. We explore the applicability of EAT to address various forms of social bias in text generation using BOLD framework. BOLD consists of thousands of prompts pertaining to five distinct groups. Following previous work (Dhamala et al., 2021) , we use the toxicity of both the prompt and the model's continuation as a proxy for its bias towards any particular group. Figure 4 shows the percentage change in toxicity using EAT on GPT-Neo for different groups, relative to the vanilla model. A 0.7 0.8 0.9 1.01.1 1.2 1.3 0.0 0.1 0.2 0.3 0.4 0.5 Performance ARC HellaSwag MMLU TruthfulQA Winogrande GSM8K DROPFigure 6 : Accuracy of EAT on 7 different downstream tasks using GPT-Neo with 125M parameters. The values for all tasks refer to accuracy scores, except for the DROP task which uses F1 score. We evaluate 0-shot accuracy in all tasks, except for GSM8K where we compute the 5-shot accuracy. The table shows that attention entropy modulation (i.e. maximization or minimization) in text generation does not result in performance degradation on downstream tasks.qualitative comparison is shown in Table 3 . Figures 5 and 6 illustrate the same comparison for performance. EAT reduces toxicity without sacrificing language modeling ability. Conversely, random perturbation increases toxicity for all groups, with values falling outside the specified range, while substantially increasing perplexity. The percentages of increase in toxicity using random perturbation for gender, political, profession, race, and religion biases using GPT-Neo with 1.3 and 2.7 billion parameters are as follows: 3.2, 4.7, 4.2, 3.7, 4.8 and 1.6, 8, 2.2, 4.2, 0.4, respectively. We did not implement other intra-processing baselines due to resource constraints. In addition, EAT does not lead to substantial degradation on the downstream tasks used, while improving the performance in certain cases. More specifically, attention entropy minimization (β > 1) leads to better performance on ARC, HellaSwag, TruthfulQA, GSM8K, and Winogrande; while attention entropy maximization (β < 1) improves performance on GSM8K and Winogrande. 
conclusion: 
 In this work, we examined the impact of entropy in the attention distribution on fairness and performance in different language models. Our results indicate that, in contrast with previous research (Attanasio et al., 2022) , both attention entropy maximization and minimization may enhance fairness depending on the model and task at hand. With this in mind, we propose a computationally efficient and novel bias mitigation technique that modulates the entropy of the attention distribution after training and prior to inference. Our extensive results on both text classification and generation datasets show that we are able to improve fairness while maintaining most of the performance of the original biased model. 
 
 Appendix: The entropy attention-based regularization method (EAR) (Attanasio et al., 2022) has one hyperparameter α, which regulates the trade-off between the cross-entropy loss and the entropy maximization loss. We adopt the same pattern for identifying the optimal values of β, but with a wider search space: {10 -6 , 10 -5 , .., 1, 10, 100}. We note that the used α range is also wider than the one used in the original work by Attanasio et al. (2022) to ensure a fair comparison.Table 4 presents the optimal values of β selected in Experiment 2. A thorough analysis of the results reveals that the fairness of the model is depends on the dataset and architecture used, with the model's fairness improving through either maximization or minimization of attention entropy. Furthermore, the results indicate that the combination of pre-processing and in-processing techniques with EAT also plays a role in determining the optimal value of β. Specifically, when using the entropy maximization baseline (EAR), the optimal value of β was found to be 10 for RoBERTa on the Wikipedia dataset. Given that values of β that are larger than 1 minimize the attention entropy, this result supports the conclusion that attention maximization was not an appropriate choice for this specific model and dataset.Our proposed method, EAT, effectively modulates attention to improve fairness.Table 5 : A comparison between the running time of EAT and other pre-processing and in-processing methods relative to the vanilla model with no debiasing. The total running time for any method is calculated as the time to train the biased model plus the time to train the debiased model, to compute the percentage of change in performance and fairness. EAT's running time is the same as the vanilla model without debiasing since the temperature scaling does not introduce additional time overhead.To implement the baselines for counterfactual data augmentation (CDA) (Lu et al., 2020) , counterfactual data substitution (CDS) (Hall Maudslay et al., 2019) , and gender blindness (De-Arteaga et al., 2019) , it is essential to accurately detect gender-specific words for modification or removal. We used the publicly available gender-bender Python package3 for this purpose. This package provides a comprehensive list of gender-specific words and their corresponding alternatives, which enabled us to effectively implement the aforementioned methods. We used the detoxify library4 for measuring toxicity.and y = 0 (EqOpp0):EQUATIONand computing the average: EQUATIONIn this section, we provide additional results on the bias in text classification and generation models.Table 6 shows the tokens with the highest attention weights in BERT and RoBERTa models on the Wikipedia toxicity detection task using different debiasing methods. EAT puts the highest attention weights on the tokens that are more relevant to the task, as opposed to the vanilla model and EAR, which focus more on gender tokens.Our bias assessment uses the toxicity in the model's continuation as a proxy for its bias towards different groups. This is in line with the original BOLD paper (Dhamala et al., 2021) . Figures 8 and 9 show a detailed comparison for the percentage of change in toxicity when using EAT with β = 0.9 and random perturbation using GPT-Neo with 1.3 and 2.7 billion parameters, respectively. We used the detoxify library for measuring toxicity. The results show that EAT is more effective in producing less toxic output with an almost negligible decrease in the language modeling ability, as shown in Figure 5 . The percentage of change in toxicity when using random perturbation and EAT with β = 0.9 for GPT-Neo with 2.7 billion parameters, compared to the vanilla model. The comparison is on different subgroups that belong to professions, genders, races, as well as religious and political groups.https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classificationhttps://github.com/chandar-lab/EAThttps://www.github.com/Garrett-R/gender benderhttps://pypi.org/project/detoxify/ 


Decision: This paper reevaluates a previously recognized solution to bias-mitigation, resulting in a very different conclusion (relating to attention entropy, namely that increasing or decreasing entropy can improve fairness, depending on task and dataset). This is an interesting, and the approach taken simple and easily applied without fine-tuning; moreover the experiments were not limited to classification only, which is common in fairness papers. However the paper is lacking any deep insights to explain the key finding - that is, what is the underlying cause of the need to sharpen or smooth the attention, in terms of the data/task/linguistic context/etc. The author response is not satisfying, just summarizing the observed model behavior (see response to vM14). In other respects, the paper is strong: well written, evaluation sound, and the authors added new evaluations in their response.

Reviews: review: paper_summary: paper_summary: This paper focuses on minimizing social bias in language models. Specifically, it builds on prior work by Attansio et al., (2022) and attempts to provide an alternative solution to building an accurate language model that also improves fairness metrics. Attansio et al. posit that maximizing the entropy of the attention weights distribution during fine-tuning leads to less biased models (called EAR). This paper argues that this is not necessarily always true; instead it proposes to modulate the attention entropy using a hyperparameter (temperature scaling factor), which could lead to either an increase of decrease in entropy, arguing that the modulation is a factor of the dataset and architecture (henceforth, EAT). 

This hypothesis is experimentally validated on three classification datasets (Wikipedia, toxic v/s non-toxic classification; Jigsaw, toxic v/s non-toxic classification; Twitter, sexist v/s non-sexist comment classification) and one open ended generation dataset (BOLD). A variety of alternative and previously proposed approaches are considered as baselines. The quality of the models are measured by AUC for classification and perplexity for open-ended generation, while fairness is measured using Demographic Parity for gender bias and pinned AUC equality difference for other biases (e.g. religion). Findings reveal that the choice of the temperature scaling factor can indeed affect fairness metrics on classification tasks, with EAT consistently improving these metrics compared to EAR. EAT more frequently leads to improved fairness metrics compared to other baselines as well.  On text generation, EAT outperforms the random perturbation baseline on both quality and fairness metrics.

Overall, I like that this work proposes a post-training (or what is called as *intra-processing* in this paper) approach to mitigating fairness issues, compared to Attansio et al., which requires fine-tuning the underlying model. This makes the approach fairly easy to try. I also appreciate that this work does not limit itself to classification tasks. With the direction the community is heading in, it is paramount to measure and mitigate biases in language modeling tasks as well. Finally, I also appreciate that this work attempts to provide a generalization to a prior, well-recognized solution to the problem (i.e. entropy maximization is not sufficient, and modulating the entropy is a more general solution)

However, I do think this work is not technically exciting as it proposes a straightforward application of temperature scaling, which has been used in knowledge distillation, calibration of neural networks [1], etc. Another drawback of this approach is that it requires a per dataset approach to mitigating bias (by identifying the right temperature scaling parameter). Finally, I also think that this paper is missing some side-by-side comparison of examples where EAT improves upon EAR. In the introduction to section 4, the authors argue that *"If there are stereotypical tokens in the narrower context, then attending to a broader context is likely to improve fairness. However, if the narrower context is already devoid of stereotypical tokens, then attending to a broader context could potentially expose the model to more bias".* I think having some examples that actually validate this will strengthen the paper. 


[1] - https://arxiv.org/pdf/1706.04599
strength_weakness: strength_weakness: * The temperature scaling hyperparameter indicates that this approach needs to be configured for different data distributions/architectures, which makes it less broadly useful, when test data distribution is unknown. 
* The technical novelty in this paper is limited as the approach is an application of temperature scaling
* Qualitative insights to back up quantitative results are completely missing.
questions: questions: * Citations are incorrectly parenthesized throughout the paper, and this needs fixing [e.g. intra-processing methods Savani et al. (2020) --> intra-processing methods (Savani et al., 2020)].
* Why did you choose the bias metrics that you did in section 5.3? How confident are you that a single bias/fairness metrics supports the findings in this paper? If other fairness metrics are considered, do you expect the findings in this work to generalize?
paper_summary: paper_summary: This paper proposes a novel intra-processing method, Entropy-based Attention Temperature Scaling (EAT), to modulate attention entropy for improving fairness in NLP models. The problem is that transformer models memorize stereotypical tokens. EAT does temperature scaling after training. It adds noise to the softmax in the attention mechanism. 

The study uses three datasets: Twitter, Wikipedia, and Jigsaw, all in English, for two text classification tasks,  sexism and toxicity detection. Performance goes down in the overall performance, but it reduces disparities in AUC between messages that refer to various groups.
strength_weakness: strength_weakness: * The method is arguably not a great advance over previous work with temp scaing.
* It is not clear how to trade off the decreased performance and improved AUC equality score. 
* Optimal parameters vary a lot across datasets, so it would not be useful with data that is not labeled for the sensitive attributes.
* The datasets and metrics used seem specific -- the choices could have been motivated more and compared to alternatives.
paper_summary: paper_summary: This paper proposes an intra-processing social bias mitigation method that modulates the attention matrix through a validation set. The proposed method is task/model-agnostic and is supposed to be a light-weight solution for the social bias area. However, there are some concerns that should be clarified to improve the paper (please see Weaknesses and Questions below).
strength_weakness: strength_weakness: - I really appreciate the authors including testing for other transformer architectures and language generation tasks in the Appendix. However, the paper would benefit from including more experimental results using other popular debiasing methods such as Mable [1] and [2]. I do not expect the performance of the proposed method to be better than these two methods, but it is important to include these comparisons so that we can discern any real differences among these different methods.
- The authors show the perplexity results to indicate the influence on the language modeling ability, but additional and more detailed results, for example on GLUE benchmarks or any other downstream classification tasks, are necessary to support the paper’s claims.

[1] He, Jacqueline, et al. "MABEL: Attenuating Gender Bias using Textual Entailment Data." Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.
[2] Ravfogel, Shauli, et al. "Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection." Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020.
questions: questions: - It is rather straightforward that the choice of beta can be applied to different layers, implying the knowledge of social bias or spurious correlation is stored among various layers. Do the authors have any insights or empirical observations in this respect?
- Do the authors have any case studies regarding how the choice of beta (or the modulation choice minimization/maximization) is relevant to the linguistic characteristics of a sentence?
paper_summary: paper_summary: In this work the authors empirically investigate the impact of scaling learned attention weights based on a temperature parameter. 
They argue against the previously proposed strategy of maximizing attention entropy in order to improve model fairness, and instead say that minimizing attention entropy may improve fairness, in certain conditions. 

I appreciate the thorough experiments and results.
However, I'm having a hard time seeing the significance of this work. 
If I understand correctly, the major takeaway is that the answer to "should we minimize or maximize attention entropy?" is "It depends."
This empirical result is certainly useful, especially in contrast to Attanasio et al. where the proposed "fix" is always to maximize entropy.

However, it is unclear when minimization or maximization should be deployed. 
What is underneath the fact that for some architectures/datasets/in-processing method, minimization is better? 
It would be really interesting to see a detailed analysis of the conditions under which minimization outperforms maximization and vice versa. 
Also, examples of the impacts of applying EAT would also be helpful. 
Show the attention outputs pre- and post-EAT to demonstrate that the bias is being mitigated.
Otherwise it looks a lot like a fine-tuning exercise that just happens to reduce demographic parity. 
Even in Appendix B, there is just a lot of variety in the results, which makes it hard to draw conclusions.
strength_weakness: strength_weakness: - Results are difficult to parse and not contextualized
- It is unclear why and when minimizing versus maximizing attention entropy is preferred

Comments: title: Thank you
Cases where we minimize/maximize entropy
Results of MABLE baseline suggested by the reviewer
The effect of attention entropy modulation on downstream tasks in text generation models
comment: Thank you for the clear response.
Thank you for your time and effort in reviewing our work.
We appreciate your response and engagement.

The short answer to your question is: it depends on the data and model used. The longer answer is: For different architectures and datasets, models memorize different shortcuts to solve the task by focusing on an overly narrow/wide context, which leads to bias. The optimal beta mitigates this bias by modulating the model’s attention entropy.
Thank you for the response. I appreciate the comments. 
I'm still at the same point with my scores. The major question for me is: is it enough to say that there are various scenarios where prior work doesn't hold, or are detailed delineations of cases where we minimize/maximize entropy required?
Thank you for your response. I think adding the qualitative examples will help the paper. Overall, I'll still stand by my initial review that the paper has limited technical novelty, but I appreciate the refutation of prior work by finding a more general approach that solves the underlying problem.
We also obtained the results for the in-processing baseline suggested by the reviewer: MABLE. Our results show that MABLE consistently improves fairness, and its effect is amplified when combined with our proposed post-processing method: EAT. The results are averaged over 5 seeds and use demographic parity as the gender fairness metric.

 | **Dataset** | **Model** | **Method** | **DP ↑** |
 |:---:|:---:|:---:|:---:|
 | Twitter | BERT | Vanilla | 0.955 | 
 | Twitter | BERT | MABLE| 0.965 | 
 | Twitter | BERT | EAT (ours) | 0.993  |
 | **Twitter** | **BERT** | **MABLE+EAT**| **0.994** | 
 | | | | | |
 | Twitter | RoBERTa | Vanilla | 0.958 | 
 | Twitter | RoBERTa | MABLE| 0.984 | 
 | Twitter | RoBERTa | EAT (ours) | 0.967 | 
 | **Twitter** | **RoBERTa** | **MABLE+EAT**| **0.992** | 
 | | | | | 
| Wikipedia | BERT | Vanilla | 0.975 | 
| Wikipedia | BERT | MABLE| 1.00 | 
| Wikipedia | BERT | EAT (ours)| 1.00 |
| **Wikipedia** | **BERT** | **MABLE+EAT**| **1.00** | 
| | | | |  
| | | | | 
| Wikipedia | RoBERTa | Vanilla | 0.974 |
| Wikipedia | RoBERTa | MABLE| 0.998 |
| Wikipedia | RoBERTa | EAT (ours) | 0.997 |
| Wikipedia | RoBERTa | **MABLE+EAT**| **0.999** |
| | | | |
Thanks for your patience. We have obtained the results showing the effect of attention entropy modulation on 7 different downstream tasks using GPT-Neo. The values for all tasks refer to accuracy scores, except for the DROP task which uses F1 score.
| **Beta\Task** | **ARC (0-shot)** ↑| HellaSwag (0-shot) ↑| MMLU (0-shot) ↑| TruthfulQA (0-shot) ↑| Winogrande (0-shot) ↑| GSM8K (5-shot) ↑|DROP (0-shot) ↑|
|:---:|:---:|---|---|---|---|---|---|
| 0.7 (attention entropy max.) | 0.2201 ± 0.0121 | 0.2913 ± 0.0045 | 0.2290 ± 0.0035 | 0.3430 ± 0.0108 | 0.5280 ± 0.0140 | 0.0129 ± 0.0031 | 0.0206 ± 0.0008 |
| 0.8 (attention entropy max.) | 0.2261 ± 0.0122 | 0.2960 ± 0.0046 |  0.2293 ±  0.0035 | 0.3496 ± 0.0108 | 0.5114 ± 0.0140 | 0.0182 ±  0.0037 | 0.0197 ± 0.0007 |
| 0.9 (attention entropy max.) | 0.2287 ± 0.0123 | 0.3009 ± 0.0046 | 0.2302 ± 0.0035 | 0.3511 ± 0.0108 | 0.5099 ± 0.0140 | 0.0136 ± 0.0032 | 0.0199 ± 0.0007 |
| **1 (no attention entropy modulation)** | **0.2312 ± 0.0123** | **0.3040 ± 0.0046** | **0.2323 ± 0.0036** | **0.3570 ± 0.0109** | **0.5043 ± 0.0141** | **0.0159 ± 0.0034** | **0.0211 ± 0.0008** |
| 1.1 (attention entropy min.)| 0.2321 ± 0.0123 | 0.3051 ± 0.0046 |  0.2318 ± 0.0036 | 0.3580 ± 0.0109 | 0.5059 ± 0.0141 | 0.0136 ± 0.0032 | 0.0197 ± 0.0008 |
| 1.2 (attention entropy min.)| 0.2329 ± 0.0124 | 0.3053 ± 0.0046 | 0.2305 ± 0.0035 | 0.3609 ± 0.0109 | 0.5122 ± 0.0140 |0.0220 ± 0.0040 | 0.0191 ± 0.0008 |
| 1.3 (attention entropy min.)|0.2304 ± 0.0123 | 0.3049 ± 0.0046 | 0.2295 ± 0.0035 | 0.3611 ± 0.0109 | 0.5036 ± 0.0141 | 0.0174 ± 0.0036 | 0.0185 ± 0.0007 |

The results show that attention entropy modulation (*i.e.* maximization or minimization) does not lead to substantial degradation on the downstream tasks used, while improving the performance in certain cases.
More specifically, attention entropy minimization leads to better performance on ARC, HellaSwag, TruthfulQA, GSM8K, and Winogrande; while attention entropy maximization improves performance on GSM8K and Winogrande. We will add these results to the manuscript, and we thank the reviewer for the constructive suggestion.