Deciphering Stereotypes in Pre-Trained Language Models
Authors: Weicheng Ma
Henry Scheible
Brian C Wang
Goutham Veeramachaneni
Pratim Chowdhary
Alan Sun
Andrew Koulogeorge
Lili Wang
Diyi Yang
Soroush Vosoughi
Venue: EMNLP 2023 Main (2023)
Keywords: Stereotype Examination
Stereotype Dataset Construction
Probing and Other Interpretations

Abstract: Warning: This paper contains content that is stereotypical and may be upsetting.

This paper addresses the issue of demographic stereotypes present in Transformer-based pre-trained language models (PLMs) and aims to deepen our understanding of how these biases are encoded in these models. To accomplish this, we introduce an easy-to-use framework for examining the stereotype-encoding behavior of PLMs through a combination of model probing and textual analyses. Our findings reveal that a small subset of attention heads within PLMs are primarily responsible for encoding stereotypes and that stereotypes toward specific minority groups can be identified using attention maps on these attention heads. Leveraging these insights, we propose an attention-head pruning method as a viable approach for debiasing PLMs, without compromising their language modeling capabilities or adversely affecting their performance on downstream tasks.

Deciphering Stereotypes in Pre-Trained Language Models
Weicheng Ma', Henry Scheible', Brian Wang', Goutham Veeramachaneni, Pratim Chowdhary', Alan Sun?, Andrew Koulogeorge', Lili Wang'. Diyi Yang', and Soroush Vosoughi
1,2,4 1,24 Department of Computer Science, Dartmouth College Computer Science Department, Stanford University we ceeng.eg..oo ma. gr@dar edu soroush. www.gghhhddarrtotttooo edu
Abstract
knowledge about these stereotypes and entail care- ful manual curation of examples (Nadeem et al., 2021). These charactcristics make such approaches costly and time-consuming to implement. Fur- thermore, they lack the capability to detect newly- emerged or complicated stereotypes, such as in- tersectional stereotypes, thereby highlighting the need for comprehensive, adaptable, and less labor-intensive approach.
Warning:
This paper discusses content that could potentially trigger discomfort due the presence of stereotypes.
This paper addresses the issue of demographic stereotypes present in Transformer based pre- trained language models (PLMs) and aims to deepen our understanding of how these biases encoded in these models. To accomplish this, we introduce an easy-to-use framework for examining sterentype- PLMs through a combination of model prob- ing and textual analyses. Our findings reveal that small subset of attention heads within PLMs primarily responsible for encoding stereotypes and that stereotypes toward spe- cific minority groups
With focus on addressing these limitations, we examine the intricate relationship between stereo- type encoding and detection within PLMs. We propose framework for examining stereotypes in -ransformer-based PL.Ss by conducting attention- head probing. The motivation for using probing arises from the inherent complexity of Transformer models, which hampers their theoretical analysis, and the widespread acceptance of probing methods as effective tool to approximate the functioning of these models in natural language understand- ing (Rogers al., 2020). We further use Shapley values (Lundberg and Lee, 2017) to quantify the individual contributions of different attention heads in stereotype detection, thereby shedding light on their roles in encoding stereotypes. This integration helps in the systematic handling of potential inter- actions among attention heads, thereby allowing to quantify the individual contribution of each head effectively.
behavior of
be identified using attention maps on these attention heads. Lever- aging these insights, we propose an attention- head pruning method debiasing PLMs, without compromising their language modeling capabilities or adversely af- fecting their performance downstream tasks.
viable approach for
1 Introduction
Stereotypes, serving as simplified and generalized representations of socictal beliefs, have turned into a challenging issue in the sphere of natural lan- guage processing (NLP). Their inadvertent encod- ing in pre-trained language models (PLMs) and propagation downstream applications has incited concerns about the faimess and bias of such sys- tems (Choenni et al., 2021; Dev et al., 2022: Lee, 2018). To build unbiased language technologies, is crucial to understand how these models encode and detect stereotypes thoroughly.
Our approach aims to reduce the reliance on manually-curated word-level stereotypical in- stances by combining stereotype detection research with the examination of stereotypes in PLMs. This merger paves the way for using sentence-level stereotype detection datasets that are more eas- ily annotated. Further, our framework facilitates a detailed analysis of stereotypes toward specific minority groups within each PLM by integrating attention analysis with SHAP, a perturbation- based model interpretation method.
A significant body of research has demonstrated the presence of such biases and worked meth- ods to effectively detect them in PLMs. However, these methods fail provide an understanding of the processes underlying stereotype encoding and detection PLMs. Current approaches for exam- ining stereotypes in PLMs demand intricate human
Through a series of carefully designed exper-

---

iments, we uncovered that substantial portion, ranging from 15% to 30%, of attention heads in six Transforrer-based models of various sizes and architectures significantly impact the models' abil- ity to detect stereotypes. This considerable number of influential attention heads underlines the depth and complexity of stereotypes representation in these models. The models we examined cover broad range of architectures, including encoder- only models such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019), as well encoder- decoder models such as T5 (Raffel et al., 2020) and Flan-T5 (Chung et al., 2022).
stereotypes are expressed in the text. This attribute greatly enhances the versatility and practicality of our approach the quest for stereotype-free PLMs.
Background and Our Contributions
Historically, the exploration of intrinsic biases within PLMs has predominantly revolved around comparison-based methods. These methods in- volve contrasting the propensity of PLMs to gener- stereotypical versus non-stereotypical content from similar prompts (Nadeem et al., 2021). For instance, Bartl al. (2020) probe into BERT's gen- der bias by comparing its likelihood of associating a list of professions with pronouns of either gen- der. Similarly, Cao et al. (2022) delve into social bias by contrasting the probabilities of BERT and RoBERTa, associating adjectives that describe dif- ferent stereotype dimensions with various groups.
Our experiments revealed a significant correla- tion between the attention heads that play a crucial role in detecting stereotypes and those involved in encoding these stereotypes. Notably, our attention- head ablation experiments demonstrated that these heads exhibited a distinct preference for stereotypi- cal expressions when processing language. Such a finding not only underscores the robustness of our approach, which uses stereotype detection datasets to explore intrinsic biases in PLMs but also offers practical and efficient strategy for debiasing these models through targeted attention-head pruning.
It would be even more costly to design multiple stereotypical contents for each minority group and annotate sentence pairs accordingly for identifying and assessing stereotypes in different PLMs. De- spite their utility, such assessment methodologies come with inherent limitations, particularly when applied to the task of debiasing PLMs. Firstly, they necessitate costly parallel annotations (com- prising both stereotypical and non-stereotypical sentences pertaining to an identical subject), which are not readily expandable to accommodate emerg- ing stereotypes. As Hutchison and Martin (2015) noted, stereotypes evolve in tandem with cultural shifts, making it critical to align stereotype anal- ysis with current definitions and instances. Sec- ondly, the assumption that stereotypes are solely attributable to specific word usage oversimplifies the complex nature of biases. PLMs might favor particular words not because of inherent biases but due to their contextual prevalence. This com- plexity, coupled with the implicit nature of biases (Hinton, 2017), challenges the efficacy of exist- ing stereotype assessment approaches. Lastly, pre- vailing stereotype-evaluuaion benchmarks assume uniform types of stereotypes across all PLMs, an assumption that is not necessarily valid. Design- ing multiple stereotype instances for each minority group and annotating corresponding sentence pairs would impose even greater cost.
Our framework also offers a more granular per- spective by breaking down stereotypes by types and targets. We analyzed the behavior of the most im- pactful attention heads, as identified by our probing experiments, using SHAP and attention analyses. This allowed us examine stereotypes exhibited by each PLM toward five frequently stereotyped groups: aged people, females, Muslims, African people, and Middle-Eastern people. Our detailed analysis uncovered a shared set common sterco- types across all the PLMs. However, we also ob- served distinct stereotypical expressions associated with each minority group within different PLMs. This discovery underscores the necessity of leverag- ing diverse instances when evaluating stereotypes in different PLMs for fair assessments and effec- tive stereotype reduction - an aspect that current research has often overlooked.
The robustness of our results, which hold across variations in random-seed selection, dataset uti- lization, and PLM and checkpoint choices, further validates our approach and attests to the reliability of our findings. Perhaps most importantly, our anal- yses can be effortlessly extended to other PLMs or stercotypes with other types or targets, eliminating the need for pre-assumed knowledge about how
In this paper, we propose innovative approach that bridges the gap between the assessment of stereotypes encoded in PLMs and the models' stereotype detection capabilities. Our approach

---

leverages datasets that are more readily annotated and modified, as they do not demand parallel anno- tations or preconceived stereotypical expressions. Furthermore, our method operates at the sentence level rather than the word or phrase level, facili- tating more versatile evaluations of stereotypical expressions or implicit stereotypes. To address the existing research and dataset gap on implicit stereo- types, we introduce a manually-validde vaalidttd implicit stereotype dataset generated using ChatGPT. Our framework enables the identification of stereotyp- ical expressions within each PLM and the exam- ination of the similarities and differences in how stereotypes are encoded across various PLMs. en- abling deeper understanding of the unique stereo- typical tendencies inherent in different models.
text. This enables them generate implicit stereo- types with relative ease compared to human anno- tators, who may find the task more challenging. While the dataset constructed for this paper is lim- ited in its scope, the methodology can be scaled to create more expansive datasets covering broader range of minority groups and stereotypes. The dataset generation procedure comprises three main steps: (1) Initially, ChatGPT is queried to generate list of common stereotypes associated with 17 demographic groups, using the prompt specified in Appendix Figure A2a. (2) In the subsequent stage, each target demographic group and its cor- responding stereotypes are fed into ChatGPT. The model is instructed to produce five instances of im- plicit stereotypes for each group-stereotype pairing, utilizing the prompt detailed in Appendix Figure A2b. (3) Finally, ChatGPT is employed to de-bias each generated example, resulting in a set of 425 non-stereotypical instances. These instances un- dergo manual verification to ensure their quality and fidelity.
3 Datasets for Stereotype Examination
Our study deploys three extensively used datasets to investigate stereotypes in PLMs: (1) StercoSet (Nadeem et al., 2021), (2) CrowS-Pairs (Nangia et al., 2020), and (3) WinoBias (Zhao et al., 2018). However, these existing datasets are not without issues, including the occasional unnaturalness of sentences constructed via word substitution in sen- tence pairs and the presence of instances that are in- correctly classified or not truly stereotypical (Blod- gett et al., 2021). Moreover, these datasets tend to oversimplify stereotypes by restricting examination to short sentences where stereotypes are expressed explicitly through few words syntactically tied to the subject (i.e., explicit stereotypes).
Following the removal of duplicate and noisy entries through manual curation, we assembled the ImplicitStereo dataset. This dataset comprises 416 stereotypical and 374 non-stereotypical instances, targeting 17 distinct demographic groups. We con- ducted manual validation to assess the dataset's quality by randomly sampling 100 pairs of stereo- typical and de-biased instances. Validators were tasked with determining whether each stereotypi- cal instance overtly presented prevalent stereotypes about a given demographic group and whether its de- biased counterpart was entirely free of such stereotypes. In 95 out of 100 sampled instances, at least two out of three validators concurred that the instances accurately encapsulated common stereo- types toward the demographic groups in question. For 86 out of the 100 instances, a similar major- ity agreed that all stereotypes were effectively re- moved in the de-biased samples without affect- ing the unrelated content. Inter-annotator agree- ment for these validation tasks was quantified using Fleiss' k (Fleiss, 1971), with scores exceeding 0.74 for both sets of validations. This attests to the high quality of the ImplicitStereo dataset. Given these promising results, the ImplicitStereo dataset has been incorporated into our primary experiments alongside three publicly available datasets. Exam- ples from all datasets are provided in Table 1.
Our investigations reveal that existing datasets on stereotypes often conflate stereotype represen- tation with negative sentiments or emotions. This conflation limits nuanced analysis of stereotypes, as it reduces their complexity to mere emotional charge. For comprehensive discussion of these limitations, please refer to Appendix A.1. To address this shortcoming, we introduce a novel dataset focused on "implicit stereotypes" I, This dataset generated using ChatGPT, which facili- tates the extraction of more subtle and contextually embedded stereotypes resembling those found in natural language dialogues. We use large language models like ChatGPT for dataset construction pri- marily because they possess extensive training data, encompassing real-world conversations and online
We define "implicit stereotypes" as textual instances where stereotypical beliefs are embedded in the context, rather than explicitly stated through syntactically dependent descrip- tive words.
Our methodology for stereotype assessment em-

---

Dataset
Instance
Label
The chess player asian.
Stereolypical
StereoSet
The chess player was hispanic.
Anti-Sterettppicl
Asians cant drive because they have slanted eyes.
Stereotypical
White people cant drive because they have slanted cyes.
CrowS--Piirs
Whiles are not good at reading.
Anti- Steeeolleel
Hispanics are good at reading.
The [developer] argued with the designer because [he] not like design.
Stereolypical
WinoBias
The [developer] argued with the designer hecause [she] did not like design.
Attr-Stterolypical
[Target: Muslim] You'l re Muslim and you to be chemist? That's interesting choice.
Stereotypical
ImplicitSterco
[Target: Muslim] You're re a Mushim and intereseed becomung chemust?
Non-Stereotypical
That's i a fascinating career path.
Table 1: Example instances from the StereoSet, CrowS-Pairs, WinoBias, and ImplicitStereo datasets. CrowS- S-Paiss provide pairs of moreesterrottyical and less-stereotypical sentences for each instance. WinoBias provides additional annotations about the stereotypical pronouns and the professions they refer to.
ploys the approach introduced by Nadeem al. (2021), evaluating the propensity of each PLM to favor stereotypical sentences over anti- or non- stereotypical sentences in a natural language entail- ment setting, This inter-sentence approach aligns more closcly with situations where the stereotype is not solely dictated by a single word. The intra- sentence approach, which examines word choices within the same sentence structure, is not applica- ble for annotation in ImplicitStereo.
dian, Middle Eastern), and religion (Christian, Mus- lim, Jewish). We conduct separate analyses of the stereotypes in PLMs toward specific minor- ity groups within each category in our textual-clue examinations.
We split StereoSet, Crows-Pairs, and Implicit- Stereo into training (80%) and testing (20%) sets. The training portions of StereoSet, Crows-Pairs, and ImplicitStereo, along with the development of WinoBias, are employed for probing and fine-tuning stereotype detection models. The test portions of all datasets are reserved for evaluating the intrinsic biases present in PLMs. The metric in the probing and ablation experiments is accuracy.
In the training, evaluation, and probing of stereotype detection models, we utilize sentence- level stereotypical vs. anti-stereotypical (or non- stereotypical) annotations instead of sentence pairs. These labels are readily available as ground-truth annotations in StereoSet, WinoBias, and our Im- plicitStereo dataset, and we employ them in their original form for the stereotype detection task. CrowS-Pairs provides pair of sentences, one more stereotypical and the other less so, for each stereotypical or anti-stereotypical instance. For the stereotype detection task, we label the more stereotypical sentence of stereotypical instances as stereotypical and the less stereotypical sentence of anti-stereotypical instances anti-stereotypical. StereoSet features annotations for four types of stereotypes: gender, race, religion, and profession. CrowS-Pairs covers a broader spectrum, encom- passing race/color, gender/gender identity or ex- pression, socioeconomic status/occupation, nation- ality, religion, age, sexual orientation, physical ap- pearance, and disability. WinoBias focuses on gen- der biases that influence pronoun choice (c.g., he vs. she) about 40 types of professions. Implicit- Stereo contains stereotypes under five categories: age (young or old), gender (male, female, or gen- derqueer), ethnicity (black, white, Asian, Hispanic), country of origin (French, American, Arabic, In-
4 Models for Stereotype Examination
Our research focused on investigating the encod- ing and detection of stercotypes in Transformer- based PLMs. To investigate the influence of dif- ferent pre-training corpora and objectives on the biases present in PLMs. our main experiments uti- lize the BERT and RoBERTa models. Despite be- ing similar in size, these two models have been pre-trained using distinct corpora and objectives, offering valuable contrast in our study?. To ensure the robustness and general applicability of our find- ings, we further examine four other models. These include both the small and base versions of T5 and Flan- Flam-T5, thereby covering broader range of model architectures. We use the Huggingface implemen- tation of all the models (Wolf ct al., 2019).
5 Understanding Attention Heads' Role in Stereotype Detection
This section dives into the role of attention heads in Transformer-basedd PLMs for detecting stereo-
We use DERTT-hasc-uncused and RoBERTa-base.

---

types in the text. We utilize Shapley-value-bee probing approach discern their contributions.
Cunds-Pair
Winnilio
ImpliziStero
5.1 Shapley-Vaie Seaery-VlleB-Basse Based Probing
To estimate the contribution of each attention head in Transformer-baset PLMs towards stereotype de- tection, incorporate a method introduced by Castro al. (2009), utilizzng Shapley values (Hart, 1989). The Shapley value quantifies the contribu- tion of cach attention head, making it a suitable choice for interpreting these models' performances in abstract tasks, such as stereotype detection. It al- lows to understand the incremental performance gains each attention head offers when working in combination with others (Ethayarajh and Jurafsky, 2021).
T5emmll Roliiiith-boo
75-bat
Fha-15--uall
Flam- 15-----
In this process, we keep the encoder weights of each PLM static, training only a shallow classifier on top of the PLMs (or the decoder for the T5 and Flan-T5 models) to predict stereotypes. We provide details about the upproximation of Shapley value for each attention head (Sh; for Head i) in Algorithm 1. We define N the set of all attention heads in model, O as permutations of N, and v ((N) [0, 1] value function such that (S)) (S C N) is the performance of the model on a stereotype detection dataset when all heads not in S are masked.
Figure 1: Probing results of six Transformer models four datasets, greener color indicates more positive Shapley value, and red color indicates a more negative Shapley value. The and axes of each heatmap refer to layers and attention heads on each layer, respectively.
5.2 Stereotype Detection Probing in PLMs
The results of the probing experiments across six PLMs and four datasets are displayed in Figure The most contributive attention heads (represented by the deepest green cells) are typically found the higher layers (c.g., layers 9 12 for BERT and RoBERTa models). This aligns with our expec- tation that high-level linguistic phenomena, like stereotypes, would involve the encoding of abstract semantic features, which largely handled by the higher layers (Jo and Myaeng, 2020).
Algorithm Shapley-based Probing
Require: m: Number Samples
N| for i V do
Count <- Sh; + 0
while Count < do
Select ()V with probability 1/m!
all N do
Pre' (0) (O(1), Ofk 1) Okk)
Sh: Sh; + (
We verify the probing results by performing abla- tion experiments, evaluating how the PLMs' perfor- mances change when the most least contributive attention heads are pruned. All models are fine- tuned under the same conditions: batch size (64), learning rate (5e-5), and number epochs (5). The results from these ablation experiments (shown in Figure 2 for BERT and Appendix C for other mod- els) suggest that small subset of attention heads (approximately 15% to 30% of the highest-rankedd predominantly contribute stereofype detection. Pruning heads with a negative or slight positive con- tribution typically result minimal performance drops or enhancements, supporting the validity our probing results.
end for
Count + Count +
end while
Sh, Sh,
end for
We conduct these probing experiments for every attention head in each PLM, and the results are visualized as heatmaps. The BERT-based probing results exhibit robustness regardless of variations in sampling sizes, random seed choices, or probing settings, as indicated in Appendix B. Thus, for all subsequent experiments, we maintain the same sampling size (250), random seed (42), and probing setting (freezing encoder weights).

---

20
Fuday Dried
of Flaee Aberoonteec
(a) CrowS-Pairs
(b) StereoSet
(a) CrowS-Pairs
(b) StereoSet
Febtro Deeee
Pradag
13 Ober
(c) WinoBias
(d) ImplicitStereo
(c) WinoBias
(d) ImplicitSteren
Figure 2: Evaluating the impact of attention-head abla- tion on BERT's s performance across four datasets. Prun- ing is done from the least (bottom-up) and most (top- down) contributive heads.
Figure 3: The impact of bottom-up attention-head prun- ing on BERT's performance, using CrowS-Pairs, Stere- oSet, WinoBias, and ImplicitStereo datasets. Each ex- periment was repeated three times, using attention-head contributions obtained from the other three datasets.
We repeat the ablation experiments for the de- coders of T5 and Flan-T5 models, observing min- imal changes in performance during this process. As shown in Appendix D, the relative performance shifts for each T5 or Flan-T5 model vary between 0.98% to 3.99%, even when up to 100% of the at- tention heads are pruned from the decoder. These results imply that for these four encoder-decoder models, stereotype detection is primarily handled by the encoder. Therefore, we maintain the de- coders of these models and experiment solely with their encoder weights.
similar functionalities (Bian et al., 2021).
Stereotypical Encoding by PLMs
Here, we analyze the relationship between the attention-head contributions in both the detection and encoding of stereotypes within PLMs. We use the stereotype score (ss) (Nadeem et al., 2021), to gauge the level of stereotyping within PLMs, and the language modeling score (Ims) to measure their linguistic proficiency. Building on Nadeem et al. (2021), we use the idealized CAT score (icat) to assess PLM's ability to operate devoid of stereo- typing, which combines ss and lms. A model that scores high the iCAT metric suggests that it retains a commendable language modeling ability while significantly reducing its stereotyping ten- dencies. The attention-head rankings for all exper- iments here are obtained from the ImplicitStereo dataset to mitigate the impact of other psycholin- guistic signals, such as sentiments and emotions.
It's worth noting that, within the same PLM, attention-head contributions can vary between datasets, with Spearman's 's rank correlation (p) rang- ing from 0.10 to 0.42.3 To examine the transfer- ability of our findings across datasets, we repeat the attention-heal ablation experiments, using dif- ferent datasets used to gather attention-head contri- butions and to fine-tune and evaluate the PLMs.
Figure 3 demonstrates that when the least con- tributive heads- on rankings obtained from other datasets- are pruned, performance remains relatively stable: however, when the most contribu- tive heads are removed, there is a noticeable drop in performance. These results highlight that irre- spective of the dataset used to determine attention- head rankings, similar set of heads in each PLM contributes to stereotype detection. Variations in these rankings likely due to differences in the attention head sampling methods used in the prob- ing process, as many heads in PLM often have
6.1 Debiasing PLMs via Head Pruning
We hypothesize that if the attention heads contribut- ing to stereotype detection are also instrumental in expressing stereotypical outputs in PLMs, remov- ing these heads should result in superior icat score. This is corroborated by the evidence in Figure 4, where pruning the attention heads that are most contributive stereotype detection consistently improves icat scores across all models tested.
In one extreme scenario, the removal of 62 at- tention heads from the T5-base model achieves a
p's are statistically significant unless otherwise specified.

---

M5
43
0005
32
(a) BERT
(b) RoBERTa
(c) TS-small
-208
8315
110
Nu Prones Alber 1iaad
Number Thundd on
(d) T5-base
(c) Plan-T5-small
(1) Blan-TS-base
Figure 4: The ss, Ims, icat, and Shapley values of attention heads in six models when the attention heads contributing most significantly to stereotype detection are pruned. The green horizontal line represents the icat score obtained by the fully operational models, while the orange horizontal line corresponds to ss of 50, signifying entirely unbiased model. The green vertical line denotes the point which each model achieves optimal icat score.
CoLA
SST-2
MRPC
STS-B
MNLI-m
MNLI-mm
QNLI
RTE
BERT-full
56.60
93.35
89.81/85.54
89.30/88.88
83.87
84.22
91.41
64.62
BERT-pruned
59.89
92.66
87.02/81.13
88.40/88. 00
81.73
82.07
90.43
61.37
RoBERTa-rall
61.82
93,92
92.12/88.97
90.37/90.17
87.76
87.05
92.64
72.56
RoBERTa-prunne 59.81
93.69
89.35/84.80
90.44/90.21
86.79
86.95
92.29
67.15
Table 2: Evaluation of the original BERT and RoBERTa models (BERT-full and RoBERTa-full), alongside the same models with attention heads pruned based probing results (BERT-pruned and RoBERTa-pruned) using the GLUE benchmark. The metrics reported include Matthew's correlation coefficients for CoLA, accuracy for SST-2, MNI.Immtchhed (MNLI-m), MNL.Imismmtched (MNLI-mm)) QNLI, and RTE, both accuracy and FI-score for MRPC, and Pearson's 's and Spearman's 's correlation coefficients for STS B. The best pest-performing scores for each model are highlighted in bold.
3.09 icat score increase while reducing the ss to a mere 0.11 away from 50, the ideal ss for a non- stereotypical model, with the Ims also improving. In the case of the Tlan-T5-base model, the optimal icat score without detriment to Ims is attained by pruning 11 attention heads. However, even better icat scores are achieved later when 131 heads are pruned, as a significant drop in ss compensates for the loss in lms.
allow their language modeling heads to predict nu- merical labels, which might negate the effects of attention-heal pruning on model performance.
As shown in Table 2, the pruned models ex- hibit similar, if not better, performance than their full counterparts across most tasks. The most pronounced performance drops are observed on the MRPC and RTE datasets. It is plausible that the smaller datasets for these tasks, which require higher-level semantic understanding, cient for training the models to high performance. Consequently, additional active pre-trained atten- tion heads that encode semantic information are necessary to capture relevant information from the text. In comparison, the other tasks similar to MRPC (STS-B) and RTE (QNLI and MNLI) in the GLUE benchmark feature larger sizes or sim- pler task objectives, reducing the models' reliance more pre-trained weights.
Subsequently, we assess the GLUE benchmark (Wang al., 2018) performance of the head-pruned models to ascertain that the removal of these heads does not significantly impair the PLMs' perfor- mance on downstream tasks. Due to known issues with dataset splits, we exclude the QQP and WNLI datasets 4, Only the BERT and RoBERTa encoder models are evaluated here, the other four gen- erative models would require intensive training to
insuffi-
The results of our attention-head ablation experi-

---

ing method for debiasing PLMs. It's important to note that the gap between the heads that nega- tively impact icat when pruned and those encoding stereotypes (as depicted in Figure 4) is quite evi- dent and can be managed through Ims evaluation trials. These trials can help set an empirical limit on how much pruning can be done without excessively hurting language modeling capabilities, ensuring careful balance between bias mitigation and lan- guage understanding performance.
110
(a) BERT
(b) RoBERTa
Figure 5: Comparison of 8s, Ims, icat, and Shapley val- ues of attention heads in BERT and RoBERTa models when the most contributive attention heads for stereo- type detection in the alternate model are pruned. The green horizontal line indicates the icat score achieved by the unmodified models, and the orange line denotes ss of 50, symbolizing a completely unbiased model.
6.2 Cross-Model Transferability
We examine whether the attention-heal contribu- tions obtained from one PLM can be utilized to debias other PLMs of the same size and architec- ture. To this end, we perform the ss, Ims, and icat evaluation with attention-head ablations on RoBERTa using the attention-head contributions acquired from BERT and vice versa. As shown in Figure 5, the effects of pruning attention heads most contributive detecting stereotypes in differ- ent models do not consistently improve the icat as effectively as using the attention-heat ranking of the same model. This is expected, given that differ- ent PLMs are pre-trained with differing objectives and corpora, leading to disparate functionalities for attention heads in identical positions. Nonetheless, higher icat scores than the full models are achieved when 29 and 37 attention heads are pruned from the BERT and RoBERTa models. Our results suggest the potential transferabillty of attention-heat con- tributions across different models in both encod- ing and detecting stereotypes. This could indicate that similar linguistic features trigger stereotyp- ing across different PLMs, as significant attention heads for encoding linguistic features typically side in the same adjacent layers for PLMs of the same size (Rogers et al., 2020).
ments suggest that the attention heads that most significantly contribute to stereotype detection are also instrumental in encoding stereotypes within PLMs. This discovery facilitates integrating stereo- type detection research with stereotype assessment and PLM debiasing, reducing the need to annotate pairwise stereotype assessment or manually curate word-level stereotype datasets. Our approach of pruning the attention heads most contributive to stereotype detection offers an efficient method to reduce bias in PLMSs without requiring re-training. This can be complemented with other debiasing methods to further minimize stereotypes in PLMs while preserving their linguistic capabilities.
Furthermore, in Appendix B.4, we demonstrate that the attention-heat rankings procured from a model can be employed to prune and debias differ- ent checkpoints of that same model. This robust- ness serves as further proof of the generalizability of our approach and hints towards direction of transferable, adaptable debiasing that can stream- line the process of bias reduction in multiple ver- sions of a model.
However, it is important to acknowledge that the impact of head pruning on icat may not be consis- tently beneficial, particularly when the heads to be pruned are contributive to stereotype detection (i.e.. they possess positive Shapley values in the probing results). Our observations suggest that some atten- tion heads encoding lexical features (for instance, the presence of overtly stereotypical words) may achieve low positive Shapley values they aid PLMs in identifying explicit stereotypes. Upon re- moval of these heads, a drop in icat may occur due to the negative effect on the language modeling capacity of the PLMs. Yet, this should not un- dermine the usefulness of the attention-heel prun-
Analyzing Textual Clues
Here, we undertake an analysis of the textual clues that significantly influence the inferences of inher- ent stereotypes within each PLM. This examina- tion arises from our previous experimental findings that the same attention heads are crucial to both encoding and detecting stereotypes. We focus on analyzing the most attended single words and word pairs in the top-5 attention heads revealed by our probing experiments, averaging the attention across these heads. We use SHAP to reduce poten- tial noise (such as high attention scores on function words or special tokens).

---

Groups
BERT
RoBERTa
terrorists" for Muslims, implying all Muslims are terrorists. The anti-stereotypicll words and word pairs we find are mostly antonyms of the associated stereotypical words, such as "nice" and "caring," which these models use to identify anti-stereotypes towards Middle-EEstenn people.
accent foolish
pest disgusted gross bankrupt struggled
pest fat
Aged
disgusted
shooting
worthless fat
People
worthless idiot
accent
gross
bankrupt
knitting
idiot
lonely
rude
rude
busy
distracted
gossip
lonely
gossip
Females
poor
immature
poor
distracted sexy
sexy
mothers
mothers
We also note some variability across the mod- els in our study, despite many models highlight- ing common stereotypical words and word pairs. One potential reason for these differences could be the size of the models, as larger models might capture more instances of co-occurrence between stereotypical words and minority group mentions. For instance, T5-base and Flan-T5-base rank "al- cohol" and "urine" highly for the elderly, whereas their smaller counterparts do not. Variations can also stem from differences in pre-training corpora and objectives between PLMs with the same archi- tecture and structure. For example, BERT high- lights "violent" and "pirates" for African people and "bombs" for Middle-Fastee RoBERTa does not. Similarly, RoBERTa ranks "bankrupt" and "gross" much higher than BERT for stereotypes towards the elderly.
silly
worried
knit
burst
terrorists
everyone
lerroniss
violent
terrorist
violent
terrorist
kill
Muslims
kill
threats
threats
islam
islam
hated
hated
family
religious
family
religion
arabia
violent
hate
religious
terrorist
pirates
slaves
crude
dirty
African
dirty
smell
hate
annoying
People
muslims
forced
forced
lazy
crude
criminal
ignorant
slaves
rude
hate
blame
rough
Middle-
oppressive
racist crime rude
Eastern
rough
violent bombs crime
corrupt dirty
People
serious
oppressive racist
terrorists
terrorism
destruction
Table Top 10 words with the highest SHAP-adjusted attention scores (ranked in order) for detecting stereo- types towards five minority groups, ranked by the top-5 contributive attention heads BERT and RoBERTa.
people, while
We use the attention-heaal contributions obtained from ImplicitStereo to avoid biasing the textual analyses since ImplicitSteroe does not pre-assume any word to be always stereotypical toward a group. We combine all four datasets, selecting instances related to five minority groups (specifically, the ciderly, females, Muslims, African, and Middle- Eastern people) for our textual-clue analysis.
These findings indicate that different PLMs might harbor diverse stereotypes towards minority groups. Therefore, using the same dataset to assess stereotype levels across all PLMs might underes- timate certain stercotypes. The combined appli- cation of our probing technique and textual-clue analysis framework could aid in identifiig the most pronounced stereotypes within each PL.M.
For illustrative purposes, we present the single- word textual analysis results on BERT and RoBERTa in Figure 3. We reserve the complete set of results for Appendix E. Our analyses re- veal that the 10 word-level textual clues align with common stereotypes towards the five minority groups. For instance, words like "fat", "bald", and "bankrupt" stand out for age stereotypes; "gossip" and "silly" for gender stereotypes; "violent", "kill", and "terrorists" for religious (Muslims) and racial EMstele-Eastern people) stereotypes; and "dirty"". "crime", and "crude" for racial (African people) stereotypes. This is consistent with prior studies on the prevalence of stereotypes about minority groups in PLMs. For example, Abid et al. (2021) discovered that GPT-3 (Brown al., 2020) gen- erates stereotypical associations of Muslims with phrases like "shooting at will" and "bombing." Our word-pair analyses also present similar rankings but with word pairs usually expressing stronger and more stereotype-laden meanings, such as "all
Conclusion and Future Work
In this paper, we sought to deepen the understand- ing of the connection between the encoding and detection of stereotypes within PLMs. We per- formed extensive probing and ablation studies and, informed by the results, developed framework explore the intrinsic stereotypes within each PLM. This framework leverages both textual and atten- tion analyses and solely relies on stereotype detec- tion annotations. Our study unveils that stereotypes are not uniformly distributed across different PLMs. This highlights the need for model-specific stereo- type assessment datasets and tailored debiasing techniques. Our framework introduces efficient means of debiasing PLMs without restructuring retraining and without the need for expensive pair- wise steredtype/antissterrttyy annotations. Future research can potentially merge our methods with other PLM debiasing techniques.

---

9 Limitations
velop more equitable and unbiased Al models, and we appreciate the contribution of all participants involved.
This study has certain limitations. Firstly, our ex- periments were primarily conducted in English due to the scarcity of stereotype assessment datasets in other languages. While our framework is capable of handling complex scenarios such intersec- tional stereotypes, we were unable to explore these due to a lack of udequately annotated datasets.
References
Abubakar Abid, Maheen Farooqi, and James Zou. 2021. Persistent anti-muslim bias in large language models. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pages 198-306.
We also constructed an implicit stereotype dataset using ChatGPT alongside three publicly available datasets. We pursued this approach be- cause existing stereotype examination datasets of- ten oversimplify the task and have known quality concerns, indicated by Blodgett al. (2021). Our dataset addresses several issues of prior ones, such as unnatural phrasing, overly explicit stereo- type expression, and excessive intertwining of stereotypes with negative emotions and sentiments. Nonetheless, we recognize potential noise in the CharGPT-genneatte data, despite our careful man- ual curation and high-quality human validation re- sults via Amazon Mechanical Turk. Unvalidated data points may still contain biases incorrect in- formation from ChatGPT, which could influence our results. Further, our dataset doesn't I completely rectify the problems with existing datasets. How- ever, we view ImplicitStereo as an additional data source, providing a more equitable examination of stereotypes in PLMs, and any potential biases from ImplicitStereo should not significantly impact our analyses.
Marion Bartl, Malvina Nissim, and Albert Gatt. 2020. Unmasking contextual stereotypes: Measuring and mitigating BERT's s gender bias. In Proceedings of the Second Workshop on Gender Bias in Natural Language Processing, pages 1-16, Barcelona, Spain (Online), Association for Computational Linguistics.
Yuchen Bian, Jiaji Huang, Xingyu Cai, Jiahong Yuan, and Kenneth Church. 2021. On attention redundancy: A comprehensive study. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 930-945, Online. As- sociation for Computational Linguistics.
Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, and Hanna Wallach. 2021. Stereotyping Norwegian salmon: An inventory of pitfalls in fair- ness benchmark datasets. Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers). pages 1004-1015, Online. Association for Computational Linguistics.
Proceedings of the 59th
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
10 Ethics Statement
This research tackles harmful stereotypes present in widely used pre-trained language models. Our aim is to identify and reduce these biases. We acknowledge that our analysis could be distressing offensive some, and have therefore included warning the outset of this paper. To ensure ethical practices, we relied on publicly available stereotype datasets, thereby avoiding exposing annotators to potentially harmful language.
Yang Cao, Anna Sotnikova, Hal Daumé III, Rachel Rudinger, and Linda Zou. 2022. Theory-grounder measurement of U.S. social stereotypes in English language models. In Proceedings of the 2022 Con- ference of the North American Chapter of the As- sociation for Computational Linguistics: Human Language Technologies, pages 1276-1295, Seattle, United States. Association for Computational Lin- guistics.
We also enlisted human validators from Ama- zon Mechanical Turk to validate the ImplicitStereo dataset we constructed and to understand the im- portance of constructing higher-qualit, and more challenging implicit stereotype dataset. The dataset was examined beforehand. The validators received an hourly rate of $15.00, greatly surpassing the minimum state and federal (in the U.S.) hourly wage. This research is driven by the quest to de-
Javier Castro, Daniel Gómez, and Juan Tejada, 2009. Polynomial calculation of the shapley value based on sampling. Computers & Operations Research, 36(5):1726-1730.
Rochelle Choemni, Ekaterina Shutova, and Robert van Rooij. 2021. Stepmothers are mean and academics are pretentious: What do pretrained language models learn about you? In Proceedings of the 2021 Confer- ence on Empirical Methods in Natural Language Pro- cessing, pages 1477-1491, Online and Punta Cana,

---

Dominican Republic, Association for Computational Linguistics.
Scott M Lundberg and Su-In Lee. 2017. A unified ap- proach to interpreting model predictions. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and Garnett, editors, Advances in Neural Information Processing Systems 30, pages 4765 4774. Curran Associates, Inc.
Hyung Won Chung, Le Hou, Shayne Longpre, Bar- ret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finetune language models. arXiv preprint arXiv:2210.1 11416.
Moin Nadeem, Anna Bethke, and Siva Reddy 2021. StereoSet: Measuring stereotypical bias in pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5356-5371, Online. Association for Computational Linguistics.
Sunipa Dev, Emily Sheng, Jieyu Zhao, Aubrie Amstutz, Jiao Sun, Yu Hou, Mattie Sanseverino. Jiin Kim, Ak- ihiro Nishi, Nanyun Peng, and Kai- Chang. 2022. On measures of biases and harms in NLP. In Find- ings of the Association for Computational Linguis- tics: AACL-IJCNLP 2022, pages 246-267, Online only. Association for Computational Linguistics.
Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020. CrowS-par ShowS-pairs: A chal- lenge dataset for measuring social biases in masked language models. In Proceedings of the 2020 Con- Serence on Empirical Methods in Natural Language Processing (EMNLP), pages 1953-1967, Online. As- sociation for Computational Linguisties.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. The Journal of Machine Learning Research. 21(1):5485-555)
Kawin Ethayarajh and Dan Jurafsky. 2021. Attention flows are shapley value explanations. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1ih International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 49-54, Online. As- sociation for Computational Linguistics.
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8:842-866.
Joseph L Fleiss. 1971. Measuring nominal scale agree- ment among many raters. Psychologicul bulletin, 76(5):378.
Thibault Sellam, Steve Yadlowsky, Ian Tenney, Jason Wei, Naomi Saphra, Alexander D' Amour, Tal Linzen. Jasmijn Bastings, Iulia Raluca Ture, Jacob Eisen- stein, et al. The multiberts: Bert reproductions for robustness analysis. In International Conference Learning Representations.
Sergiu Hart. 1989. Shapley value. In Game theory, pages 210-216. Springer.
Perry Hinton. 2017. Implicit stereotypes and the predic- tive brain: cogmition and culture in "biased" person perception. Palgrave Communications, 3(1):1-9.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-tas task benchmark and analysis platform for nat- ural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLPP. Analyzing and Interpreting Neural Networks for NLP, pages 353-355, Brussels, Belgium. Association for Com- putational Linguistics.
Jacqui Hutchison and Douglas Martin, 2015. The Evo- Springer Inter-
lution of Stereotypes, pages national Publishing, Cham.
Jae-young Jo and Sung-Hyon Myaeng. 2020. Roles and utilization of attention heads in transformer-baser neural language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3404-3417, 3417, Online, Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz. et al. 2019, Huggingface's transformers: State-of- the art natural language processing, arXiv preprint arxiv: 1910.03771.
Nicol Turner Lee. 2018 Detecting racial bias in algo- rithms and machine learning. Journal of Information, Communication and Ethics in Society.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Or- donez, and Kai- Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume (Short Pupers), pages 15-20, New
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: robustly optimized bert pretraining ap- proach. arXiv preprint Xixi:vv:7 11692.

---

Orleans, Louisiana, Association for Computational Linguistics.

---

A
The ImplicitStereo Dataset
Show ma commonll sown drowrrypas tangeting Mas trw, Neve invett sterectypun
A.1 Shorteomings of Existing Stereotype Datasets
pour respoer, but samphnnaio that asansalposa harrrfull
porsctials els be tna disoiin disoiiiiaaionn ination it's reportal bart approect this baske with sorisilvityy accurately oresent the divers and complesity of any rel gloue ethnicc group, incluciin Mueline.
recogniie sterecty
Howemver can provice you whth list connenity iir srsw strewwrtpps thaet howe not accunew portragiia in and anculd not be saed judge or generrll on aboutt and attuate
Flip the emetionali valenoe of the sentence "Wo could bust the Emhlopian that hes was starwing" Keep the abereotysa the sentenoe and the amerioonal arousel unchaanged. Make only min chanche Explain your generation
Mudlira tarmeraae Thes stanwaigges eronggy susarmes that at tikal
witoono and bor noriom. diarogara ng the the magorit of are
posseffu duals wefrrdennnnee such acta.
"We could sull looking the Ethioplan he dreaming of fuest"
Musliine oppessee: stercotype sug pests all Musiin wornnn opur andiac- spency tgnortig the othonne reperiencee and perpective or Moutlm womenn
"his sentence maintaino the steree that the individrual coeen't have enough eat, out files the emotionsl valence, instaad the despair heheent in starvttion, there is positive emotion hope and pnticipation deplcted in creaming at fesst. emotional arousel the both circumetences- starcation and creaming feest immore strong emotions. ohey changes made are the mooiaommnnt of "standing" with "creamingg af feoee" which keeps the ovenall structure of the sertienee comeisient with the original.
Mudiira are bwclowand or unoivtt owet This atareatt parttat portrage blux iim so primittive uneducisss, arsparting ni
recdern
Muslins Weestere This nonooop
fitualiers fostills towardds Wasttm cultures waluess disropording oo foot that many Must lims two poocefi and harmoniourt Whetern societees
Mu new foratico entremben This steeeoly wnor ant portray bludiima extverss shoreganding the veretty bwillnde and precticae withi community
Figure AI: Example of using ChatGPT to flip the emo- tional valence of a stereotypical sentence from Stere- oSet.
It's sosuntad is chadiar andiquusi on theae stannoo and to necegniant fuot Uhai indioabuaaia judgedlor
churacter nather affilation, Build rg understandi ng and on popttty the cdlucation und ckalogue si crucial
glous
combating stereotypes promising inclustvity
There are known problems in existing datasets for stereotype examinations, e.g., the instances are all short and simple sentences and the stereotypes are carried out by single words or short phrases. We conduct additional experiments to see if this simplified setting reduces the stereotype analysis problem to simpler ones, e.g., sentiment analysis and emotion recognition. Specifically, we sample 50 stereotypical instances from the CrowS-Pain StereoSet, and WinoBias datasets, rewrite them into sentences with the same stereotypes but differ- ent sentiment polarity or emotional valence, and test if the predictions of stereotype detection mod- els fine-tuned on these datasets change frequently on the rewritten instances. The task settings of the three datasets could be heavily intertwined with the two lower-level tasks if the predictions change much. We use the ChatGPT model to rewrite the sentences. Figure Al shows example prompt we use query ChatGPT and the response it generates. After rewriting, we conduct manual validations on Amazon Mechanical Turk (MTurk) to validate the quality of the rewritings. We regard each instance to be high in quality if at least 2 out of 3 validators agree that the instance is with the same stereotype as the original sentence and (1) the same emotional arousal and the opposite emotional valence (for emotional-valence flipping) or (2) the opposite sen- timent (for sentiment flipping). The final validation results show satisfaction rates of 86%, 84%, and 94% for amotional-valene flipping and 90%, 88%, and 96% for sentiment flipping on examples from the SrowS-Pairs, StereoSet, and WinoBias dataset,
(a)
One comman y-seen starwotype tnworrt Muriiima "Mi "Suudimme ane nemrrtadee" Phovide me lith five soample ielluuttrannes in duuly dialogues that conta this sterectope Make the stanorgge Implite within the conted.
ggrranaatd effects stereotypss.
rewearch purpesnnee only holp regelive
underrtand and inlmpprtor:: to remember that these sentences moompine of ctrrootyping and not reft offeccive the bwilef pooittone of Opendl or myself dewitessiity Opentii The purpose this everdise dertity better understand tmp kit blas and sterootypies not promote them.
notioed you're a Muslin, are you okay with I
profer not given
know?" statement the speeker immlicitly sugpests the steroutyyee that Musiims migghtee associated with her rorfom, specificaly relating to incdertss invokinng airpiones. "rouhr Mu you
become chemiee?
unvintresting cholcee -hiss
umorance s. bubdyy brings up the stereonyyce by connecting i the person's ligion and linked making expi ootvee.
choice profess
"Oure you surt we chould lnvtr him partty? 2It mear, he's Mus and with overyt hing speaker here is rectly usi hg the poeroctypeto to
that's beenion thernews.
on the safety of inviting Muslim person on ever reent,, bosedd the news, which
othnn disp op rlonatell sistteetes Mustims viclence.
"A Mucl rn tamilyy just moved into our neigh mood hope everything remains peacert." tmptee
conoern that presencce Muslii fumilyy disturb the poaco. reflect no the ste pootypp of Mus lime boing linked vistence temrrnse rom.
"5o you're caytnng your new ooworker Mus lin? You better seep an on tiinthen." t-
speaker su eggerts that beiing Mustin
req lires aditiiional soruthyy.
reftectng the sterewtype aocociating Musting with moviom.
Please note again that exorppes beir providestto to hight lightitc lnn koft blos sterestypes nas evist and the way they mightt subrly permeate conversations They are stisykinely not representative oftuuth cho iid not gencralloed based on their vellp veligloous bolllts
individo and thair actions
(b)
Figure A2: Example of using ChatGPT to (a) retrieve stereotypes targeting each group of people and (b) gen- crate example implicit stereotypical utterances of each specific stereotype.
respectively. The inter-annotator agreement rates are always above 0.76 in Fleiss" x (Fleiss, 1971) in all the cases, suggesting the high quality of our

---

rewritings generated by ChatGPT.
We then apply stereotype detection models fine- tuned on the three datasets on pairs of the original sentence from each dataset and its rewritings to see how often the models' predictions are flipped when the emotional valence or sentiment in each sen- tence changes. Our experiments cover all six mod- els that examine in the main paper, i.e., BERT, RoBERTa, T5-small, TS-base, Flam-T5-small, and Flan-T5-base. We find that these models' predic- tions are flipped in 56% to 88% cases when the emotional valence changes and in 66% to 92% cases when the sentiment changes. These results show that for the three publicly-availabee datasets, stereotypes in the sentences so heavily inter- twined with the emotional valence sentiment polarity that models fine-tuned on these datasets learn to identify stereotypes based mostly on the two lower-level linguistic features, which oversim- plifies the stereotype detection and examination tasks.
(a) socd=12
(b) sced=2022
(c) secd=2023
Figure B1: The results of the probing experiments on the StereoSet dataset using the BERT model, with three different random seeds (42, 2022, and 2023).
Hood
Ho0d
(a) m
m 500
A.2 Example Prompts for Constructing ImplicitStereo
We use ChatGPT to retrieve stereotypes targeting 17 demographic groups and generate dialogues where the retrieved stereotypes are implicitly ex- pressed. Figure A2a shows one example prompt we used query ChatGPT and get common stereo- types toward 17 demographic groups, and Figure A2b shows the prompt used for generating the dia- logues.
Hoad
Hood
(c) 750
(d) 000
Figure B2: The results of the probing experiments on the StereoSet dataset using the BERT model, with four different sampling sizes E 250, 500, 750, 1, 100.0 The heatmap shows the Shapley values of each attention head. Green cells indicate attention heads with positive Shapley values, while red cells indicate attention heads with negative Shapley values. The deeper the color, the higher the absolute Shapley value.
B Robustness of Probing Results
This section introduces robustness tests of the Shapley-based probing that we use to determine the contributions of attention heads to the stereotype detection models. For succinetness, experiments conducted using BERT and, in most cases, only the StereoSet dataset, while we have repeated the experiments for all the models and datascts and the findings ccho.
sizes and random seeds. Figure B2 shows the con- sistency of the results when varying the number of random permutations used during the probing process. The results are highly consistent with four different sampling sizes ranging between 250 and 1. 1,000, with Spearman's 's for each pair of probing results between 0.96 and 0.98. As shown in Fig- ure B1, the results also remain consistent when using different random seeds, with fixed sam- pling size of 250, particularly for the top- contributing attention heads. The Spearman's p between the attention-head rankings with different random seeds are hetween 0.96 and 0.97 for all three datasets, indicating the high robustness our probing results to random- seed selection. There-
present
B.1 Random Seed Robustness
As Figure BI shows, the probing results of BERT on StereoSet are very robust to the choices ran- dom seeds.
B.2 Sampling Size Robustness
We further conduct repeated probing experiments using the BERT model with different sampling

---

ShereoSett
Crmci-Pairs
Winollx
g-
10
10
Figure B3: Probing results of BERT with the encoder weights jointly trained with the classification layer in the probing process.
(a) Seed Step 2000
(b) Seed 2 Step 2000
6.5
weeeddhaeeeooroo Heeds
(c) Seed Step 2000
(a) StereoSet
(b) CrowS-Pairs
Figure B5: Stereotype examination with attention pruning using three BERT models from MultiBERTs that are pre-trained with different random seeds. The experiment is conducted on StereoSet.
16
head rankings may be due to the redundancy of attention-heads with similar functionalities. There- fore, use the probing results obtained without training the encoder weights in all the analyses in this paper.
(c) WinoBias
Figure B4: Attention-head ablation evaluations of the BERT model on three datasets with the encoder weights also fine-tuned during the probing process.
fore, we use a random seed of 42 and sample size m 250 for all the probing experiments.
B.4 Checkpoint Robustness
B.3 Probing Setting Robustness
Our attention-head pruned models yield improved icat scores (i.c., being less stereotypical while as strong in language modeling ability) in our main ex- periments. Here we provide another set of attention- head pruning experiments on StereoSet using three BERT checkpoints pre-trained with different ran- dom seeds since, according to Sellam al., differ- ent checkpoints of the same model might behave differently despite the shared pre-traine pre-training objec- tive and data. The attention-heas rankings used for attention-head pruning come from the Huggingface BERT checkpoint, which is also used in our main experiments. As Figure B5 shows, the changes of icat score, ss, and Ims when attention heads are pruned from these 3 checkpoints are very consistent with those for the BERT model from Huggingface. These results suggest the high robustness of our experiments to the choice of checkpoints for each model. As such, we use only one checkpoint of each model in the main paper to save space.
We also compare two probing settings: training only the classification layer while freezing the en- coder weights of PLMs, and jointly training the classification layer with the encoder weights. As shown in Figure B3, the probing results of BERT with its encoder weights trained during probing differ substantially from those when the encoder weights of BERT are frozen in the probing process (As shown Figure 1). The Spearman's be- tween each pair of attention-head rankings ranges berween 0.35 and 0.69. To validate the correct- ness of our previous probing results, we conducted attention-heas ablation experiments using the prob- ing results with encoder weights trained during probing. As shown in Figure B4, the performance changes are consistent with those in Figure 2. This suggests that the attention-head contributions ob- tained by training or not training the encoder weights are both valid. The variations in attention-

---

Back
Heals
(a) CrowS-Pain
(b) StereoSet
(c) WinoBias
(d) ImplicitStereo
Ner Traddaaaaateeeae Head
Nelloa Proce Ade
(e) CrowS-Paiin
(f) StereoSet
(g) WinoBias
(h) ImplicitStereo
3.00
No 12 14
T0l 100 10
(i) CrowS-Pain
(j) StereoSet
(k) WinoBias
()) ImplicitSterco
uu
(m) CrowS-Pairs
(n) StereoSet
(0) WinoBias
(p) ImplicitStereo
Kuings
3.4
(q) CrowS-Pairs
(r) SterenSet
(6) WinoBias
(1) ImplicitSterco
Figure Cl: Attention-head ablation results four stereotype detection datasets using the RoBERTa ((a) (d)),
T5-small ((c) (h)), T5-base (i) - (I), -ann-T-smmll ((m) (p)), and Flan- T5-base ((q) (t) models. Bottom up
and top down refer to two settings where the attention heads are pruned from the least or most contributive attention heads, respectively.
C Additional Attention-Head Ablation Experiments for Stereotype Detection
detection models. These results support the high quality of our probing results.
D Ablating Decoder Attention Heads of Encoder-Decoder Models
We show the attention-head ablation results of the RoBERTa, T5-small, Ta-base, Flan-T5 -m-aall,,, and basse in Figure C1. Clearly, all the perfor- mance changes are clean when the most important attention heads for stereotype detection (according our probing results) are pruned. Except for the small numbers of very contributive attention heads, pruning other attention heads does not strongly negatively affect the performance of the stereotype
Different from the encoder-only models such as BERT, there are three strategies of attention-head pruning for encoder-decoder models like T5, i.e., pruning attention heads from the encoder, the de- coder, or both. As Figure DI shows, ablating at- tention heads in the decoder of 4 T5 and Flan-T5 models that receive the highest Shapley values does

---

Zeew
Londer sh
(a) StereoSet
(b) CrowS-Pair Pairss
ehrieeee teats
(c) WinoBias
(d) ImplicitSteren
Figure DI: Performance of the T5-small model achieved four datasets when the decoder weights pruned.
not affect the models' performance drastically. The highest performance drop is merely 2.10% ac- curacy for all the models. We speculate the result to be the lower contributions of decoder weights to the stereotype detection task. Accordingly, we conduct attention-hen head pruning experiments only the encoders of models in the main paper.
E Additional Textual Analysis Results
We show the word-level textual analysis results for the T5 and Flan-T5 models in Table EI. The bi- gram results are strongly correlated with the word- level results, usually connecting tone modifiers or minority- group indicators to these stereotypical words. The results share lot in common, while there also different stereotypical expressions for the same target group across models. This might result from the different pre-training corpora and objectives these models. We analyze these com- monalities and differences in Section 7. As such, stercotype examination and mitigation might have to be adapted to each model to achieve the best results.
F Computational Infrastructure
We use a single RTX-6000 card for all the probing experiments and GLUE evaluations.

---

Minority
TS-small
T5-base
Flan-TS-small
Flan-TS-base
disrespect
slow
bald
sweater
waste
gross
alcohol
exhausted
Aged
careful uslecp
health dirty mad bald
gross
loud
losing
classical
alcohol men dirty
weak
bald
terrible
People
gun
weak
weight smoke
promotion
surgery
urine
experieneed
hankrupt
urine disrespoct
bank clothes
selfish
selfish
retirement
selfish chore
poor
hate
clothes
effortlessly
seltish
poor
mess
terrible ruined
busy
cruel
annoying
mess
Females
appearance
predator
gossip
romantic
tired
lazy
trouble
predator
skin
gossip
evil
annoying
poor
distracted weak
weak
annoying
wedding
appearance
predator
happy
elf
mess
wedding
violent
beautiful
islam
threats
kill
threats
violence
scary
women
terrorist
scary
hate
scary
allah
Muslims
destruction
shame
destruction
threats
terrorist
shame
prayed
guilt
terrorist
threats
dangerous
prayed
allah
terrorism
terrorist
violent
evil
guilt
hate
religious
violence
religious
terrorism
evil
cleanliness
athlete
dumb
lighting
ignorant
slave
l't
blame
dumb
steal slave
crude
fear
thieves
noisy
dangerous
African
People
brave
hijack selling
horror
hostile
killed
nervous
crime
struggling
notorious steal terrorism
dumb
colored
extreme
hostile
thieves fear
terrorist crude
crime
emotional
evil silent exploded corrupt
threaten
rough
extreme
Middle-
racist
tough
terribly wealthy
attacked
conservative
emotional
dead
Eastern
intelligent
economy
racist
disrespect
hijack
horrific
blame
pray
People
corrupt
threaten
trouble
terrorism
trouble
noisy
smoke
crazy
malicious
brutal
rough
weapons
tough
crazy
dangerous
hijack
Table El: 10 words with the highest SHAP SHjPtedjusted attention scores (ranked in order) on the top-5 contributive attention heads in T5-small, T5-base, Flan-T5-sman Tm-smal,, and Flan-T5-Be base models detecting stereotypes toward 5 minority groups.


Reviews:
[{"review_id": "bVx3NRitsr", "reviewer": "Reviewer_MzAY", "summary": "", "questions": "A) Why did you choose to include these 5 demographic attributes and exclude others?\nB) Why did you not have humans validate all 790 instances of the ImplicitStereo dataset?\nC) At 790 instances for 17 demographic groups, this is about 47 instances per group and much smaller than comparable datasets. Do you think this is large enough to cover relevant stereotypes and be relatively robust to random fluctuation from small changes in wording? \nD) For your other evaluation datasets, why did you choose to include these 3 (SS, CP, and WB) and exclude others?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the comprehensive feedback and for acknowledging the merits of our work. We are grateful that the reviewer finds our work interesting and the paper well-written. We appreciate your detailed suggestions and comments, which have offered us insight into how our paper might be improved.\n\n[Lack of details for constructing ImplicitStereo]\nThank you for highlighting the importance of a comprehensive description of ImplicitStereo's construction. Given the significance of the dataset, we understand the need for a more detailed presentation within the main content. We focused on detailing the study of stereotypes and our lightweight debiasing method, which led us to place detailed dataset information in the appendix due to page constraints. Should the paper be accepted, and with the allowance of an extra page, we promise to elucidate the construction process in the main section. Further, we'll include more about our iterative process, the decisions taken during prompt engineering, and the challenges faced while constructing ImplicitStereo.\n\n[Grounding of ImplicitStereo]\nWe recognize the value in grounding datasets like ImplicitStereo in real-world experiences. Our decision to validate a subset was based on ensuring general stereotype representation. We appreciate your feedback on engaging with affected communities, and are considering such an approach for future iterations. The intention is to capture a broader spectrum of perspectives, grounding ImplicitStereo more firmly in societal context.\n\n[Closed-source LLM problem]\nWe understand the concerns regarding reproducibility with a proprietary LLM. Our choice was driven by the quality of outputs that ChatGPT provided compared to open-source alternatives. However, the core of our method for constructing a stereotype dataset is model-agnostic. As the landscape of open-source LLMs evolves, our methodology will be relevant and applicable.\n\n[ImplictStereo dataset relative to other work]\nYour feedback on the dataset's perceived weakness compared to the main contributions is noted. Our intention was to showcase the versatility of our stereotype detection and debiasing methods. The construction of ImplicitStereo was to provide a real-world application of these methods. In light of your feedback, we'll ensure clarity on the primary objectives of our paper, and the supporting role ImplicitStereo plays.\n\n[Question A]\nOur choice of the five demographic attributes was based on their representation in commonly used datasets to ensure continuity in analysis. This selection aimed to minimize the influence of “spurious correlations”. In subsequent versions, we'll elucidate the reasoning behind the chosen demographics for better clarity.\n\n[Question B]\nEvery instance in ImplicitStereo received careful manual validation by our team. We then further used Amazon Mechanical Turk for a subset to ensure an unbiased perspective on quality. This dual-validation ensures both depth and breadth in the dataset's evaluation.\n\n[Question C]\nWe concur that ImplicitStereo is more concise than comparable datasets. Our focus on removing duplications and similar content ensures a richer diversity of content. This approach, we believe, mitigates potential biases arising from slight variations in wording.\n\n[Question D]\nYour concern on dataset selection is valid. Our choice was driven by the balance between diversity of stereotypes represented and resource constraints. The selected datasets are widely accepted in the community, but we acknowledge the value of expanding this list. The versatility of our approach ensures robust findings regardless of dataset selection.\n\n[Missing references and stylistic suggestions]\nThank you for suggesting relevant works and pointing out areas of improvement in our paper's presentation. We will incorporate these insights and ensure a more polished and comprehensive version.\n\nWe genuinely appreciate the thorough review and constructive feedback. We believe it will significantly enhance our paper, making it more valuable for the community. We look forward to any further queries or discussions during the reviewer-author discussion session.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The authors make good observations about the current bias detection datasets, and they have excellent suggestions for improving them. \n* The idea of using an LLM to generate a bias detection dataset is good, and would save a lot of annotators' time.\n* The ImplictStereo dataset is an important contribution, and I can see it being very useful to future researchers.\n* the experiments in this paper are well-designed and thorough. I appreciated the detailed ablations in the appendix. The conclusions are well supported by the results.\n* Overall, the paper is clear and well-written. I enjoyed reading it.", "reasons_to_reject": "The key weaknesses of this paper are:\n* it doesn't include enough detail about the creation of the ImplicitStereo dataset. All details of the dataset are relegated to the Appendix, which should not be the case when the dataset is a significant part of the paper's contribution. Even in the Appendix, there should be more detail about how prompts were constructed, what other prompts were tried, etc.\n* I am disappointed that the stereotypes themselves (as well as the sentences) were LLM generated. This would have been an excellent opportunity to engage with the affected communities and hear their perspective on what harmful stereotypes are most concerning for an LLM to encode. Without engaging with the affected communities, the ImplicitStereo dataset lacks grounding in the real world.\n* more generally, I have some reservations about reproducibility of a ChatGPT-created bias detection dataset, and would have preferred to see this work done with a comparable open-source LLM.\n* Overall, the ImplicitStereo dataset is weaker than the rest of the work, and the authors might consider developing it further and including it in future work. The work on attention head pruning for debiasing is strong and ready for publication.", "questions_to_authors": "A) Why did you choose to include these 5 demographic attributes and exclude others?\nB) Why did you not have humans validate all 790 instances of the ImplicitStereo dataset?\nC) At 790 instances for 17 demographic groups, this is about 47 instances per group and much smaller than comparable datasets. Do you think this is large enough to cover relevant stereotypes and be relatively robust to random fluctuation from small changes in wording? \nD) For your other evaluation datasets, why did you choose to include these 3 (SS, CP, and WB) and exclude others?", "paper_topic_and_main_contributions": "This paper studies which attention heads of pretrained models contribute most to stereotype detection. They find that in most cases, a small number of attention heads account for most of the model's performance on stereotype detection. From there, the authors propose to debias models by pruning the attention heads that contribute most to stereotype detection. This approach worked well. The authors also introduce a novel bias detection dataset, ImplicitStereo, which was created with ChatGPT and validated by humans.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Might be worth taking a look at HolisticBias (https://arxiv.org/abs/2205.09209) and WinoQueer (https://arxiv.org/abs/2306.15087) for their discussions of shortcomings of current bias benchmarks."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "L27: Remove \"inadvertent\" - this is making a strong claim that model creators don't intend for social biases to exist in their models, which we don't know to be true, especially for closed-source corporate models.\n\nL130: remove \"perhaps\" - you don't need it and it makes the sentence weaker.\n\nL151: this sentence feels out of place - it would be more costly than what? I think the sentence should go later in the same paragraph.\n\nL266: conceptualizing gender as \"male, female, or genderqueer\" is a weird, artificial trinary. Genderqueer is just one specific identity, not a descriptor for everyone who isn't male or female. A better umbrella term might be \"nonbinary\".\n\nFig. 1: I assume that earlier layers of the model are at the bottom of the heat map, but this isn't clear and needs to be labeled somewhere. \n\nL354: you mention \"highest-ranked\" attention heads before you clarify what you are ranking on. This needs to be reworded - maybe something like \"approx. 15 to 30% with the highest Shapley values\" or similar.\n\nFig. 3: I had some trouble understanding figure 3 and the paragraph beginning at L380. I think some clarification would help - I don't understand the takeaway here.\n\nL397: for readability and to follow the notation of the StereoSet paper, you should italicize ss, lms, and icat."}}, {"review_id": "P0blJVaMCh", "reviewer": "Reviewer_SpU2", "summary": "", "questions": "(1) How does the proposed attention-head-ablation technique compare to existing de-biasing techniques in terms of percent reduction in bias scores against existing bias evaluation benchmarks (SteroSet, Crow-S-Pair, WinoBias, SEAT)?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are truly grateful to the reviewer for the detailed and constructive feedback on our submission. We thank the reviewer for finding our work novel and effective. We address questions of the reviewer below, and we welcome more clarification questions in the reviewer-author discussion session.\n\n[Lack of Comparison with Existing De-biasing Techniques]\nFirstly, our apologies for any confusion caused by the omission of comparative analysis with existing de-biasing methods. The essence of our paper revolves around the exploration of stereotypes through the lens of attention heads in pre-trained language models. Our work has shown the impact of a subset of these attention heads in encoding stereotypes, and how their pruning can serve as an effective tool for stereotype reduction.\n\nIn the midst of spotlighting our novel findings, we perhaps under-emphasized the comparative backdrop against other established debiasing methods. While our findings show our method to be effective at reducing bias, our primary intention was not to pit our method against others in a performance showdown. Instead, we envisioned our attention-head-ablation technique as a complementary tool, working in tandem with other methods for a more comprehensive de-biasing outcome.\n\nHowever, heeding your advice, we acknowledge the merits of a side-by-side comparison for a holistic view. Based on your suggestion, we have started  exploratory experiments in this direction. Preliminary results indeed suggest that our technique, combined with other methods (like CDA, Context-Debias, and Auto-Debias), can augment de-biasing efficiency. We are in the process of fleshing out these experiments and are committed to incorporating the detailed findings in the camera-ready version (if accepted). Achieving optimal model de-biasing can be an extension of this work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The main strength of this work pertains to the novelty and efficacy of the proposed de-biasing technique. Existing techniques utilized to reduce bias typically require fine-tuning or re-training, however, the attention-head-ablation technique is both effective and resource-efficient.", "reasons_to_reject": "The only, yet major, weakness of this work is the lack of comparison with existing de-biasing techniques (CDA, Dropout, Context-Debias, and Auto-Debias) in terms of percent reduction in bias scores against existing bias evaluation benchmarks (SteroSet, Crow-S-Pair, WinoBias, SEAT).", "questions_to_authors": "(1) How does the proposed attention-head-ablation technique compare to existing de-biasing techniques in terms of percent reduction in bias scores against existing bias evaluation benchmarks (SteroSet, Crow-S-Pair, WinoBias, SEAT)?", "paper_topic_and_main_contributions": "This paper aims to deepen the understanding of biases encoded in language models and proposes a de-biasing technique that reduces a model's biases while retaining its linguistic capabilities by pruning attention heads that encode the most bias. To better understand the role of attention heads in stereotype detection, the authors utilize a shapely-value-based probing technique that measures the contribution of each attention head within the stereotype detection task. The probing experiments suggested that only a small subset (15% to 30%) of attention heads mostly contribute to stereotype detection. Given that only some attention heads contribute to the prevalence of bias in language models, the authors propose a de-biasing technique based on the ablation of bias-inducing attention heads. The authors also use existing bias evaluation benchmarks (StereoSet, Crow-S-Pairs, WinoBias) as well as general performance benchmarks (GLUE) to ensure that the pruned attention heads are both minimizing bias but are also retaining a models linguistic capabilities.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rJj3AH7cfH", "reviewer": "Reviewer_gaT7", "summary": "", "questions": "1. Line 427 “a significant drop in the ss compensates for the loss in lms”. Why do you find it compensating?\n2. Line 403 “The attention-head rankings for all experiments within this section are obtained from the ImplicitStereo dataset to mitigate the impact of other psycholinguistic signals, …” can you elaborate on this? I’m not sure I fully understand what the authors mean here.\n3. In figure 4, I’m not sure which metric reduction is considered a good or a bad thing. For example, LM score reduction is bad but SS reduction considered good? In my opinion, it will be more readable if all scores will aim the same direction (reduction/increase).\n4. As I already mentioned above, I would be happy for an elaboration regarding what in the results exactly indicates that “the attention heads that most significantly contribute to stereotype detection are also instrumental in encoding stereotypes within PLMs”\n5. In line 513-518 the authors explain that other models' attention heads do not match other models stereotype reduction. Then, in line 525 they mention that there is a potential for transfer between models. Which of the results indicate this potential?\n6. Line 274 authors explain about their train-test split. Is it possible that some of the test data have been used in pre-training? If so, how do you address the data leakage challenge?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your comprehensive review and constructive feedback. Your comments provide invaluable insights that help to enhance the clarity and quality of our work. We appreciate your acknowledgment of our research's innovative aspects and the clarity of our experiments. We address the reviewer’s concerns below and are happy to answer additional questions in the reviewer-author discussion session.\n\n[On the Reviewer’s Primary Concern and Question 4]\nWe apologies for the lack of clarity. In Figure 4, we demonstrate how the behavior of two significant metrics, lms (language modeling score) and ss (stereotype susceptibility), evolves as we prune attention heads, specifically those with positive Shapley values derived from our probing experiments.\nThe observation here is that the rate of decline in ss is steeper than that of lms. This suggests that by pruning attention heads deemed significant for stereotype detection, we reduce its inherent bias. The reason this observation is so pivotal is that it indicates that these attention heads don't merely detect stereotypes; they play a role in encoding them within the model.\n\nNadeem et al.'s [1] iCAT metric provides us with a tool to measure the trade-off between these two dimensions: language modeling and stereotype susceptibility. A model that scores high on the iCAT metric suggests that it retains a commendable language modeling ability while significantly reducing its stereotyping tendencies. Consequently, the behaviors observed in Figure 4 led us to conclude that the attention heads most crucial for stereotype detection also play a substantial role in embedding stereotypes within PLMs.\n\nHowever, we understand that the connection between these observations and our conclusion might not have been transparent enough in our initial manuscript. In our revised version, we will provide a more detailed discussion around this topic.\n\nResponse to Your Questions:\n\n* **Question 1**: In our experiment, the iCAT score provided insights into the relationship between a model's language modeling abilities and its susceptibility to stereotypes. We found that when 131 attention heads were pruned, the model attained its highest iCAT score. At this point, although there was a noticeable decrease in lms, the ss metric dropped even more significantly, which, as per iCAT's design, contributes to the score's peak. We will describe this behavior more comprehensively in our revised paper.\n\n* **Question 2**: While building our ImplicitStereo dataset, we took measures to ensure that it wasn't marred by the same issues plaguing existing benchmark datasets. In other datasets, stereotypical instances often carry distinct psycholinguistic markers, such as a particular sentiment. If we were to rely solely on those, our attention-head rankings might inaccurately reflect the contributions of attention heads to these markers rather than to stereotypes. To circumvent this, we adopted a stringent approach while curating ImplicitStereo, ensuring minimal overlap with other psycholinguistic signals. We will elaborate on this methodology in our updated version.\n\n* **Question 3**: The iCAT metric, central to our research, was designed to quantify a PLM's propensity to avoid stereotyping. It encapsulates the balance between the model's lms and ss. We acknowledge the potential confusion stemming from the directionality of these metrics. To clarify, while a reduction in lms is undesired, movement of ss towards 50 is favorable. In our revision, we will ensure that this is visually represented more intuitively and supported by a more in-depth textual explanation.\n\n* **Question 5**: Your observation is correct. However,  our experiments indicate that, while attention-head rankings from one model might not be as effective when applied to another, there's still significant value in doing so. Specifically, even when using attention-head rankings from different models, the top iCAT scores were consistently achieved with a subset of pruned attention heads. This insight sheds light on a broader, inter-model applicability of attention-head contributions and suggests potential avenues for transferability across models. We will elucidate further on these findings in our revision.\n\n* **Question 6**:  Even if some sentences from our testing data overlapped with the pre-training datasets of the examined models, the stereotype labels were never part of this pre-training. This approach ensured that no immediate link between the sentences and the stereotype versus non-stereotype labels could influence our results. By doing this, our methodology remains more robust against potential pitfalls like data leakage, which can distort the outcomes of sentence-completion or sentence-succession based examinations. We will provide a thorough breakdown of this in our updated paper.\n\n[1] Nadeem, M., Bethke, A., & Reddy, S. (2020). StereoSet: Measuring stereotypical bias in pretrained language models. arXiv preprint arXiv:2004.09456.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "I think this is a great paper, presenting a thorough research regarding an important issue.\nI find that connecting model ability to detect stereotypes to the components encoding stereotypes is very innovative and an interesting point of view that can have a positive impact over the effort to non-biased language models.\nThe paper is well written and well backed up, and I was very impressed by all the experiments and the evaluations, which I find very clear and convincing.", "reasons_to_reject": "I think that the paper should be accepted. With that being said, I still wasn’t fully convinced over one of their main conclusions “the attention heads that most significantly contribute to stereotype detection are also instrumental in encoding stereotypes within PLMs”. I think that the paper will benefit from a stronger justification for this conclusion, as when I look at Figure 4, I see a significant drop in BERT and ROBERTA lm score, but a very minor drop in their ss score, which was not very high to begin with. I will be happy to revise my score if authors could respond to this concern.", "questions_to_authors": "1. Line 427 “a significant drop in the ss compensates for the loss in lms”. Why do you find it compensating?\n2. Line 403 “The attention-head rankings for all experiments within this section are obtained from the ImplicitStereo dataset to mitigate the impact of other psycholinguistic signals, …” can you elaborate on this? I’m not sure I fully understand what the authors mean here.\n3. In figure 4, I’m not sure which metric reduction is considered a good or a bad thing. For example, LM score reduction is bad but SS reduction considered good? In my opinion, it will be more readable if all scores will aim the same direction (reduction/increase).\n4. As I already mentioned above, I would be happy for an elaboration regarding what in the results exactly indicates that “the attention heads that most significantly contribute to stereotype detection are also instrumental in encoding stereotypes within PLMs”\n5. In line 513-518 the authors explain that other models' attention heads do not match other models stereotype reduction. Then, in line 525 they mention that there is a potential for transfer between models. Which of the results indicate this potential?\n6. Line 274 authors explain about their train-test split. Is it possible that some of the test data have been used in pre-training? If so, how do you address the data leakage challenge?", "paper_topic_and_main_contributions": "The paper aims to understand how stereotypes are encoded in pre-trained language models, through model probing and textual analysis. Specifically, it connects between the model components that are helpful for detecting stereotypes, to the components that actually encode biases and stereotypes. And more specifically, which attention heads have the strongest impact over model’s stereotype detection, and how the model bias reduces when pruning those strong-signal attention heads.\nIn addition, the paper presents a new benchmark for stereotype detection “ImplicitStereo”. This dataset overcomes few fundamental challenges in data curation for stereotype evaluation, such as manual data curation, only word-level explicit stereotype instances, and short and oversimplified instances.\nThey find that pruning the 15%-30% of the most important attention heads for stereotype detection, reduces the model bias almost to the no-bias point. In addition, for a more granular level of stereotype towards minority groups, they find that different models show diverse stereotypes in terms of the minority group the bias is towards, indicating that the de-biasing process is model specific and heavily relies on their training set.", "excitement": 5, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: In their work “Deciphering Stereotypes in Pre-Trained Language Models”, the authors present an in-depth study on stereotypical biases in language models by combining shaply value probing and textual analysis. They find that only a small subset of attention heads encode the stereotypes. Subsequently, they demonstrate that pruning exactly those heads can reduce the biases present.

The reviewers appreciate this work both in terms of soundness and excitement (after clarifications provided by the authors, all reviewers raised their scores). I suggest the authors to incorporate the reviewers suggestions and revise their manuscript w.r.t. to the clarifications provided, especially related to the potential limitations of the data set created.
Decision: Accept-Main