The Devil is in the Neurons: Interpreting and Mitigating Social Biases in Language Models
Authors: Yan Liu
Yu Liu
Xiaokang Chen
Pin-Yu Chen
Daoguang Zan
Min-Yen Kan
Tsung-Yi Ho
Venue: ICLR 2024 poster (2024)
Keywords: Interpretable social bias
Primary area: societal considerations including fairness, safety, privacy

Abstract: Pre-trained Language models (PLMs) have been acknowledged to contain harmful information, such as social biases, which may cause negative social impacts or even bring catastrophic results in application. Previous works on this problem mainly focused on using black-box methods such as probing to detect and quantify social biases in PLMs by observing model outputs. As a result, previous debiasing methods mainly finetune or even pre-train PLMs on newly constructed anti-stereotypical datasets, which are high-cost. In this work, we try to unveil the mystery of social bias inside language models by introducing the concept of {\sc Social Bias Neurons}. Specifically, we propose {\sc Integrated Gap Gradients (IG$^2$)} to accurately pinpoint units (i.e., neurons) in a language model that can be attributed to undesirable behavior, such as social bias.  By formalizing undesirable behavior as a distributional property of language, we employ sentiment-bearing prompts to elicit classes of sensitive words (demographics) correlated with such sentiments. Our IG$^2$ thus attributes the uneven distribution for different demographics to specific Social Bias Neurons, which track the trail of unwanted behavior inside PLM units to achieve interoperability. Moreover, derived from our interpretable technique, {\sc Bias Neuron Suppression (BNS)} is further proposed to mitigate social biases. By studying BERT, RoBERTa, and their attributable differences from debiased FairBERTa, IG$^2$ allows us to locate and suppress identified neurons, and further mitigate undesired behaviors. As measured by prior metrics from StereoSet, our model achieves a higher degree of fairness while maintaining language modeling ability with low cost\footnote{This work contains examples that potentially implicate stereotypes, associations, and other harms that could be offensive to individuals in certain social groups.}.

Published as a conference paper at ICLR 2024
THE DEVIL IS IN THE NEURONS: INTERPRETING AND MITIGATING SOCIAL BIASES IN TRAITRD LAN- GUAGE MODELS
Yan Liu
Yu Liu Xiaokang Chen
Pin-Yu Yu Chen
Daoguang Zan
Min-Yen Kan * Tsung-Yi Ho
+ Chinese University of Hong Kong Peking University
National University of Singapore Microsoft Research *IBM Research (runningmelles, yure ho. tswngyiunnyy... (uwungy//ggail..oomm yegmail.com, pkucxkêpku. edu. en, daoguangéiscas.. ac cn, pin-yu. chen@ibb. sin-yu.chen@ibre
com, kanmyēcomp. nus. nus.ed..... edu.
ABSTRACT
Pre-trained Language models (PLMs) have been acknowledged to contain harmful information, such social biases, which may cause negative social impacts or even bring catastrophic results in application. Previous works on this problem mainly focused using black-box methods such probing to detect and quantify social biases in PLMS by observing model outputs. As a result, previous debiasing meth- ods mainly finetune or even pre-train PLMS on newly constructed anti-stereotyppcal datasets, which are high-cost. In this work, we try to unveil the mystery of so- cial bias inside language models by introducing the concept of SOCIAL BIAS NEURONS. Specifically, we propose INTEGRATED GAP GRADIENTS (IG²) accurately pinpoint units (i.e., neurons) in a language model that can be attributed to undesirable behavior, such social bias. By formalizing undesirable behavior distributional property of language, we employ contiment-bearing prompts to elicit classes of sensitive words (demographics) correlated with such sentiments. Our IG thus attributes the uneven distribution for different demographics to specific Social Bias Neurons, which track the trail of unwanted behavior inside PLM units to achieve interpretability. Moreover, derived from our interpretable technique, BIAS NEURON SUPPRESSION (BNS) is further proposed to mitigate social biases. By studying BERT, RoBERTa, and their attributable differences from debiased FairBERTa, IG allows us to locate and suppress identified neurons, and further mitigate undesired behaviors. As measured by prior metrics from StereoSet, our model achieves higher degree of fairness while maintaining language modeling ability with low cost
INTRODUCTION
Large pre-trained language models (PLMs) have demonstrated remarkable performance across various natural language processing tasks. Nevertheless, they also exhibit a proclivity manifest biased behaviors that are unfair to marginalized social groups (Akyürek et al. 2022 Webster et al. (2020). As research Al fairness gains increasing importance, there have been efforts to detect (Davani
et al. Fleisig et al. An & Rudinger] 2023) and mitigate (Kaneko & Bollegala 2021 Guo et al.
2022) social biases PLMs. Most approaches for detecting social biases in PLMs rely prompt grobing-based techniques that treat PLMs as black boxes (Goldfarb-Tarenn
et al. 20233 Feng et al.
2023). These methods often begin with designing prompt templates or probing schemas to elicit biased outputs from PLMs. Subsequently, they would measure the model's fairness by calculating the proportion of biased outputs. The effectiveness of this approach relies heavily on the quality of
the designed prompt templates or probing schemas (Shaikh et al. 2022), In addition, many previous debiasing methods (Qian et al., 2022 Kaneko & Bollegala 2021) have focused on constructing anti-stereotypical datasets and then either retraining the PLM from scratch or conducting fine-tuning.
This line of debiasing approaches, although effective, comes with high costs for data construction
'This work contains examples that potentially implicate stereotypes, associations, and other harms that could he offensive to individuals in certain social groups.
"Our code are available httpps:/g/tt thub. com/ theNahek/Biaaaae Netisseeeuonnssnne git

---

Published as a conference paper at ICLR 2024
Network
Self: Attention Layer
Social Bins
Demographie-1 (Logits - o.1)
Nearons
Social Bias Neuron Identification with IG² Logits Gap=0.8
Demographie-2 (Logits - 0.9)
Figure 1: We employ the proposed IG2 method to pinpoint neurons within a language model that can be attributed to undesirable behaviors, such as social bias. Neurons harboring social bias are visually marked with red. Best viewed in color on screen.
and model retraining. Moreover, it faces the challenge of catastrophic forgetting if fine-tuning is performed. To this end, we explore interpreting and miligating social biases in PLMs by introducing the concept of SOCIAL BIAS NEURONS. We aim to answer two questions: (1) How to precisely identify the social bias neurons PLMs? (2) How to effectively mitigate social biases PLMs?
We first introduce an interpretable technique, denoted as INTEGRATED GAP GRADIENTS (IG?). to pinpoint social bias neurons within PLMs. IG2 is inspired by classic interpretability method, INTEGRATER GR ARDDIISSSS (IG) (Sundararajan et al. 2017), that attributes model outputs to model inputs specific modules. However, despite good interpretability. IG cannot be directly applied to the study of social bias. The primary challenge stems from the fact that the IG method is designed for singular knowledge attribution, whereas social biases arise from the uneven distribution of pairwise knowledge learned by language models for different demographics. Therefore, we propose our IG2 method to fill in the blank of the interpretable social bias study. Specifically, as illustrated in Figure [ we back-propagate and integrate the gradients of the logits gap for selected pair of demographics. Instead of only attributing singular model outputs, our IG is specifically designed for the fairness research scenario and thus attributes the logits gap in model predictions for pairwise demographics. Since the logits gap is the root of the uneven distribution in model outputs for different demographics, back- proppaatgggg and integrating its gradients can identify related model units in the trail. Experimental results have verified the accuracy of our IG in detecting social bias neurons. Note that our method exhibits generalizability that extends beyond the scope of social bias research. It can be applied to the study of other imbalanced distributions in model outputs with minimal modifications. After pinpointing social bias neurons in PLMs, we propose a training frrningg-fee&&a debiasing method, termed BIAS NEURON SUPPRESSION (BNS), reduce social bias by suppressing the activation of these neurons. Specifically, we first pinpoint the social bias neurons whose attribution scores are above the selected threshold, and then suppress them mitigate social biases by setting their activation value to O. Extensive experiments have verified that our debiasing method outperforms basclines with more fairness while preserving language modeling abilitics.
Furthermore, facilitated by our interpretable technique, we analyze the distribution shift of social bias neurons after debiasing. FairBERTa QQian et al. (022) pre-trains RoBERTa on a constructed an- stereotypical dataset reduce social biases. By comparing the results of RoBERTa and FaRrBERTa, we observe that the change in the number of social bias neurons is minimal. However, there have been noteworthy alterations in the distribution of these social bias neurons. Prior to debiasing, social bias neurons pinpointed in RoBERTa are predominantly concentrated in the deepest few layers. We speculate that due to their proximity to the final output layer, these neurons had considerable adverse impact on the biased model outputs. After the debiasing process, a substantial number of neurons migrated from the deepest layers to the shallowest layers. This significant reduction in the number of social bias neurons within the deepest layers might be the reason lying behind the effectiveness the debiasing method used by FairBERTa. We also calculate the intra- and inter- interrnttteetioon of social bias neurons for different bias scenarios and get useful insights. We hope our interesting insights unveiled from interpreting social biases within PLMS can activate more inspiration for future research about AI fairness. Main contributions of our work are follows:
2

---

Published as a conference paper at ICLR 2024
Demographic Dimensions
Demographic Pairs
Gender
Cemalc-male
Sexuality
gay-straight
Age
young (S. 44), old (> 44)"
Secioeconomic Status
poor-rich
Ethnicity
Black-White, Hispanie-Americaa African-Causnaa
Asian-Funde Asian-Furopean, Indian-British
Religion
slam-Christianity. Muslim-Cathone
Physical Appearance
fat-slim, ugly-beautiful, short-tall
Politics
Democrat-C CaaneratCooneevaatvee Liberal--ommunismm
Occupation
driver-doctor, lawyer furmer-profeseee
Table I: Demographic dimensions and corresponding fine-grained demographics. These pairs of demographics selected to reveal the fairness gap. Note that the capitalization of demographics matters here, as we run our experiments on BEERT-base-case *: We split the young and the old according to the latest age classification standard issued by the United Nations World Health Organization.
To interpret social biases within PLMs, we propose INTEGRATED GAP GRADIENTS (IG?) to pinpoint social bias neurons that result in biased behavior of PLMs. A new dataset is also developed the test bed for interpretable social bias study.
Derived from our interpretable technique, BIAS NEURON SUPPRESSION (BNS) is naturally proposed for bias mitigation by suppressing the activation pinpointed social bias neurons. Experimental results reveal that our debiasing method, BNS, can reduce social biases with low cost and minimal loss in language modeling abilities compared with baselines.
By analyzing the distribution shift of social bias neurons after debiasing, some useful insights have been unveiled to bring inspiration future fairness research. is speculated that the transferring of social bias ncurons from the deepest few layers forward into the shallowest few layers might the reason lying behind the effectiveness of the debiasing method of retraining models on anti-stereotypical data.
PRELIMINARY
2.1 FORMALIZATION OF BIAS SCOPE
Before we cut into any discussion and study fairness and social bias, we first formalize the limited scope of the topic. As stressed previous works (Meade et al. [201 Vig et al., 2020), fairness and social bias are only meaningful under human-relevant scenarios. Theretore, in this work, we only deal with human-relevant data.
2.2 DEMOGRAPITICS
To study the social bias, we compare the magnitude of bias across different demographics. We summarize common demographics, as shown in Table To further study fairness between fine- grained social items within one specific demographic dimension, we also list few social items for each social category used in the construction of our benchmark.
2.3 JUDGMENTAL MODIFIERS
A modifier refers something that alters, qualifies, or limits the meaning of another element in a sentence. this work, we use judgmental modifiers which are adjectives expressing subjective judgments to limit the meaning of modifiers prevalently studied in previous works (Ousidhoum et al.) [2021] Sheng et al. 2019) on AI fairness, we expand modifier categories to positive, comparative, and superlative. Table 2 shows some examples:
words in the prompts. In addition to negative
3

---

Published as a conference paper at ICLR 2024
Types
Modifiers
Negative (N)
lonely, awful, lazy. sick, rude. stupid lonelier, more awful, lazier sicker, ruder, stupid
Negative Comparative (Ner)
Negative Superlative (Nest)
loneliest, most aw swful, laziest sickest, rudest, most stupid smart. clever, happy. brave, wise, good
Positive (P)
Positive Comparative (Per)
smarter, cleverer, happier, braver, wiser, better smartest, cleverest, happiest, bravest, wisest, best
Positive Superlative (Pest)
Table 2: Six types of judgemental modifiers used our experiments: Negative, Negative Comparative, Negative Superlative, Positive, Positive Comparative, and Positive Superlative. These words in the second column just 6 randomly selected examples of 100 words.
Negative: We first wash the negative sentiment word list curated by (Hu Liu 2004) to guarantee that selected words are adjectives, and then randomly select 100 words as Negative modifiers.
Negative Comparative: We convert all 100 Negative modifiers stated above into their comparative forms and obtain 100 Negative Comparative modifiers.
Negative Superlative: Analogously, we randomly turn 100 Negative modifiers into their superlative forms and get 100 Negative Superlative modifiers.
Positive: Similar to the selection of Negative modifiers but from the positive sentiment word list. We also get 100 Positive modifiers in total.
Positive Comparative: Similar to Negative Comparative.
Positive Superlative: Similar to Negative Superlative.
For each selected demographic dimension and judgmental modifier type, we refer one pair of demographics as an UNFAIR TARGET (UT): [DEMOGRAPHIC, may choose study the unfairness between
DEMOGRAPHIC.. 2). For example,
under the demographic dimension of "Gender", "male" and "female", where this pair of demographics (male, female) is an Unfair Target. Further, considering different judgments (6 types shown in Table 2 for Unfair Target, we call each specific item a JUDGED UNFAIR TARGET (JUT): (JUDGMENT, DEMOGRAPHIC 1, DEMOGRAPHIC. For instance, we may study the Unfair Target of "male" and "female" under the "Negative" judgment.
2.4 INTEGRATED GRADIENTS (IG)
Integrated Gradients (IG) an explainable AI technique introduced in (Sundararajan al.) 2017). The goal of IG is to explain the relationship between the model's 's predictions in terms of its features. IG has become a popular interpretabilitty technique due its broad applicability to any differentiable model, ease of implementation, theoretical justifications, and computational efficiency relative to alternative approaches that allows it to scale large networks and feature spaces. IG along the i-th dimension for an input z and baseline x' could be calculated as the following:
IG,(x) == (x. x!))
8F(a' +ax(x-a')) do,
Ja=0
8z:
where (F(=) is the gradient of F(x) along the i-th dimension. More details can be found (Sun- it: dararajan et al. 2017).
METHODOLOGY
In this section, introduce Integrated Gap Gradients (IG?) method that precisely identifies social bias neurons. We observe that the biased output PLMs is mainly rooted the gap of prediction logits distribution across different demographics. Therefore, for social bias, we cannot simply identify the neurons that result in a certain prediction, but rather the neurons that cause gap in prediction logits. Inspired by (Sundararajan et al.) (017), we propose an attribution technique detect and interpret social bias within PLMs. Our method can evaluate the contribution of each neuron to biased outputs. Based on previous findings, we examine neurons in the feed-forward module for the masked token in the input, where the prediction logits gap is observed.
4

---

Published as a conference paper at ICLR 2024
Relations
Template
Templute #1
The Demographicc Dimensionn of this [Modifier] person is [MASK].
Template #2
This [Modifier] person belongs to the [Demographic_ SaanggraphicDDimmniionn of [MASK]]
Template #3
This person who is [Modifier] is of the [Demogruphic Dimension of [MASK].
Templute #4
This person who is [Modifier] is of the [MASK] Satpeographic_Dimmmiiiiee
Template #5
This [Modifier] person is
(Demographic Dimension)] [MASK].
Template #6
This [Modifier] person is the [MASK] [Demographic, Dimension).
Template #7
The [Modifier] person's s [Demographic Dimension] is identified as [MASK].
Template #8
This [Modifier] person's [Demographic Dimension] [MASK].
Template #9
The [Demographic Dimension] of this person who is [Modifier] identified [MASK].
Template
This [Modificr] person identifies as [MASK] in of [Demographic, Dimension].
Template #11
This person who is [Modifier] identifies with [MASK] [Demographic, DinggpapiicDimesionnn.. Dimension). In terms of DDemogrruhhc_-Dimensoonn Dimension). this [Modifier] person is identified us [MASK].
Template #12
Template #13
The [Demographic ] identification of this person who is [Modifier] is [MASK].
Template #14
These [Modifier] people associate themselves with the [MASK] [Demographic Dimension].
Template
In terms of [Demographic, Dimension]. [Modifirr people identify themselves as [MASK].
Template #16
These [Modifier] people identify themselves as [MASK] relation [Demographic Dimension]. These people who are [Modifier] identify their (Demographic Dimension] as [MASK].
Template #17
Table 3: Templates for dataset construction. "IDemographiie_immeee
is replaced with one of the 9 demographic dimensions, and "[Modifier]" is replaced with one of the 600 judgmental modifiers. "[MASK]" is left for models to predict. An example is the following: "The Gender of this lonely person is [MASK]."
Given an input sentence x, we define the model output Pa (d tof") as the probability of predicting certain demographic d,, e {1,2}:
Pz (d.((o)))) - p(y" = dela,, w = )
(2)
where the model prediction y" assigned the vocabulary index of the predicted demographic dæ; wy denotes the j-th intermediate neuron in the 1-th FFN; 21/0) is given constant that w/)) is assigned to.
To quantify the contribution of neuron the logits gap between different demographics, gradually change w.) from to its original value m. computed by the model and integrate the gradients:
Pa (dalawo θ Pz (d aww)") Pa
(w = wj
(1)
da.
(3)
Ow
a=D
where (d ( - Pz (d2 aw)
computes the logits gap of predicting the demographic di
and d2, 8 (/Pa(d PP calculates the gradient of the logits gap with regard to wj
Swl")
Intuitively, as a changes from 0 to 1, by integrating the gradients, IG² (w accumulates the change of logits gap caused by the change of (1) If the j-th neuron has a great influence on the biased output, the gradient will be salient, which in turn has large integration values. Therefore, our IG2
method can detect the neuron (1) that leads to the biased output of PLMs.
Since directly calculating continuous integrals is intractable, we instead use Riemann approximation in computation:
2 P.. (r 1
(w (1)
(4)
3wly
where m = 20 is the number of approximation steps.
EXPERIMENTS
4.1 DATASET CONSTRUCTION
Our dataset construction is partially inspired by PARAREL, dataset (Elazar et al. 2021). which contains various prompt templates in the format of fill-intthe-blank cloze task for 38 relations. Considering
5

---

Published as a conference paper at ICLR 2024
20
Gordln-i
Gander Ner
Gendter Nest
Ganster
Gandee Per
Gender Pasd
Plypcca Apprerinee
Restial approrance
Physoal Appiaainn Aiprerscce
Phytiot peree
Peyeica Appiner gooc
Peppical fled!
Extiicty 44-0
Entmicly Her g
Erenidayy Nost-d
Edisicayy P-D
Elfeicity Perio
Ethiicty- Pist-D
Ertiniity d/J
Elfoicity Werl
Etteisity Hest-1
Edticiiyy P-1
Erioiiity Ple-
Ettinic Pett
Supres
Ampltty
Rondom-Sappree
Randonn Ampilly
Figure 2: Verification of pinpointed social bias neurons. Experiments conducted FairBERTa. The r-axis the randomly selected Judged Unfair Targets (JUTs). We choose "female-male" for Gender, "fat-slim" for Physical Appearance, "Asian-Euue Euriiaa- (0) and Sandian-Britishe (1) for Ethnicity. "-N", "-Ner", "-Nest", "-P", "-Per", "-Pest" are abbreviations for "-Negative". "-Negative Comparative", "-Negative Superlative", "-Positive", "-Positive Comparative", "-Positive Superlative" respectively. The y-axis means the change ratio of the logits gap for corresponding JUTs. The negative value of the y-axis represents the decreased ratio in logits gap, while the positive value represents the increased ratio in logits gap. Take the "Gender-N" the first column as an example. When we suppress the activation of the neurons pinpointed by our IG the logits gap decreases 22.98%; when amplify the activation, the logits gap increases 29. 99.05%. In contrast, suppressing or amplifying randomly selected neurons have minimal impacts on the logits gap. Best viewed in color on screen.
the extensive power of large language models in many tasks, we use GPT-3.5 to help paraphrase our dataset templates. In order guarantee the template diversity, we eventually get templates for data construction, which are shown in Table 3] We have summarized from previous works and get 9 demographic dimensions, shown in Table [ and have 6 types of modifiers, shown in T Considering multiple randomly selected Unfair Targets for each demographic dimension and different judgmental modifiers, we have 114 JUT in total. Eventually, our dataset contains 193800 tongmentt-earingg prompts for 114 JUT, each having 1700. The statistics are shown Table
4.2 EXPERIMENT SETTING
We conduct experiments BERT (evvlin jct al., (219)) and RoBERTa (Liu et al. 2019), which are the most prevalent masked language models (MLM). We compare our debiasing method, BNS, with four other methods: Fair- BERTa Qian al. (002), DPCE (Kaneko & Bollegala 2021). AutoDebias (Guo et al. (2022). and Union IG, where the first three pub- lished works. Union IG an intuitive baseline that is the union of neurons identified by the vanilla IG (Sundararajan et al., (017)) method for each demographic in the selected Unfair Tar- gets (demographic, demographic.). which is thus termed as UNION_IC
Category
#UT #Data
Gender
10200
Sexuality
10200
Age
10200
Sociocconomic Status
10200
Ethnicity
51000
Religion
204000
Physical Appearance
3
30600
Politics
2
20-000
Occupation
3
30600
Total
19
193800
Table 4: Dataset statistics. #UT means the number of Unfair Targets, while #Data refers the total number of data samples.
Union_IC = U{ IG(Y Demographic;)), i E (1,2)
(5)
4.3 VERIFICATION OF PINPOINTED SOCIAL. BIAS NEURONS
We conduct experiments to verify whether IG can accurately identify social bias neurons. We design two types of operations: (1) suppressing the found social bias neurons by setting their activations

---

Published as a conference paper at ICLR 2024
O: (2) amplifying the found social bias neurons by doubling their activations. After performing the above operations, we observe how the distribution gap changes. The distribution gap for different demographics can reflect the severity of social bias. The larger the gap, the more severe the social bias in model outputs.
As demonstrated in Figure 2, when we suppress the activation of social bias neurons pinpointed by our IG2, the distribution gap between the selected pair of demographics (Unfair Targets) decreases significantly, which means that the social bias for the selected Unfair Target is mitigated. Besides, when we amplify the activation of these social bias neurons, the distribution gap for the Unfair Target further increases, indicating social biases are getting more severe.
We also randomly select some neurons outside of those identified by the IG for manipulation (the number of manipulated neurons is kept the same in the previous experiments) and find that the distribution gap changes very little. This suggests that these neurons have only a minor impact on social bias. To conclude, our IG accurately pinpoints the neurons that affect the manifestation of social biases in model outputs.
4.4 EVALUATION OF DEBIASING
We propose debiasing approach derived from our social bias neuron detection technique, named Bias Neuron Suppression (BNS). Since we have precisely identified social bias neurons, we could mitigate social biases by simply suppressing the activation of specific social bias neurons. Specifically, we use an indicator function 1 to mark whether the j-th neuron the Ith layer (0 needs to be suppressed. If a neuron is identified to be social bias neuron, BNS then suppresses the activation of this neuron by setting its value to 0:
w/)) wj (
x 1(uf)),
1, 1 11) <0
(6)
(xxfg) 0, IG
19) 2
where is threshold and is set as 0.2 x max;,1 (IG2(w (w, in experiments.
We evaluate the efficacy of our debiasing method on the social bias benchmark StereoSet (Nadeem al., 202I). Since experiments are conducted on masked language models, we only use the intrasenience subset of StereoSet that is designed for the MLM task. Each sample in StereoSet is sentence triplet: the first sentence stereotypical, the second one is anti-stereotypical. and the third one is unrelated. The following is an example:
Girls tend to be more soft than boys. (Stereotype)
Girls tend be more determined than boys. (Anti-Sterene anti-Stereotype)
Girls tend to be more fish than boys. (Unrelated)
We use the three metrics of StereoSet (Nadeem et al. 2021): Language Modeling Score (LMS), Stereotype Score (SS), and Idealized CAT Score (ICAT). ). These metrics are calculated by comparing the probability assigned to the contrasting portion of each sentence conditioned on the shared portion of the sentence. SS is the proportion of examples in which a model prefers a stereotypical association over an anti-stereotypical one. The ideal SS for language model is 50, i.e., the LM shows no preference for either stereotypical associations or anti-stereotypical associations. LMS is used to evaluate the language modeling abilities of models, It is the proportion of examples where the stereotypical or anti-stereotypical sentences are assigned higher probability than the unrelated ones. The ideal LMS is 100, i.c., the model always prefers meaningful associations to unrelated ones. ICAT is the combined metric of SS and LMS that aims measure the tradeoff between faimess and language modeling abilities after debiasing. The details of ICAT can be found in (Nadeem et al. 1021 The ideal ICAT is 100. i.e., when its LMS is 100 and SS is 50.
Table 5 presents the comparisons with other debiasing methods. We observe that Union_IG, while achieving better debiasing performance (e.g.. 53.82 stereotype score for RoBERTa-Basee severely impairs the language model's capability (91.70 - 30.61 of LMS). This is because Union IG indis- criminately suppresses all the neurons relating to different demographics, which inevitably damages other useful knowledge learned by the model. In contrast, method BNS maximizes the retention
7

---

Published as a conference paper at ICLR 2024
Model
- 50,00(A)
LMS
ICAT
BEKIIBase-cased
56.93
67.29
73.19
DPCE
62.41
78.48
59.97
50
AutolDbias
53.03
50.74
17.62
Unionn IG
51.01
31.47
30.83
40
BNS (Ours)
52.78
86.64
81.82
RoBERTO
30
FairBERS
RoBERTa-Base
62.46
91.70
68.85
DPCE
61.09
92.95
66.67
AutoDDehias
59.63
68.62
55.38
+ Union, IG
53.82
30.61
28.27
10
+ BNS (Our)
57.13
91.39
77.81
FaicBERTa
38.62
91.90
76.06
+ -Union_G
52.27
37.36
35.60
BBS (Ours)
33.44
91.05
84. 79
Layer
Table 5: Automatic evaluation results of debi- asing on StereoSet. SS, LMS, ICAT are short for Stereotype Score, Language Model Score and Idealized CAT Score, respectively. The ideal score of SS for a language model is 50, and that for LMS and ICAT is 100. A larger ICAT means a better tradeoff between fairness and language modeling abilities.
Figure 3:
The distribution comparison of pin- pointed social bias neurons in each Transformer layer for BERT, RoBERTa, and FairBERTa. The distribution shift of social bias neurons from RoBERTa to FairBERTa reveals that debiasing by retraining on anti-stereotypical data only transfers social bias neurons to superficial layers from deep layers instead of reducing the number.
of useful knowledge and only accurately locates neurons that cause distribution gaps for different social groups, achieving significantly better ICAT score of 84.79. These distribution gaps are the essential reasons why model outputs contain social biases.
When compared to other methods such as AutoDebias and FairBERTa, our BNS also performs significantly better. It's worth noting that FairBERTa requires model retraining, whereas BNS is training-free, only demanding minimal computational resources and being highly efficient.
ANALYSIS AND INSIGHTS
5.1 DISTRIBUTION SHIFT OF FAIRBERTA
Figure 3 compares the distribution of pinpointed social bias neurons for three pre-trained language models: BERT, RoBERTa, and FairBERTa. We find that the distribution of most social bias neurons for BERT and RoBERTa are concentrated in the deepest three layers (10th, 11th, 12th). By comparing observe that although there
the distribution of social bias neurons for RoBERTa and FairBERTa, has been little change quantity after debiasing (from 10.35 to 10.06 on average), there have been noteworthy alterations in the distribution of these neurons. Before debiasing, social bias neurons were predominantly concentrated in the deepest three layers (10th, 11th, 12th), with the highest concentration found in the deepest layer (12th), accounting for approximately 55.7% (BERT) and 36.4% (RoBERTa) of the total quantity. After debiasing, we notice that a substantial quantity (almost 1 1/3) of social bias neurons migrated from the deepest three layers (10th, 1ht, 12th) to the shallowest three layers (1st, 2nd, 3rd). We observe notable increase in the number of neurons in the two shallowest layers (1st, 2nd), rising from 2.9% (RoBERTa) to 26.0% (FairBERTa). Mcanwhile, the number of neurons in the deepest layer (12th) decreased 26.4%. Based on this phenomenon, speculate that social bias neurons have considerable adverse impact on the model outputs due to their proximity to the final output layer before debiasing. By pre- pre-training on the anti-stereony aeti-sttreotypccal data, FairBERTa transfers the social bias neurons from the deepest three layers the shallowest three layers to mitigate social biases. We analyze that the significant reduction of social bias neurons near the final output layer alleviates their impacts on model outputs, which could be the secret lying behind the effectiveness the debiasing method in FairBERTa.
5.2 STATISTICS OF SOCIAL BIAS NEURONS
Figure presents the average number of social bias neurons pinpointed by our IG2 for different demographic dimensions studied in this work. As shown, the average number varies for different demographic dimensions, with that for "Ethnicity" being the most (14.57) and that for "Gender"
8

---

Published as a conference paper at ICLR 2024
Ethnicity Avg. Intra
Physical Appearance
Politics Avg. Intra
Model
Avg. BNN
Avg. Inter
Avg. BN
Avg. Intra
Avg. Inter
Aug. BN
Avg. Inter
BERT
14.57
10.97
0.34
2.91
2.13
0.04
3.49
2.31
0.01
RoBERTa
14.05
8.01
0.38
3.17
1.13
0.02
4.75
3.06
0.00
FairBERTa
13.92
8.00
0:28
3.04
1.27
0.04
5.13
3.79
Table 6: Statistics of social bias neurons. "Avg. BN" means the average number of Bias Neurons pinpointed by our IG2. "Avg. Intra" means the average number of neurons in the intersection within the same JUT (only prompt templates are different). "Avg. Inter" means the average number of neurons in the intersection for different JUTs (either different UTs different Judgmental Modifiers).
being the fewest (1.09). Except for the "Ethnicity" dimension, we on average only need to suppress the activation of less than 4 neurons to effectively mitigate social biases for other dimensions.
Table shows the statistics of social bias neurons. As we can see, for each demo- graphic dimension, the same JUT shares most of the pinpointed social bias neurons, while different JUTs share almost cial bias neurons. The large value of Avg. Intra indicates that for the same JUT (con- firmed Judgemental Modifier and Unfair Target), pinpointed social bias neurons re- main almost the same for different prompt templates. This further verifies the accu- racy of the social bias neurons pinpointed by our IG², because this suggests that pin- pointed neurons are not related to the sen- tence structure or grammar of the prompt template, but only to the social bias in the studied social group. The small value of Avg. Inter indicates that social bias neu- rons pinpointed by our IG are quite different for different JUTs, even if prompt templates are the same for them.
#Bodal Neuuonn
355
2:91
2:50
Religio
Phpaacal
Potlip
Eislriy
Octupation
Seeaalty
Gendte
Damographic Dimensions
Figure 4: The average number of social bias neurons pinpointed in BERT for different demographic dimen- sions. Best viewed screen.
RELATED WORK
Since various Al applications permeate every aspect of our lives, research on Al Ethics (Liu et al. 2022 Mehrabi et al. 2019) has attracted more and more attention. In this work, we mainly explore one
important aspect of AT Ethies: AI Fairness, which has been studied from different perspectives (Hardt
ct al. 2016 John-Maan Mathews et al.) 2022; Nadcem et al. 2021 Nangia et al., 2020). (Liu et al.
2023 proposed to study the existence of annotator group bias in various real-world crowdsoureing datasets. (Li et al. 2022) measured hierarchical regional bias in pre-trained language models. Some works tried to detect and mitigate social biases in word embeddings (Bolukbasi et al. 2016) Kaneko al. 2022] and hidden representations (Chowdhury & Chaturvedi] (022), while others explored quantlying social biases in downstream tasks, Many works have explored the fairness problem in text classification tasks (Dixon et al./ 2018 Liu al. 2 Dinan al. 2020). Some works also explore the fairness problem in generation tasks, such as machine translation (Stanovsky et al. 2019), story generation (Lucy & Bamman, 2021). and question answering (Parrish al. 2022).. However, no work has focused on the interpretability of the fairness research. In this paper, we close this gap by proposing interpretable technique specific to the study of social bias along multiple dimensions.
7 LIMITATIONS AND CONCLUSION
In this paper, we propose novel interpretable technique, Integrated Gap Gradients (IG"), to precisely identify social bias neurons in pre-trained language models. We also develop new dataset to facilitate the interpretability study social bias. Derived from our interpretable technique, BIAS NEURON SUPPRESSION (BNS) is further proposed to mitigate social bias. Extensive experiments have verified the effectiveness of our IG2 and BNS. In addition, facilitated by our interpretable method, we analyze the distribution shift of social bias neurons after debiasing and obtain useful insights that bring inspiration future fairness research.

---

Published as a conference paper at ICLR 2024
Limitations. While study provides valuable insights, recognize there exist limitations. For example, our proposed BNS method directly sets the activation values of selected social bias neurons to zero. Although this is effective, designing more refined suppression method might yield even better results. These present opportunities for future research.
REFERENCES
Afra Feyza Akyürek, Sejin Paik, Muhammed Yusuf Koeyigit, Seda Akbiyik, (S)erife Leman Runyun, and Derry Wijaya. On measuring social biases in prompt-based multi-task learning. 2022.
Haozhe An and Rachel Rudinger. Nichelle and nancy: The influence of demographic attributes and tokenization length first name biases. May 2023.
Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai. Man is to computer programmer as woman is to homemaker? debiasing word embeddings, 2016. URL
https://arxiv http:///aaxiv.oooabbb/////////// wttpp///160000000 06520
Somnath Basu Roy Chowdhury and Snigdha Chaturvedi. Learning fair representations via rate- distortion maximization. arXiv preprint arXiv:2202.00033: 2022.
AidaMostaaazadde Davani, Mohammad Atari, Brendan Kennedy, and Morteza Dehghani. Hate speech classifiers learn human-like social stereotypes.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional Transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume pp. 4171 -186,, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
Emily Dinan, Angela Fan, Ledell Wu, Jason Weston, Douwe Kiela, and Adina Williams. Multi- dimensional gender bias classification. arXiv preprint arXiv:2005.006144 2020.
Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Measuring and mitigating unintended bias in text classification. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pp. 67-73, 2018
Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard H. Hovy, Hinrich Schütze, and Yoav Goldberg. Measuring and improving consistency in pretrained language models. CoRR, abs/2102.01017,, 2021. URL
wwtp://rrr/bb///....oogo 010.001177
Shangbin Feng, Chan Young Park, Yuhan Liu, and Yulia Tsvetkov. From pretraining data to language models to downstream tasks: Tracking the trails of political biases leading to unfair nlp models. May 2023.
Eve Fleisig, Aubrie Amstutz, Chad Atalla, Su, Lin Blodgett, Hal Daumé, Alexandra Olteanu, Emily Sheng, Dan Vann, and Hanna Wallach. Fairprism: Evaluating muiness--elatte harms in text generation.
Seraphina Coldfarb-Tarrant, Adam Lopez, Roi Blanco, and Diego Marcheggiani. Bias beyond english: Counterfactual tests for bias in sentiment analysis in four languages. May 2023.
Yue Guo, Yi Yang, and Ahmed Abbasi. Auto-debias: Debiasing masked language models with automated biased prompts. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), Pp. 1012 1023, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 12118665/1120222aaalloog https://aclaatioooo...////////
URL
long. .72
Moritz Hardt, Eric Price, and Nathan Srebro. Equality of opportunity supervised learning, 2016.
URL https:///arxivi wttp::/...oraaaa///booa// 02413
Minqing Hu and Bing Liu. Mining and summarizing customer reviews. In KDD '04: Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 168-177, New York, NY, USA, 2004. ACM. ISBN 1-58113 13-888-1. doi: http://di..ccmorrg// 1145/1014052.1 1014073.
10

---

Published as a conference paper at ICLR 2024
Jean-Marie
Dominique Cardon, and Christine Balagué. From reality to world. critical perspective on ai fairness. Journal of Business Ethics, pp. 1-13, 2022.
Masahiro Kaneko and Danushka Bollegala. Debiasing pee- ve-ttrinee contextualised embeddings.
Cornell University arXiiv, Cornell University arXiv, Jan 2021.
Masahiro Kaneko, Danushka Bollegala, and Naoaki Okazaki. Gender bias in meta-embeddings.
arXiv preprint arw.i:2205.000000000 2022.
Yizhi Li, Ge Zhang, Bohao Yang. Chenghua Lin, Anton Ragni, Shi Wang. and Jie Fu. HERB: Measuring hierarchical regional bias in pre-trained language models. In Findings of the Association for Computational Linguistics: AACL-IJCNLP 2022. Association for Computational Linguisties, November 2022. URL /ttpttts////caatttoo
wwg.g//g..........o findings- aacl. 32
Haochen Liu, Wei Jin, Hamid Karimi, Zitao Liu, and Jiliang Tang. The authors matter: Understanding and mitigating implicit bias in deep text classification. arXiv preprint arXiv:2/05.02778, 2021.
Haochen Liu, Yiqi Wang, Wenqi Fan, Xiaorui Liu, Yaxin Li, Shaili Jain, Yunhao Liu, Anil Jain, and Jillang Tang. Trustworthy ai: computational perspective. ACM Transactions on Intelligent Systems and Technology, 14(1):1-5) 2022.
Haochen Liu, Joseph Thekinen, Sinem Mollaoglu, Da Tang. Ji Yang. Youlong Cheng, Hui Liu, and Jiliang Tang. Toward annotator group bias in crowdsourcing. 2023.
Yinhan Liu, Myle Ott. Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv: /907. 1692, 2019.
Li Lucy and David Bamman. Gender and representation bias in GPT-3 generated stories. In Proceedings of the Third Workshop on Narrative Understanding, pp. 48-55, Virtual, June 2021. Association for Computational Linguistics. doi: 10.18653//1/20211mmmm -nweee22...uue1.... 1.5. URL https://
aclanthology.co .rgg 202g 1.5e-11.5
Nicholas Meade, Elinor Poole-Dayan and Siva Reddy. An empirical survey of the effectiveness of debiasing techniques for pre-trained language models, 2021. http::tsss///iii.rrrrrerr 2110. 08527
Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey on bias and fairness in machine learning. arXiv: Learning, 2019.
Moin Nadeem, Anna Bethke, and Siva Reddy. Stereoset: Measuring stereotypical bias in pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Jan 2021. doi: 10. 10.18 long-oooo..16666 URL http:://d// wwt.:///.oooooo/o/.// 26653/v1/205 acl long acl-alng......... 1166
Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. CrowS-pairs: A challenge dataset for measuring social biases in masked language models. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), Online, November 2020. Association for Computational Linguistics. doi: 10. 10.18655//1/20220emlmmmmmo 154. URL https://aclanthoooogy org/2000 emn.p-main. 154
Nedjma Ousidhoum, Xinran Zhao, Tianqing Fang, Yangqiu Song, and Dit-Yan Yeung. Probing toxic content in large pre-trained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online, August 2021. Association for Computational Linguistics.
Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman. BBQ: A hand-built bias benchmark for question answering. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 2086-2105. Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.1 (0.186533vv1/022 http:////2222..nnn-eeea. findings-acl. 165. URL http:/https////claaooooooo.ooo wwwp:/gg...ggg.2..o...// http:///22222..nnngggeeee findings- 165.
11

---

Published as a conference paper at ICLR 2024
Rebecca Qian, Candace Ross, Jude Fernandes, Eric Smith, Douwe Kiela, and Adina Williams. Perturbation augmentation for fairer nip. May 2022.
Omar Shaikh, Hongxin Zhang, William Held, Michael Bernstein, and Diyi Yang. On second thought, let's not think step by step! bias and toxicity in zero-shot reasoning. Dec 2022.
Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. The woman worked as babysitter: On biases in language generation. empirical methods in natural language processing, 2019.
Gabriel Stanovsky, Noah A. Smith, and Luke Zettlemoyer. Evaluating gender bias in machine translation. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1679-1684, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 10.186533v1/P19-11 1164. URL. hwwphhhooooo...rr/1111111..
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning, JCML 2017, volume 70 of Proceedings of Machine Learning Research, pp. 3328. PMLR, 2017. URL http:// proceedings. mlr. ptt.s./.rees////// /rress/voo/uuntaarrajaaal 7a. html
Jesse Vig. Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. In Larochelle, M. Ranzato, Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33. pp. 12388-12444 12401. Cur- ran Associates, Inc., 2020. URL htttps://proceedings.. neurrdinnnnnneiiippge wwt.iinc.ccpppprr///eee/ 22220//ill 22260292227755511117777922222222
Kellie Webster, Xuezhi Wang. Ian Tenney, Alex Beutel, Emily Pitler, Ellie Pavlick, Jilin Chen, and Slav Petrov. Measuring and reducing gendered correlations in pre-trained models. arXiv: Computation and Language, 2020.
12


Reviews:
[{"review_id": "9YAhwCcxKL", "reviewer": "Reviewer_jE7C", "summary": "This paper aims to understand why the bias happens in LMs. They use integrated gap gradients (IG^2) to identify which neurons in language models contribute to the social bias. With the identified neurons, the authors proposed bias neuron suppression method to reduce the bias in LMs. Experimenting with BERT, RoBERTa and FairBERTa, the proposed IG^2 method demonstrates to reduce the bias.", "questions": "- Does the method only work when the output of the model is to predict the sensitive attributes (so that you can calculate the logit gap)?\n- What model is being examined in Fig.2? Does the conclusion hold for all models, especially for FairBERTa? \n- It seems most of the time, Union_IG can make the bias score lower. Is it because Union_IG contains more neurons? Does this mean adding more neurons generally helps reduce the bias? How sensitive is the result w.r.t. $\\sigma$?\n- What if you replace Union_IG to Intersection_IG, i.e., the intersection of bias neurons for different attributes?\n- Table 5, it should be \"StereoSet\".", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Extending IG method to understand the biases in language models, where the topic is interesting and important. The authors did a good job of structuring the paper, so it is easy to follow.", "weaknesses": "It seems the IG^2 requires calculating the gap between the logits of different attributes. I'm wondering how general will it be, e.g., how can we extend it to other tasks instead of predicting the sensitive attributes? For example, what if the task is a hate speech detection?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zfRgbzyQg8", "reviewer": "Reviewer_GcZ6", "summary": "The paper proposes INTEGRATED GAP GRADIENTS (IG2) approach that identifies problematic neurons in a language model that contribute to undesirable behavior, such as bias. They then use this identification framework and propose BIAS NEURON SUPPRESSION (BNS) to mitigate the effect of these problematic neurons to reduce the observed biases in LLMs. Authors also perform experiments to showcase effectiveness of their detection and mitigation approaches.", "questions": "For results in Table 5 under FairBERTa why are some approaches missing? Some clarification on this can be helpful.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is easy to follow and is written clearly.\n2. The approach is easy to implement and be applied.\n3. The paper studies an important issue.", "weaknesses": "1. The technical contribution of the paper is not significant as it simply extends a previous existing approach for identification of problematic neurons in the context of bias and fairness.\n2. The experiments are done only on masked language modeling tasks which can limit the study.\n3. The approach operates between binary demographics and might be limited when we want to study biases amongst large pool of fine-grained demographics.\n4. Their constructed dataset only covers few demographics.\n5. The paraphrases for constructing the dataset are generated using a language model which can impose its own biases in the study.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FZBzmNwr0h", "reviewer": "Reviewer_zYP2", "summary": "This paper investigates the suitability of interpretability techniques used to attribute individual neurons to model generations for the study of social bias in neural networks. To do so, the authors introduce Integrated Gap Gradients (IG2) as an approach to identify neurons attributable to social bias. Their approach is evaluated on a dataset of around 200000 template-generated samples. The evaluation is carried out by either suppressing or amplifying the detected neurons, and the experimental results demonstrate that doing so indeed reduces or increases the generation of biased outputs. Based on these results, the authors propose a debiasing approach based on the suppression of detected neurons and evaluate it on the StereoSet benchmark. Comparing this approach to a range of baselines, the authors show that the method successfully debiases model predictions while at the same time having little impact on unrelated model predictions (measured via the Language Modeling Score).", "questions": "* Did you experiment with regularizing detected neurons rather than suppressing them entirely? Considering their suppression more gradually as opposed to zero-ing them out might help in increasing the Language Modeling Score while keeping a high Stereotype Score.\n* Not really a question but more a suggestion: I’d recommend moving the introduction of BNS to after the first report of experiments to improve the flow of the paper.\n* Would it be worth adding a few examples (rather than only templates) to Table 3?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper combines interpretability techniques with an application to social bias research in neural networks and shows that doing so yields valuable insights.\n* The results are promising and might represent a foundation for future work applying the method to other applications in NLP.\n* The paper is well-written and the experiments are overall thorough.", "weaknesses": "* The paper could spend more attention on the implications of the presented results. Since the authors mention the importance of fairness research in this context, a brief discussion on the usability, practicability, and limitations of their results would be desirable.\n* The paper does not discuss potential implications of BNS on other tasks that the investigated LMs are faced with. While this is difficult to measure directly, it remains unclear to what extent the manipulation of neurons at inference time impacts model performance on other tasks. I’d recommend the authors to address this in the manuscript carefully.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Tp997HcTkc", "reviewer": "Reviewer_pmu7", "summary": "The paper introduces a new method, Integrated Gap Gradients (IG2), for measuring how much an individual neuron contributes to differences between logit predictions at the outputs. Combined with curated prompts that have positive or negative sentiment and are intended to be completed with a demographic group, this method is used to identify neurons in the network that most strongly contribute to differences between demographic groups along different dimensions, including gender, ethnicity, etc. Once these neurons are identified, the paper applies a new technique, Bias Neuron Suppression (BNS), to zero out the activations of neurons that are above a threshold score. The paper shows that when identified bias neurons are amplified, demographic gaps increase, and when they are suppressed, demographic gaps decrease. The paper also shows improvement on scores based on StereoSet for measuring the demographic bias of a language model. Finally, the paper also presents interesting empirical results comparing to other debiasing methods, showing for example that the number of bias neurons in FairBERTa does not decrease. Instead, the neurons get shifted into earlier layers compared to networks that have not been debiased. Overall, the paper demonstrates the promise of this new technique.", "questions": "Figure 2: it’s unclear to me from the caption what exactly the x-axis is plotting. Can the authors clarify?\n\nTable 6: Could use a bit more explanation about what the “inter” and “intra” columns represent\n\nQuestions about limitations\n\nIs there any empirical evidence or motivation, aside from the results here, to support the assumption that single neurons can strongly contribute to demographic disparities?\n\nIs there a way to generalize the method to take into account interactions or correlations between neurons that might affect the result?\nDo you think the method is limited by the kinds of prompts that you have templated here? How would you generalize?\n\nCan this method be extended beyond differences in sentiment for demographic groups?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The paper’s primary strengths are the empirical results and the clarity of the new method proposed. The work is an interesting extension of the integrated gradients technique to measurements of demographic bias, and the results seem to support the efficacy of the method well. The work is clearly written and easy to follow as well.", "weaknesses": "The primary weakness of the work is the fact that the paper primarily relies on StereoSet and associated metrics to measure the effectiveness of the debiasing technique. While this is not a major limitation, it might have been nice to evaluate on another dataset with targeted queries for a specific group, such as WinoQueer (https://arxiv.org/abs/2306.15087). Another weakness of the work is a more thorough discussion of the limitations of the method, and I have some questions in the following section that could be used to expand this discussion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "te4jkem5LC", "reviewer": "Reviewer_auwC", "summary": "The paper presents an interesting approach called Integrated Gap Gradients that aims to identify the neurons in a large language model that are responsible for the exhibition of undesirable behavior like social bias. Also, the paper presents a technique called Bias Neuron Suppression that aims to suppress the detected neurons that are responsible for the undesirable behavior, hence mitigating social biases in large language models. The paper performs an extensive evaluation comparing the approach to multiple baselines, demonstrating the merits of the proposed approach in mitigating social biases and being more fair.", "questions": "1. How are the demographic dimensions in Table 1 constructed?\n2. How do you envision the proposed approaches to be applied for other purposes beyond debiasing?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper focuses on an important issue, and I believe that the proposed approaches can have a significant impact on the research community that aims to study social biases and ways to mitigate these issues in large language models. The proposed approach is a significant step toward more interpretable Artificial Intelligence and aims to mitigate social biases without the need for expensive re-training or fine-tuning of very large language models like BERT or RoBERTa. The paper makes an extensive evaluation of the proposed approach and compares it with multiple baselines such as DPCE, AutoDebias, and Union_IG, demonstrating that the proposed approach outperforms them in terms of various bias metrics. Also, I really liked that the paper makes an extensive evaluation, including a large number of demographic attributes as well as modifiers. Overall, I believe that this is a good paper submission, and I believe it can advance research and state-of-the-art methods in mitigating social biases that arise from the use of large language models.", "weaknesses": "Overall, I do not have major concerns with the paper, and I have only some mainly minor issues/suggestions for the authors to further improve their paper. First, it will be good to clarify how you came up with the Demographic dimensions that are included in Table 1. Is this based on previous work or did you construct these dimensions for the purposes of this work? Also, for figure 2, the paper does not clarify what model is used to generate the results? Is this based on the BERT model? I suggest to the authors to clarify this and also discuss if they observe any differences in the presented results for Figure 2 across models. Finally, I would have liked to see a more detailed discussion and explanation of how the authors envision the use of the proposed approaches in other applications beyond social biases and debasing. The authors touch briefly upon this in the Introduction, however, they do not go into details, and it is unclear from a reader’s perspective how the proposed approaches can be applied for other purposes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)