Debiasing Algorithm through Model Adaptation
Authors: Tomasz Limisiewicz
David Mareček
Tomáš Musil
Venue: ICLR 2024 poster (2024)
Keywords: Model Editingi
Gender Bias
Causal Inference
Primary area: societal considerations including fairness, safety, privacy

Abstract: Large language models are becoming the go-to solution for the ever-growing number of tasks.
However, with growing capacity, models are prone to rely on spurious correlations stemming from biases and stereotypes present in the training data.
This work proposes a novel method for detecting and mitigating gender bias in language models.
We perform causal analysis to identify problematic model components and discover that mid-upper feed-forward layers are most prone to convey bias.
Based on the analysis results, we intervene in the model by applying a linear projection to the weight matrices of these layers.
Our titular method DAMA, significantly decreases bias as measured by diverse metrics while maintaining the model's performance on downstream tasks.
We release code for our method and models, which retrain LLaMA's state-of-the-art performance while being significantly less biased.

Published as a conference paper at ICLR 2024
DEBIASING ALGORITHM THROUGH MODEL ADAPTATION
Tomasz Limisiewicz
David Mareček
Tomás Musil
Faculty of Mathematics and Physies, Charles University (1 limm 8 siewicz, ma recek, musil
@u m
auff.cuni.co
ABSTRACT
Large language models are becoming the go-to solution for the ever-growing num- ber of tasks. However, with growing capacity, models are prone to rely spu- rious correlations stemming from biases and stereotypes present in the training data. This work proposes a novel method for detecting and mitigating gender bias in language models. We perform causal analysis to identify problematic model components and discover that mid-upper feed-forward layers most prone convey bias. Based on the analysis results, we intervene in the model by apply- ing linear projection to the weight matrices of these layers. Our titular method DAMA significantly decreases bias as measured by diverse metries while main- taining the model's performance on downstream tasks. We release code for our method and models, which retrain ILaMA's state-of-the-art performance while being significantly less biased!
INTRODUCTION
Large language models have a large capacity for learning linguistic and factual information from training data, but they prone capture unwanted biases. It has been shown that LLMs are gen- der biased (Stanczak & Augenstein] 2021 Blodgett et al. 2020 van der Wal et al. 2023 Nadeem al., 2021 Nangia et al. 2020; Limisiewiez & Mareček] 2022). This bias is manifested by re- lying on a spurious correlation between seemingly gender expressions and specific gender. For instance, language models tend ascribe stereotypical gender certain practitioners, e.g. by outputting high probabilities for phrases such as "male mechanics" or "female cleaners" (Lu et al.) 2020b). In many tasks, the models also show uneven performance for the test examples involving different gender contexts.
This work analyzes the LLaMA family of models (Touvron et al. 2023). These openly available models obtain state-of-the-arr performance on variety of downstream tasks. We focus specifically on the gender bias present in these models, but our method is applicable to other types of bias as well. We specifically ask: 1) Can we identify evidence of gender bias in LLaMA? Specifically, do they associate professional names with the stereotypical gender? 2) Can we identify which components of the model store the gender-b biased representation? 3) Can edit the model's 's weights decrease the bias while preserving its performance on end-tasks?
To answer the first question, check the LLaMA performance popular tests for gender bias: WinoBias (Zhao et al. 2018) and StereoSet (Nadeem et al. (02T). We introduce an interpretable metric that evaluates bias the language generation task. To answer the second question, per- form causal tracing (Vig et al. 2020 Meng et al.) 2022a). We monitor changes in the distribution of predictions when the stereotypical representation revealed only in one of the components, such as MLP (multilayer perceptron) or attention layer. Following the terminology of Pearl (200T), we call such component gender bias mediator. To tackle the last question, we introduce "Dehiasing A Algorithm through Model Adaptation". In DAMA, we edit bias-vulnerable feed-forward layers by multiplying linear transformation weights by the orthogonal projection matrix similar to Ravfogel et al. (2022). Our results show that with directed changes in model weights, we can reduce gender bias substantially while having only a minimal impact on the model's 's performance. Specifically,
'The code available
gi github. ccm/ cm 1 imm /DAMA

---

Published as a conference paper at ICLR 2024
"The lifeguard laughed because
20%
(a)
10%
LLaMA
WFF
he she the
Atir
FF
20%
DAMA
U
10%
LL
Atin
FF
I- R
PeW
he she the (d)
(c)
Figure I: Schema (b) shows DAMA intervention in a LLaMA layer. Even though T- P. is depicted as a separate module, practice, it is multiplied with the output matrix of a feed-forward layer (WFF). Therefore, DAMA is neutral to the model's parameter count and architecture. (a) We show the behavior the model when presented with a stereotypical prompt. Specifically, (c) shows the projections of the feed-forward latent vector (i)) onto the output space. With DAMA (lower arrow), we nullify the gender component of the representation. results balanced probabilities of gendered tokens in the model's 's output, shown in (d).
we monitor performance changes in language modeling (measured by perplexity) and in four down- stream tasks.
We evaluate gender bias in LLaMA models and introduce a novel, trans- parent metric for quantifying bias directly in language generation. Most importantly, we propose DAMA, a method for editing weights the bias mediator to significantly reduce gender bias in three different tasks without sacrificing performance across unrelated tasks. This is improvement over prior methods that were focused on one type of bias manifestation (Ranaldi et al. (202)) or were not
To list our contributions:
tested for preserving language understanding capabilities of the model (Lauscher al. 2021 Gira
et al. 2022
2
METHODOLOGY AND EXPERIMENTAL SETUP
2.1 LLAMA MODELS
LLaMA models causal language models following Transformer decoder architecture (Vaswani et al. 2017). LLaMA family contains models with 7B, 13B, 30B, and 65B parameters. The original paper Touvron al. 2023) presented state-of-the-art results multiple downstream tasks, which we also use for evaluation. In our implementation, we used the model checkpoint accessible through the Huggingface library buuginggfcce col Due to the large size of the models, used half- precision weights, which we observed to have no significant impact on the results.
2.2 GENDER BIAS EVALUATION IN LANGUAGE GENERATION
To better understand gender bias in language generation, we construct our dataset of prompts and an interpretable diagnostic measure.
We use the set of professions chosen and annotated by Bolukbasi et al. 2016 Each profes- sion was assigned two scores: factual score xf (originally called definitionality) and stereotypical score Z.. They define how strongly a word is connected with the male or female gender respec- tively through semantically or through stereotypical cues. By convention, scores range from for
The data is available at: professi ons jaon
debi aswe/bl http:////bb//msttteeee ob/master/datall
2

---

Published as a conference paper at ICLR 2024
female-assen associated words to for male ones We fill the proposed profession words in the prompts of the structure presented in Figure la The lifeguard is, by definition, a gender-neutral word (z/ 0)
and associated with the male gender by a stereotypical cue (z, 0.6). We measure the probabilities for gendered prediction for a given prompt Pas (o( For that purpose, we use pronouns 04 "he" and "she", as they probable continuations for given prompts.
Subsequently for each prompt, we compute empirical score y = Par (X) (o To
estimate the relationship between the observed score and annotated ones 2, and y,, construct linear model:
y == Oz Fs + aaf 2 o +
The linear fit coefficients have the following interpretations: a, is an impact of stereotypical signal on the model's predictions; af is an impact of the factual (semantic) gender of the word. Noticeably, y, u. and æ take the values in the same range. The slope coefficient tells how shifts in annotated scores across professions impact the difference in prediction probabilities of male and female pro- nouns. The intercept by measures how much more probable the male than the female pronouns are when we marginalize the subject. We provide the details on the prompt selection and test train splits Appendix
2.3
OTHER GENDER BIAS INDICATORS
We also evaluate other well-established methods of evaluating gender bias manifestations in two downstream tasks:
WinoBias Zhao al. (2018) present the dataset containing a WinoGrad scheme (Levesque al, 2011) examples. Each example contains two gender-neutral profession names and gendered pro- nouns. The task to identify the coreference link between the pronouns and the correct profes- sional. The dataset consists of two parts: pro-stereotypical, where coreference links profession name with stereotypical gender matching the gender of the pronoum; in anti-stereotypical examples, the profession's 's stereotypically assumed gender is different from the gender of the pronouns. An example from the anti-stereotypical subset: "The nurse notified the patient that his shift would be ending in hour."
We evaluate the general accuracy of coreference resolution Acc. Inspired by the analysis of Stanovsky et al. (2019). we introduce two measures of bias: AS: the difference between the accu- racy of pro and anti-stereotypica examples, and AG: the difference between the accuracy between examples containing male and female pronouns.
StereoSet Nadeem et al. (021) propose a dataset for measuring stereotypical biases on four do- mains. We consider only gender domain and intrasentence task. This task contains 765 sentences, each with gap and three possible completions: stereotypical, anti-stereotypical, and meaningless. E.g., a sentence Girls tend be more fish. The authors propose three evaluation measures: 1) Ims the percentage of sentences where the model prefers the meaningful over the meaningless completion: 2) the percentage of sentences where the model prefers the stereotypical over the anti-stereotypical completion; and 3) icul score that combines the previous two: ical = Ims min(ss, 100 - 8s)/50. Note that typically lower 8s scores refer to less biased models since they are closer to 50.
than boys. and three completions soft, determined, and
2.4 LANGUAGE MODELING
To evaluate the performance of the model's pre-training task, we measure perplexity on the Wikipedia 103 corpus (Merity et al. 2016) available through HuggingFace.
We positive values for male gender following the original paper. This is only arbitrary choice, and switching polarities wouldn't 't affect this analysis. Importantly, do not intend ascribe negative valuations to any the genders. 'In this example, the coreferential link relies semantics, while in other instances, coreference can be ved solely through syntax.
3

---

Published as a conference paper at ICLR 2024
Factual
Stereotypical
Intercept
R2a
First subject teleen
0.2
0.1
Middle subject toloms
0.2
subject token
0.0
0.0
0.0
First subeequent token
Further toloms
16 20 25
10
12 10
24 28
12 16
20
Figure 2: Causal tracing of factual af. stereotypical a, coefficients and intercept b in regression indirect effects of the model yrg. The linear models are independently fitted for restored MI.P clean representation at each layer and token position.
2.5 DOWNSTREAM TASKS
We have selected three datasets that measure common sense reasoning and language understanding to evaluate the possible performance loss after altering the model: OpenBookQA (OBQA) (Mi-) haylov al. 2018) contains 500 multiple-choice questions aimed combining science facts with common knowledge. AT2 Reasoning Challenge (A (ARC) (Clark et al.] (2018) contains natural sci- ence questions authored for use on standardized tests. is partitioned into a Challenge Set (1172 test questions) and an Easy Set (2376 test questions). Massive Multitask Language Understand- ing (MMLU) (Hendrycks et al. 2021) contains 14 042 questions on 57 topics, including math, law, social sciences. The former two tasks evaluated in zero-shot regime. In the MMLU, provide five in-context examples. In all the evaluations, we followed closely the original setting of Touvron al. (2023).
BIAS EVALUATION AND CAUSAL TRACING
3.1
EXPERIMENTS
Bías Evaluation We assess gender bias in LLaMA by employing the linear model outlined in Section 2.2 We compare the lincar coefficients: the larger the coefficient, the more the model is biased. We also measure the bias scores for the WinoBias and StereoSet datasets.
Causal Tracing To identify the components storing gendered associations, we perform causal tracing for gender bias in text generation. We use a similar methodology as Meng al. (2022a). For each test prompt, (1) we perform clean run and collect all the activations at all layers and tokens; (2) we perform a corrupted run by adding noise the tokens of the profession (details in Appendix ); (3) we perform corrupted runs with restoration: at each step, we restore the activations from the clean runof each output of MLP one particular layer and token. For each layer 1, token position i, and prompt X we compute the score yt,,(X' (r, (X) Pi. By fitting the linear model (Equation across the prompts X, we get the as and a scores for each layer and token position i. Following Meng al. (2022b), aggregate token positions into six groups shared across the whole dataset: first, middle, last subject token, first subsequent token, further tokens, and the last token.
3.2 RESULTS
Bias Evaluation We show the coefficient of the linear model in Table We see that the linear model proposed by is moderately well fitted for all sizes of L.LaMA models R² 0.35. For all sizes, the factual coefficient is higher than the stereotypical one. The models more influenced by semantical than stereotypical cues (a, > a.). Also, we observe positive intercept all cases, showing that LLaMA models are more likely predict male than female pronouns.
Similarly, other metrics confirm that LLaMA models are biased in coreference resolution and sen- tence likelihood estimation. In WinoBias, observe that the bias stemming from stereotypes AS more prominent than the accuracy difference between examples with male and female pronouns AG.

---

Published as a conference paper at ICLR 2024
Bias in LM
WinoBias
SterenSet gender
4 4xx
4
1 Acc
4 AAS
AG
ss
ICAT
MEMIT
0.209
0,282
0.071
0.497
59.3%
40.5%.
3.3%
95.6
72.0
53.6
LoRA FT
0.144
0.261
-0.40
0.413
58.8%
34.4%
5.6%
89.0
62.9
66.0
LLaMA
40.3%
71.9
DAMA
-0.005
0.038
-0,006
0.208
57.3%
31.5%
2.3%
95.5
69.3
58.5
+ (std)
0.004
0.004
0.5%
0.9%
0.7%
0.8
LLaMA 13B
0.270
0.351
0.070
0.541
70.5%
55.7%
-1.5%
95.2
71.4
54.4
DAMA
0.148
0.222
0.059
0.472
66.4%
31.1%
-1.1%
94.4
68.6
59.4
LLaMA
71.0%:
36.0%:
4.0%
68.4
DAMA
0.105
0.172
0.059
0.471
63.7%
26.7%
-3.7%
94.8
65.7
65.0
LLaMA 65B
0.249
0.316
0.095
0,490
73.3%
35.7%:
1.4%
94.9
69.5
57.9
DAMA
0.185
0.251
0.100
0.414
71.1%
27.25
0.84
92.8
67.1
61.1
Table 1: Bias evaluation for the LLaMA models and their debiased instances Significance analysis for the 7B model was performed by running DAMA with five random seeds. We bold the score for the original model or DAMA, whichever is better if there are more than two standard deviations apart. We underline the best value each column.
LM
Downstream
L PPL
ARC-C
ARC-E
OBQQ
1 MMLU
26.1
68.9
LoRA FT
51.1
37.7
66.5
45.6
26.6
ILaMA 7B
26.1
42.2
69.1
57.2
30.3
DAMA
28.9
41.8
68.3
56.2
30.8
0.2
0.2
0.5
LLaMA
70.6
55.4
DAMA
21.0
44.7
70.3
56.2
43.5
LLaMA
20.5
72,9
DAMA
19.6
45.2
71.6
58.2
$6.1
LLaMA 65B
19.5
44.5
73,9
59.6
DAMA
20.1
40.5
67.7
57.2
Table 2: Performance evaluation for the LLaMA models and their debiased instances. The signif- icance analysis performed the same in Table [ (*) Due hardware limitations, could not run MMLU inference for 65B models. In the evaluation of 30B model, we excluded 4% longest prompts.
we observe the indirect effect of MLPs in each layer and token position of the 7B model. The best fit is obtained for the representation in the lower layers (0-5) at the subject position and mid-upper layers (18 -25) at the last position. In the search for stereotypically biased components, we direct our attention the mid-upper layers because they appear covey less signal about factual gender. We also expect that the information stored in those MI.P layers is more likely generalize unseen subjects. Interestingly, the last layers manifest weak negative slope coefficients, suggesting that these MLPs tend to counter the bias of the models.
Causal Tracing Figure 2
In Figure 4 (in Appendix B we show the results of casual tracing for attention and the whole layer. For those components, the high indirect effects are distributed more extensively across both token positions and layers, indicating that they primarily reflect bias from the MLPs. For larger models, we observe analogous patterns shifted according to the total layer count.
4 DEBIASING ALGORITHM THROUGH MODEL ADAPTATION
We introduce the algorithm that decreases bias language models by directly editing the model weights. This section describes method based on projection-based intervention selected layers, called DAMA. Further, we provide theoretical and empirical backing for the method's 'ss effec- tiveness.
5

---

Published as a conference paper at ICLR 2024
4.1 OBTAINING STEREOTYPE KEYS AND GENDERED VALUES
Following the convention from Geva et al. (2021). we treat MLP layers as memory units mapping specific input key representations value representations. Our focus lies in understanding how these layers map stereotypical keys to gendered values. As our choice of keys, we take prompts introduced in Section 2.2 which carry stereotypical signal. The values are the output vectors corresponding to one of the personal pronouns (male, female, or neutral).
To compute the stereotypical key /th layer, we feed the stereotypical prompt X up / layer's feed-forward MLP (F Fi) to obtain its vector representation. We, specifically, take the vector repre- sentation at the last token of the prompt. We denote stereotypical keys as E Rdrr following the convention from Figure Ic
To compute the value representation corresponding to a specific gender, we employ the next-token prediction task based on the stereotypical prompt X. As possible next token, we consider one of the pronouns indicating gender (O_ "he" for male, O_ = "she" for female, and Og = "they" for neutral). We use the regular cross-entropy loss and optimize the output of the /th layer's ' feed- forward denoted V:
V/ = arg min log (M]x=s](o|X)) +1 A + |=||=|1 zER
(2)
The second part of the loss is added to preserve the model's s LM capabilities for predicting the next token (o') given general (not-biased) prompts (X'). The last summand is L2 regularization, We use gradient descent with 20 iterations to obtain a value vector for each of the pronouns vo E Rdd..
4.2 OBTAINING PROJECTION ON STEREOTYPE SUBSPACE WITH PLS
To identify the stereotype subspace, we concatenate value vectors for each pronoun (male, neutral, and female) across all prompts to obtain gendered value matrices V+, Vo. and V_. The gendered value matrices normalized by subtracting the mean calculated across all three pronouns for given prompt. Analogically, we concatenate key vectors for all prompts into one matrix U. Then, we multiply by the feed-forward's output matrix denoted W FF
WFF.out. . -
We concatenate V+, Vo. and V- together and concatenate Ú three times. We use the Partial Least Squares algorithm to identify the linear mapping B1 maximizing correlation between stereotypical keys Ü, Ü, 01 and gendered values [V+, Vo, V-]:
[V+, Vo, V-] RPLS B₁ CG-1,,,,Ú1+
By definition of PLS, Bj identifies the stereotypical directions most correlated with gendered val- ues 3 Therefore, we compute the matrix projecting representation on subspace orthogonal to the one spanned by de first columns of B, to nullify the stereotypical signal. For brevity, denote the trimmed matrix Bd= B1 [,, de]. The projection is given by the equation:
(5) Finally, we perform the model editing by multiplying Ith MLP feed-forward matrix FFF,out by the projection matrix P, see Figure Ic Our algorithm DAMA is based on iterative computation and applying projections to feed-forwards of multiple subsequent MLP layers. It changes neither the model's architecture nor parameter sizes, as the result of matrix multiplication is of the same dimensionality as the original feed-forwart matrix.
1= P. = 1 - Bd (BATT pdlo)-1
4.3 THEORETICAL PERSPECTIVE
In this section, we show theoretical guarantees that multiplying linear feed- foeewro matrix W WFFoout by projection matrix P will be the optimal mapping between keys (U) and values (V), fulfilling that W FF U is orthogonal to the guarded bias subspace C.
Matrix Bg can be used to normalize the value matrix. However, have noticed that its loadings become nearly zero due the earlier normalization of [V4, Vo, V-].

---

Published as a conference paper at ICLR 2024
Theorem 1. Assume that have linear subspace C S R.. Given n-element key matrix U Rix" a value matrix V E R we search a mapping matrix W E Rox: minimizing the least squares and satisfying "== Ww, C. Specifically, solve:
W arg min IIW - VIl. such that W
V Vi Wu; 1. C
This equation is solved by:
W = -LL--
Where Po is projection matrix on a subspace C.
The proof of the theorem is in Appendix A Noteworthy VU" (UUT)-I solves the regular mean square problem of mapping prompt keys values corresponding the model's output. Due to gradient optimization in the model's pre-training, we can assume that in general case WFF VU (UD?)) Thus, the application of projections would break the correlation between stereotypical keys and gendered values without affecting other correlations stored by the MLP layer.
4.4 EMPIRICAL PERSPECTIVE
Effectivness We apply DAMA to MLPs approximately one-third of the model's s upper layers (in LLaMA 7B layers 21 29 out of 32 with projection dimensionality de 256). In the previous sec- tion, we have shown that those layers are the most prone to stereotypical bias. We check the impact of DAMA bias coefficients of linear model (see Section (2.2) and LM perplexity. Furthermore, we evaluate the modified model on a set diverse downstream tasks described in Section 2 In the choice of tasks, we focused both gender bias (WinoBias, StereoSet) and language understanding evaluation (ARC-C, ARC-E, OBQA. MMLU).
Baselines We compare the method with a similar model editing method MEMIT (Meng et al., 2023) and a parameter-efficient fine-tuning via LoRA (Hu al., (022). In both baselines, optimize by the objective of predicting a randomly sampled pronoun when presented with biased prompt.
Choice of Layers and Dimensionality We analyze how the results vary depending on the number of layers selected for debiasing Due to the iterative character of intervention, we a always start editing the fixed layer (22 LLaMA 7B) and gradually add subsequent layers. Further, we check the effect of the number of projection dimensions (d,) in the power sequence from 32 to 1024.
Scaling Lastly, we examine the algorithm's 's performance for larger scales of LLaMA model: 13B, 30B, and 65B.
4.5 RESULTS
Effectivness DAMA effectively decreases the gender bias of the model while preserving its per- formance other tasks, as seen in [a Our algorithm effectively decreased the bias manifested in language generation for a set of unseen professions 6
Morover, DAMA significantly mitigates bias in StereoSet and WinoBias, In the latter task, general accuracy decreases, presumably due to the weakening of the stereotypical cue contributing to correct predictions in numerous test examples.
Our observations confirm that MLP layers contain stereotypical correlations responsible for multiple manifestations of bias. Furthermore, we observe Table 2 that the algorithm causes slight dete- rioration in general language modeling measured by perplexity on Wikipedia texts. It has a minor reflection in performance for downstream tasks. The altered model achieves slightly lower score, yet differences are statistically significant only for one task (ARC-E). Therefore, we can conclude that DAMA does not impact the model's 's ability in question-answering tasks.
'In Table also show examples of next token probabilities in the original and debiased model.

---

Published as a conference paper at ICLR 2024
30
150
100 aa
100
20
perplexity
50
128
256
512
1024
10
Dimenssoos
Layess
(a) Number of layers fixed at
(b) Dimensionality fixed at 256
Figure 3: The effect of applying DAMA to LLaMA 7B model performance and bais in language modeling. We measured bias on gendered prompts (Section by linear coefficients: Og and coefficient, the causal language modeling capabilities measured by perplexity. Stars mark the performance of the model picked for further evaluation. The dashed line corresponds to the scores of the original LLaMA 7B model.
Baselines In contrast DAMA, MEMIT has minor effect on bias measures. We think is be- cause it is aimed to alter information specific to key-value pairs selected for intervention. Therefore, the intervention performed on the training set of professions does not generalize to unseen profes- sions or other types of gender bias. LoRA manifests stronger debiasing properties, coming close to the results of DAMA multiple bias metrics, performs better in StereoSet 88 and ACAT. Nevertheless, fine-tuning significantly deteriorates perplexity and the performance in language un- derstanding tasks.
Choice Layers and Dimensionality In Figure R we observe that the choice of the number of layers for debiasing and the dimensionality of projection affect both parameters. Expanding the depth (number of layers) and width (dimensions) the intervention increases the insensitivity of debiasing, i.e., decreases a, and b coefficients and negatively impacts perplexity. Interestingly, observe a negative impact on both measured aspects when applying DAMA on the two last layers of the models. As noted in Section 3.I the MLPs those layers tend to counter bias in the original model.
Scaling We performed coarse hyperparameter search for sensitive parameters of DAMA: number of layers and dimensionalities of the projections. Our analysis showed that the algorithm should be applied to the mid-top layers, starting from the 65th percentile to the 93rd percentile of layers ordered from input to output (the exact values are presented Table
We have achieved a notable reduction in bias scores for all models. Noticeably, although we do observe the shared pattern for the bias metrics across different model sizes, the improvements brought by DAMA are consistent. Moreover, the perplexity and downstream performance of the original models do deteriorate and even slightly improve for some settings.
5 DISCUSSION
Our approach is connected to previous methodologies in model editing Meng et al. (2022b) and bias mitigation (Ravfogel et al. (2022). The important contribution our work is the introduction of bias evaluation schema directly in language generation. To answer our first question, show that all LLaMA models are biased in this aspect.
Using the evaluation scheme closely connected to the model's pre-training task had two fundamental benefits. Firstly, allowed us to perform a causal analysis of model components. The analysis allowed us to answer second research question. We identified mid-upper MLP layers the most apparent mediator of gender bias in the model. Secondly, we could perform debiasing adaptation directly the model's weights without using a proxy task (Ravfogel al. 2022) or fine-tuning
8

---

Published as a conference paper at ICLR 2024
limited data that often deteriorates the model's general performance (Gira et al. 2022), Answering the third question, we succeeded in significantly reducing bias with a minor impact on general performance.
The proposed algorithm generalizes the applicability of model-editing (Meng et al. 2022a b Mitchell et al. 2022 De Cao et al.) 2021) to the case of modifying general dataset artifacts in- stead of the information specific to particular examples. Although we focused on gender bias, the method can be easily generalized other types of bias unwanted correlations. Additionally, is applicable not only to LLaMA but to a broad family of transformer-based causal language models.
Future Work We plan to improve the method of finding projection matrices, possibly using convex search (Ravfogel al,, (2022) or analytically derived seeudo-rojections (Belrose et al.) 2023). We aim to investigate further the ranges of layers and dimensions that convey bias to apply DAMA on other model types effectively. Lastly, we consider further investigating bias in other languages, both in multilingual LM and machine translation settings. We particularly interested in how our approach can be generalized for morphologically rich languages with more ubiquitous gender marking than English (Zmigrod et al. 2019).
6 RELATED WORK
Measuring bias in language model Gender bias in language models has multiple manifestations quantified by various metrics, which often show low mutual correlation (Delobelle al., [202]] van der Wal et al. 2023). One common approach to operationalize bias is to compare the probability assigned by model to sentences conveying neutral and stereotypical information, e-g- SeteroSet (Nadeem et al. 2021). CrowS-Paid Shirr- (Nangia et al. 2020).. Probability-bas -baed methods were criticized
for being sensitive to the annotation choices (Blodgett et al. 2021 and are hard to apply to auto- regressive models such LLaMA.
Another popular method to estimate gender bias is based on the coreference task, where personal pronouns should be assigned to the correct antecedent in Winograd scheme (Levesque al. 20111), e.g. WinoBias (Zhao et al.] 2018). Winogender (Rudinger et al.] 2018). The task is complicated by including potential antecedents, of which is stereotypically associated with specific gender. The analysis of such examples shows that models struggle with solving non-stereotypical links.
Debiasing methods Similarly to the number of bias metrics, researchers proposed various debi- asing methods (Stanczak & Augenstein 2021; Savoldi et al. (021). The common observation is that models learn the biases from training data (Navigli et al. 2023). Therefore, one approach is to curate the model's training corpus expose it to gender-balanced data in fine-tuning step (Lu al.) 2020b Ranaldi et al. 2023). Alternatively, the model can be fine-tuned on dataset balanced number of examples for each gender (Guo et al. 2022 Zmigrod et al. 2019).
Another set of approaches is to apply targeted changes to the model's parameters. Lauscher et al. (2021); Gira et al. (2022); Xie & Lukasiewicz (2023) fine-tune specific parts of the models most prone to convey biases. Alternative approaches include a null-space projection of latent states (Rav- Fogel et al., 2022), causal intervention (Vig al. 2020). model adapters (Fu al. 2002). DAMA belongs to this category of methods, merging aspects of causal intervention, model editing, and
signal projection techniques.
7 CONCLUSION
We introduced Debiasing Algorithm through Model Adaptation based on guarding stereotypical gender signals and model editing. DAMA is performed on specific modules prone to convey gender bias, as shown by causal tracing. Our novel method effectively reduces gender bias in LLaMA models in three diagnostic tests: generation, coreference (WinoBias). and stereotypical sentence likelihood (StereoSet). The method does not change the model's architecture, parameter count, or inference cost. We have also shown that the model's performance in language modeling and a diverse set of downstream tasks is almost unaffected.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
We acknowledge the contribution of Paul Mouret, who immensely helped us in the implementation and evaluation of LoRA bascline. We also thank him, Jana Straková, Ondřej Dušek, Martin Popel, and anonymous ICLR reviewers for their valuable comments on previous versions of this work. We have been supported by grant 23-06912S of the Czech Science Foundation. We have been using lan- guage resources and tools developed, stored, and distributed by the LINDAT/CLARIAH-CA project of the Ministry of Education, Youth and Sports of the Czech Republic (project LM2018101).
REFERENCES
Nora Belrose, David Schneider-Joseph, Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman. LEACE: Perfect linear concept erasure closed form. In Thirty-seventh Confer- ence on Neural Information Processing Systems, 2023. URL https: ftttps://cpenrevei/ /ppeeeeveeeeeeet................
Eorum?id=awIpxpppr
Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna M. Wallach. Language (technol- ogy) is Power: A Critical Survey of "bias" in NLP. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Associ- ation for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pp. 5454-5476. As- sociation for Computational Linguistics, 2020. doi: 10.18653/v 1211855/v122020.acl 1/2020.acl-main. 485. URL 1ttg:/010.1665//1/2222222 ama- 485
Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, and Hanna Wallach. Stereotyping Norwegian salmon: An inventory of pitfalls fairness benchmark datasets. In Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1004-1015, On- line. August 2021. Association for Computational Linguistics. doi: 10. 10.188533v1/2021.11 1acl222...cllooggg.. 81.
httpss:///laaaaahooooooooooo ogy- wwwp//gg...../....//// 2021 aacl-lon..og........
Tolga Bolukbasi, Kai-Wei Chang, James Y. Zou, Venkatesh Saligrama, and Adam Tauman Kalai. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Ad- vances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 4349-4357, 2016
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018.
Nicola De Cao. Wilker Aziz, and Ivan Titov. Editing factual knowledge in language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pp. 6491-6506, Online and Punta Cana, Dominican Republic, November 2021. Associa- tion for Computational Linguistics. doi: 10. N4.18633/1/2221 0/555//1/2221.emml-maaia mai.2-
URL https:
/aaclanthology.. org/2021. emnlp 1p main. 522
Pieter Delobelle, Ewoenam Tokpo, Toon Calders, and Bettina Berendt. Measuring fairness with biased rulers: A comparative study on bias metrics for pre-trained language models. In Proceed- ings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pp. 1693-1706, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10. 18.13// 1/2022 main. 122. URL
wwg. waw.en222nnnan2...ooa..a..aaaaa. naacl ma in. 122
Chin-Lun Fu, Zih-Ching Chen, Yun-Ru Lee, and Hung-yi Lee. AdapterBias: Parameter-efficient Token-dependent Representation Shift for Adapters in NLP Tasks. In Marine Carpuat, Marie- Catherine de Marneffe, and Iván Vladimir Meza Ruíz (eds.), Findings of the Association for LinggitttioiiLLiiiiiiiiiiiiii NAACL 2022, Seattle, WA, United States, July 10-15, 2022. pp. 2608- Computational Linguistics:
2621. Association for Computational Linguistics, 2022, doi: 10. 111853/v1/2222 findings- 199. URL htt.::tttts/dtto /doi. http//t...r.rrg//// 186118663340 /w155/2////22..222 11t66///1//2222...n.nnnn--aeaea0
wwwddiinnnnnnaaancee nnainngg-naaal...m 199
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer Forward Layers Are Key- Value Memories. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen- tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language
10

---

Published as a conference paper at ICLR 2024
Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pp. 5484-5495. Association for Computational Linguistics, 2021. doi: 10.18653/w1/2021. emnlp-maiee main 446. URL (RLLttts:///oi.or wtt.://o......11311///////.
emnlp-maint 446
Michael Gira, Ruisu Zhang, and Kangwook Lee. Debiasing Pre-trained Language Models via Effi- cient Fine-tuning. In Bharathi Raja Chakravarthi, B. Bharathi, John P. McCrae, Manel Zarrouk, Kalika Bali, and Paul Buitelaar (eds.), Proceedings of the Second Workshop on Language Tech- nology for Equality, Diversity and Inclusion, LTTEDI 2022, Dublin, Ireland, May 27, 2022, pp- 59-69, Association for Computational Linguistics, 2022. doi: (0.188655/112222..teed 1.8. URL.
https://doi org.orgg//ooo wwt.r///..1.633...111
A.S. Goldherger, W.A. Shenhart, and S.S. Wilks. Econometric Theory. WILEY SERIES in PROB-
ABILITY and STATISTICS:: APPLIED PROBABILITY and STATIST ICS SECTION Series. J.
Wiley, 1964. ISBN 978-0-471-3110144
BtLhttps:: battps://books: woopooooooooooee.oooooooooooooee wwt.coom///oo./m haonm/booksiic
K2q5AAAAIAAJ
Yue Guo. Yi Yang, and Ahmed Abbasi. Auto- Debias: Debiasing Masked Language Models with Automated Biased Prompts. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (cds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pp. 1012-1023, Association for Computational Linguistics, 2022. doi: 10. 10.18653//1/2022..ce www.:02222alllooo.... URL https://dol.
org Orc/110.1855////2222..ccoo
72
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. 2021.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Con- ference on Learning Representations, 2022. URL https::/
net forum?
nd=nZevKeeFYf
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http:
// //arxiv. org. abs 1412.444 6980
Anne Lauscher, Tobias Lucken, and Goran Glavaš. Sustainable modular debiasing of language mod- cls. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 4782-4797, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 10.18633/v1/221 http:////22...nnnneeeeeeeee findings-emnl..411... URL httpp:///cclaathooooy..o///////p
Findings-emnip 411
Hector Levesque, Ernest Davis, and Leora Morgenstern. The Winograd Schema Challenge. 2011.
Tomasz Limisiewicz and David Mareček. Don't forget about pronouns: Removing gender bias in language models without losing factual gender information. In Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP), pp. 17-29, Seattle, Washington, July 2022. Association for Computational Linguistics. doi: 10. 006663//1/2222.geblppll URL org. gwwnle-i......e
https:
Kaiji Lu, Piotr Mardziel, Fangjing Wu, Preetam Amancharla, and Anupam Datta. Gender Bias in Neural Natural Language Processing, pp. 189-202. Springer International Publishing, Cham, 2020a. ISBN 978-3-030--60077-6 doi: 10. (0011007/978-3000--607777 0tg/10.1007/40
https://doo
030 62076-077220 12207766_4
Kaiji Lu, Piotr Mardziel, Fangjing Wu, Preetam Amancharla, and Anupam Datta. Gender Bias in Neural Natural Language Processing. In Vivek Nigam, Tajana Ban Kirigin, Carolyn L. Talcott, Joshua D. Guttman, Stepan L. Kuznetsov, Boon Thau Loo, and Mitsuhiro Okada (eds.). Logic, Language, and Security Essays Dedicated to Andre Scedrov on the Occasion of His 65th Birthday, volume 12300 of Lecture Notes in Computer Science pp. 189-202. Springer, 2020b. doi: 10.1007/978-3 3-17788----30000 100-0000--777 62077-6' 14. URL https: /ooi. orq/10. 1007/978-3-030-2277--
11

---

Published as a conference paper at ICLR 2024
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. Advances in Neural Information Processing Systems, 36, 2022a.
Kevin Meng, David
Bau,
Alex Andonian, and Yonatan Belinkov. Lo- Factual Associations in GPT. httpe//ieeeeaaee/22////aaaaa Abst
cating and Editing
In NeurIPS, 2022b.
URL
. 6f1 711c456bs223777886663333333 -os oottaaaaeeetattaaaaoooooeeeeeeee
Kevin Meng, Arnab Sen Sharma, Alex J. Andonian, Yonatan Belinkov, and David Bau. Mass- Editing Memory in a Transformer. In The Eleventh International Conference Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL
bttps://opeenevier net net/pdf wttp:/pdfii-MabcAEIII. Ac-MkbsAE
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor con- duct electricity? new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods Natural Language Processing, pp. 2381-2391, Brussels, Belgium, October-Novente 10.18653/v1/D188 1260. URL | /atttppp//aalaathoooooooo.o wwtp:/oyy...0//---0011000001
2018. Association for Computational Linguistics. doi:
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. Fast model editing at scale. In International Conference on Learning Representations, 2022. URL
https: htwp://wpeeeeeeeeeeeeeeeoe net. / d=0DcZxeN
Moin Nadeem, Anna Bethke, and Siva Reddy. StereoSet: Measuring stereotypical bias in pretrained language models. In Proceedingy of the 59th Annual Meeting of the Association for Computa- tional Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 5356-5371, Online, August 2021. Association for Computational Linguistics. doi: 10. 10118865/11/22211 lcl-
URL https://aclanthoo.ooo org. ore/
2021.ac1-lome 416.
Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pp. 1953-1967. Association for Computational Linguistics, 2020. doi: 10. 10.1666//1/2220.00 www.:00..eemnl-memeee main. 154. URL
https: /ow. www.oo..oo...ooooooooooooo......
www.eem............. in. 154
Roberto Navigli, Simone Conia, and Bjöm Ross. Biases in large language models: Origins, inven- tory, and discussion. J. Data and Information Quality, 1 15(2), jun 2023. ISSN 1936-1955 1955. doi: 10. 10.1145/3597307. URL (tttstt:: //oi. http://..........1110/1
1. 111.:..1111/3333330
Judea Pearl. Direct and indirect effects. In Proceedings of the Seventeenth conference on Uncertainty in artificial intelligence, UAI'01, pp. 411-420, San Francisco, CA, USA, August 2001. Morgan Kaufmann Publishers Inc. ISBN 978-1-55860-800-9
Leonardo Ranaldi, Elena Sofia Ruzzetti, Davide Venditti, Dario Onorati, and Fabio Massimo Zan- zotto. A Trip Towards Fairness: Bias and De-biasing in Large Language Models. CoRR, abs/2305. 13862, 2023. doi: 21.5500/rXXv..20551338333 48550/arXiv. 2305.13862
URL https:://dii org/1C.
Shauli Ravfogel, Michael Twiton, Yoav Goldberg, and Ryan Cotterell. Linear Adversarial Con- cept Erasure. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17- 23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research. pp. 18421, PMLR, 2022. URL ht https: httpss:/ppoceedinggs mlr. press/ .hmll
v1
Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. Gender Bias in Coreference Resolution. In Marilyn A. Walker, Heng Ji, and Amanda Stent (eds.), Proceedings
12

---

Published as a conference paper at ICLR 2024
of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techmologies, NAACL-HLT, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 2 (Short Papers), pp. 8 8-14. Association for Computational Linguisties, 2018. doi: 10. 10.18653/v1/118 1-2200
URL https: /tt10//00000....///////////////0
2020
Beatrice Savoldi, Marco Gaido, Luisa Bentivogli, Matteo Negri, and Marco Turchi. Gender Bias in Machine Translation. Transactions of the Association for Computational Linguistics, 9:845-874, 08 2021. ISSN 2307-387X. doi: 10.1162/tacl..00040) UtL.attps://ci.oorggo tacl a 00401
Karolina Stanczak and Isabelle Augenstein. A Survey on Gender Bias in Natural Language Process- ing. CoRR, abs/21 12.114188,, 2021. URL wwwpthttttss.........aaaaam wwwpaarwwv..oo.gooaaaaasaees 2112. 14168
Gabriel Stanovsky, Noah A. Smith, and Luke Zettlemoyer. Evaluating gender bias in machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Lin- guistics. pp. 1679-1684 1684, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 18.18853/v1/// 1/P19- 1164. URL /aaaaaaaaannhhooooeee
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and Efficient Founda- tion Language Models. CoRR, abs/2302.13971, 2023. doi: 10.48550WrrXx22322213971 URL
2w1550/arxiv.. httpss:/ddo org/oc.....oooooooooo.ooo....... 22555 13971
Oskar der Wal, Dominik Bachmann, Alina Leidinger, Leendert Maanen, Willem Zuidema, and Katrin Schulz. Undesirable biases in nlp: Averting a crisis of measurement, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention All you Need. In Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL. https:: /pttpse//proceeddii ngs. mtw.://rroeednnnseeeee.s..s.... hash/ 3tte/3355e2454444 7dee9 w c4a845a- htmll
fi files//le/eeeeee pappp//pppr///////////// /
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart M. Shieber. Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias. CoRR, abs/2004. 12265, 2020. URL tttt::tts:://aaaa http::::/wwxx...r///oooooooo
12265
Zhongbin Xie and Thomas Lukasiewicz. An Empirical Analysis of Parameter-efficient Methods for Debiasing Pre-trained Language Models. In Anna Rogers, Jordan L. Boyd-Graberr and Naoaki Okazaki (eds.), Proceedings of the ólst Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 15730- 15745, Association for Computational Linguistics, 2023. doi: acl-lonq 8766
URL /BLE::///bb..oo/00.1555////////1
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai- Chang. Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods. Marilyn A. Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-IILT, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 2 (Short Pupers), pp. 15-20. Association for Computational Linguistics, 2018, doi: 10. 10.18653/w1/1188 1-0000 htRlst http:://dooo.o/1/1//
Ran Zmigrod, J. Mielke, Hanna M. Wallach, and Ryan Cotterell. Counterfactual Data Aug- mentation for Mitigating Gender Stereotypes in Languages with Rich Morphology. In Anna Korhonen, David R. Traum, and Lluis Marquez (eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume Long Papers, pp. 1651-1661, Association for Computational Linguistics, 2019. doi: 10.18653/v1/p19- 1161. URL http::ttp: .rg.l.oorr/gooo 10 http://..................5......
13

---

Published as a conference paper at ICLR 2024
A THEORETICAL BACKGROUND
In this section, we provide additional theoretical background with proofs. First, we present theorem that will help prove Theorm] I
Theorem (Ordinary Least Square Problem). Given a n-element key matrix U E' and value matrix V E Rox", we search for mapping matrix W E Roxi minimizing least squares. Specifi- cally, we solve:
W mig WU V1I
This equation is solved by:
W = VU (U
The proof for the theorem can be found, e.g., in Goldherger et al. (9964). Now we are ready to provide proof for Theorem
Proof. Without loss of generality, we consider a case where n = 1, i.e. U and are column vectors. For clarity, we will denote those vectors u IR' and e IR respectively. Therefore, aim solve an equation:
W min Wu vIi such that Wu W
(6)
Note that substitute the Furbenious norm with the Euclidean norm and decompose vector into the sum of two orthogonal vectors.
|| u w 1[ - vl12 = [W - (T - P-p Pull2
We infer that Wu - (I - P)u C from a) Wa C f6; and b) (I-P) P) C as P is projection matrix on C. Moreover, from the properties of linear projection, we have Pv C. We note thus that
Wu - (T 1 P)e 1 Po.
Now, let's get back Pythagoras Theorem saying that for pair of orthogonal vectors d 1 b, have / 1 6 1|2 = || + 112.. We can apply this theorem to b taking Wu - (1 - P)e as
d and Pv b. Thus, write:
|[ (I P)u Poll== [Wu
2) + ||Pu||
(8)
In arg min notation, we can omit the second part of the formula because it doesn't 't depend on W
W = arg min
Ww e/|² = arg min |[Wu (I- P)ul|2
(9)
w
w
Now, we can apply the same steps to all the columns in U U[[ui,.... un] and V = [vi,..., Un). to obtain:
[ui,.....
W arg min IIWU - P1-P
(10)
W
Based Theorm 2 is solved by W (T PUUUTUUT)) We easily obtain this result by substituting V by [T P)V the theorem.
Lastly, it can be shown that for any vector R' we have Wr 1 C from the fact that applying P projection to Wx x always produces null vector:
PW: P(T
(11)
14

---

Published as a conference paper at ICLR 2024
Factual
Stereotypical
Intercept
R
First sulijett tokeu
0.2
0.1
Miidle subject Lokete
0.25
Laes suljeett lokeu
0.0
0.00
First subsequent
Further tolocns
Last tokem
-0.1
(a) Attention
Factual
Stereotypical
Intercept
R2
First subject toloen
0.1
Midddle subject
0.2
Last subject token
First euleisqeet token
Purther
-0.2
Last tolen
-0.1
10
(b) Layer
Figure 4: LLaMA 7B. Gender factual and stereotypical coefficients for linear regression indirect effects of the model yrg. The indirect effect is calculated by reintroducing "clean representation" to the output of specific components (attention or whole layer) and token position.
Factual
Stereotypical
Intercept
H2
subjectt
Middle subject tokous
0.2
0.25
Lass subject token
First subsequent tolom
-0.25
Last, tolom
-0.1
20 d5
16
(a) MLP
Factual
Stereotypical
Intercept
R
First subject token
0.1
Middle subjoct tolorns
0.25
suljortt
0.0
0.00
First subsergnent tolom
Further tokone
0.2
0.25
10.15 20 30:55
20.3300300000 30335
10 15
0025 30.35
10
15:00 36.30.5
(b) Attention
Factual
Stereotypical
Intercept
R²
subjoott
0.1
Middle subject toloss
0.25
Lass subjectt token
0.0
0.0
0.00
First subsequent
Further tokeus
Last token
-0.1
(c) Layer
Figure LLaMA 13B
15

---

Published as a conference paper at ICLR 2024
Factual
Stereotypical
Intercept
R
First subject
022
Middle subject
0.2
Last sibject tolers
0.0
-.0
0.0
First subsoegentt
Further tokess
Last
0.2
0036
1421
(a) MLP
Factual
Stercotypical
Intercept
Re
First suljjett
0.1
Middlle vulject tokees
0.2
Last subject tolets
0.0
1..0
0..0
Fast subsesgont tolets
Further tokess
Last tolon
1935
(b) Attention
Factual
Stereotypical
Intercept
R
Pirst subjectt tolen
0.1
Middlle subject tokess
0.2
Last subject
0.00
Finst subseqqoutt tolen
Further tokess
Last tolon
(c) Layer
Figure 6: LLaMA 30B
Factual
Stereotypical
Intercept
R2
First
0.2
Middlle oubjoct
Lost subject
0.0
0.00
0.00)
Flust subsequent
Further tokens
0.2
0.25
East tokmn
(a) MLP
Factual
Stereotypical
Intercept
R²
Finst subject tolen
Middlle subject tokrea
01.2
Last subject solen.
0.0
Fine subseggent tolen
Further tokres
Last tokens
0..1
10
00:20
30-00 70
(b) Attention
Factual
Stereotypical
Intercept
R
Finst subject solen
0.2
0.1
Middlee subject tokres
0.2
Last subject solen
0.0
First subanggont tolens
Farther tokess
0.2
Tast tolom
20-30
10:00:10
10:00
20.00
(c) Layer
Figure 7: LLaMA 65B
16

---

Published as a conference paper at ICLR 2024
@2
@3
@4
@
Prompt
DAMA
Softmax Probability
he
she
the
26%
13%
11%
8%
8%
The lifeguard laughed because
she
the
he
it
I
10%
10%
9%
9%
8%
she
the
it
he
39%
9%
8%
6%
4%
The nurse laughed because
the
it
he
she
11%
9%
7%
5%
5%
he 54%
of 12%
his
the
it 5%
8%
6%
The boy stayed up because
of the 18% 13%
he
it
his
9%
8%
7%
The nurse notified the patient that his shift would be ending in an hour. "His" refers to the
patient 41%
nur
pron 3%
subject 2%
person 2%
25%
patient 56%
pron 2%
person 1%
hospital %
23%
The mechanic greets with the receptionist because he was in good mood. "He" refers to the
mechan 51%
reception 10%
person 4%
gre 2%
customer 2%
mechan 20%
reception 19%
person 7%
gre 3%
customer
2%
Table 3: The most probable tokens predicted by the model given stereotypical prompts. We compare LLaMA 7B with and without DAMA intervention. The prompts are based on test examples proposed by Lu et al. (2020b) and Zhao et al. (2018) (WinoBias).
B SUPLEMENTARY RESULTS
B.1 CAUSAL TRACING
The Figures 4 5 6, and I0 present causal tracing results for other types of components than MLP: attention and whole layers, as well as larger ILaMA models. For other components, the high indirect effects distributed primarily reflect bias from the MLPs.
extensively across both token positions and layers, indicating that they
For larger models, observe analogous patterns shifted according to the total layer count. Overall, gender bias is most prominent in MLPs located in layers up to the 15th and ranging from the 65th to 93rd percentile of the layers ordered from the input the output.
B.2 DISTRIBUTION OF PREDICTIONS LANGUAGE GENERATION
In Table 3 we present comparison of the softmax probabilities associated with the most likely tokens predicted by the model before and after the DAMA intervention. Notably, we notice that following model adaptation. there is more balanced distribution of pronouns, with male and female pronouns frequently changing positions in the ordering. However, when comes to the WinoBias coreference prompts, observe varied degree of success in the effectiveness of the intervention.
B.3 HYPERPARAMETER CHOICE DAMA
Table presents the width (dimensionality of projection) and depth (number of layers) chosen in LLaMA models of all sizes. The choice of layer numbers matches the observations from causal tracing. We further backed the parameter selection by limited parameter search, which results are presented in Figures and TO
17

---

Published as a conference paper at ICLR 2024
Model size
# layers
layers adapted
# dimensions
projected dimensions
Llama 7B
32
21 29
4096
256
Llama 13B
40
26 36
5120
512
Llama 30B
60
39-55 55
6656
1024
Llama 65B
80
52
8192
2048
Table 4: Number of layers and latent dimensions of LLaMA models compared with the number of DAMA adapted layers and the projected dimension.
25
60
100
100-6 40
20
perplexity
15
20
10
20
128
256
512
1024
14
Dimensions
Layers
(a) Number of layers fixed at
(b) Dimensionality fixed 512
Figure 8: Change in results for different layer and dimensionality configurations of DAMA for
LLaMA 13B model.
25
50
100
100
40
20
perplexcity
30
15
20
128
256
512
1024
16
18
Dimensions
Layers
(a) Number of layers fixed at 17
(b) Dimensionality fixed at 1024
Figure 9: Change in results for different layer and dimensionality configurations of DAMA for LLaMA 30B model.
25
100
100 b
perplexity
20
60
40
20
10
21
22
Dimensions
Layers
(a) Number of layers fixed at 20
(b) Dimensionality fixed at 2048
Figure 10: Change in results for different layer and dimensionality configurations of DAMA for LLaMA 65B model.
18

---

Published as a conference paper at ICLR 2024
0.10
0.06
0.02
0.00
desird
sad
sayed up beons
wante hat
fral brome:
was prempeeeee bcconas
wast hons becoiss
wairpennd Car
wished Cat
yollal becmme
yelled tar
ote
cited
drow
haghal
Figure 11: Gender bias for the prompts proposed by Lu al. (2020a) measured by p(he) p(she) averaged over all professions.
Stereo SHE_
AVG HE
0.3
0.3
0.2
0.2
0.1
0.0
0.0
10
14
2
(a) Stereotypically female professions
(b) Stereotypically male professions
Figure 12: Probability of the pronouns she (red), he (blue), and they (green) and their dependence on the multiplicative constant of the noise level. Averages and standard deviations over the male and female professions.
C TECHNICAL DETAILS
C.1 LANGUGE GENERATION BIAS EVALUATION DATASET
Prompt templates selection. Lu et al. (2020a) proposed several prompt templates for testing gen- der bias of professions. We filtered out some of them because we observed some verbs included in the templates are highly biased toward one of the genders. In Figure II observe the average probability differences between the prediction of he and the prediction of she Some verbs such as "yelled", "was promoted", "was fired", "slept" are highly biased towards males. On the other hand, verbs such as "wanted", "cried", "desired", or "stayed up" are only very little biased towards males. Given the general skewness of the model towards predicting male pronouns,
say these
19

---

Published as a conference paper at ICLR 2024
verbs are female-related. For the evaluation, chose the templates whose averaged difference be- tween the prediction he and she is lower than 0.8%. Thus we are excluding the prompts "slept because", "was fired because", "was promoted because", "yelled that", and "yelled because".
Test train split. For evaluation, we select a test set consisting of all professions with semantically defined gender (where l 0.25). We also include 20% of the other professions to be able to evaluate the impact of both semantic and stereotypical gender.
The remainder of the professions are assigned the train Noticeably, the trainset doesn't contain profession with semantically defined gender. It is deliberate choice because we want to preserve factual gender signals the model debiased using training data. For both splits, we use all selected prompt templates.
C.2 CORRUPTING REPRESENTATION
In step (2) of the causal tracing, we need to obfuscate the tokens in the profession's words. We use the same methodology in Meng et al. (2022a). We add random gaussian noise N/ v) to the token embeddings h/ := h. + for each token i the profesion word. The parameter was set be three times larger than the empirical standard deviation of the embeddings of professions. As shown in Figure [2] the multiplicative constant lower than three would not fully remove the stereotypical bias from the tokens. Higher values could remove too much information, e.g., the information that the subject of the prompt refers to person.
C.3 OPTIMIZING VALUE REPRESENTATION
To find the value representation, we minimize the loss given by Equation 2 We run gradient opti- mization for 20 steps with Adam scheduler (Kingma & Ba 2015) and learning rate: Ir 0.5. We picked the following regularization constants: A1 = 0.0625 and X2 = 0.2.
C.4 BASELINE IMPLEMENTATION
We implement two baselines for adapting LLaMA 7B: MEMIT (Meng al. 2023) and LoRA (Hu
et al. 2022). Both methods were applied to the output projections of MLPs in layers selected by causa tracing. We optimize the parameters with the objective of predicting randomly sampled pronoun when presented with biased prompt. The data and training hyperparameters are the same as in DAMA, not stated otherwise.
LoRA is parameter-efficien fine-tuning technique. It adapts weight by adding an update matrix, which is a product of two trainable matrices dW B A. For efficiency, matrices B and have lower dimensionality than W E ooxs i.c B caxr and A gxxi, In our implementation, used factor and learning rate = 0.0001.
20


Reviews:
[{"review_id": "j10oFcIo9u", "reviewer": "Reviewer_S9AC", "summary": "The paper investigates gender bias in the Llama model. First, an existing causal tracing method is adapted to work for measuring gender bias in different components of the model. Then, a method for modifying the model weights is described for reducing the gender bias. Evaluation is performed both on gender bias benchmarks and downstream task datasets.", "questions": "Please see above", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The methods are interesting. Evaluation is performed on a number of different datasets and using different measures.\n\nThe method for updating the model is also interesting. Particularly as it manages to not even increase the number of parameters in the model.", "weaknesses": "A weakness is only focusing on Llama, as it is unclear how much these findings generalise to other models.\n\nThe causal tracing method is interesting. It is currently unclear how much this is based on previous work and what exactly is the novel contribution. Please clarify this.\n\nIt is unclear why a linear model is fitted across the two gender scores in order to investigate the extent of bias. If the two scores are correlated (which they likely are) then the coefficients of such a linear model might not give accurate indications of the different biases. Measuring correlation with the gender scores seems like a much more straightforward method.\n\nAgain, for the method of updating weights, please clarify the difference of the proposed method to previous work, including Rafvogel et al 2022.\n\nA major selling point of the method seems to be that the proposed DAMA still achieves good performance on downstream tasks. However, the baseline MEMIT actually seems to get better results on most of this metrics. This finding somewhat weakens this claim and should be addressed in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4tI6aY54xb", "reviewer": "Reviewer_2cbg", "summary": "The paper demonstrated unwanted gender bias still exists in the popular LLaMA family of models. To combat these prevalent biases, the paper located components of the model that caused such biases and edited these weights to mitigate the effect of gender biases on downstream tasks. The method, DAMA, is an improvement over the previous method both in terms of reduction rate and maintaining original performances.", "questions": "Styling:\n1. You are missing a right parenthesis on equation (2) for the KL divergence.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Detailed and quantitative measurement of the effect of factual cues and stereotypical cues on model generations. Specifically in the result section, the authors measured the effect of factual cues and stereotypical cues based on layer number and token positions.\n2. The gender bias reduction method via model weight editing has sound theoretical backup.\n3. DAMA effectively reduces the bias rate without hurting too much of the downstream performances.", "weaknesses": "1. Seems like there is still quite some room for improvements in the debiasing methods on all three evaluations proposed in the paper. For WinoBias and SteroSet gender, I interpret the results as being still pretty gender biased even after applying DAMA. And for Bias in LM, for larger models, the a_s and a_f are still far from complete removal. It would help greatly to provide other gender bias removal methods as baselines to better assess how well DAMA did.\n2. I think the study on the effects of downstream is great. However, its negative effect on reasoning tasks such as ARC can be concerning for a practical user. Is there any hypothesis on why it doesn't fare well with reasoning type of tasks?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I8sfRESaRu", "reviewer": "Reviewer_FcGW", "summary": "This work studies the the issues of biases and stereotypes in the large language models, and propose methods to (1) identify the specific components of LLMs that is responsible for these issues; and (2) resolve these issues with a linear projection on the feed-forward components of LLMs.", "questions": "Please refer to the clarification questions mentioned in the previous section.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "- The research question is interesting and critical to LLM research and application\n- The high-level ideas of the proposed techniques (on both identification and debiasing) make sense.", "weaknesses": "I had a difficult time understanding this paper because of the following writing issues. \n\n- First of all, certain concepts are mentioned in the paper. However, they are either inconsistent or not explicitly defined. For example,\n    - Stereotypical keys vs. stereotyped keys\n    - Gender value vs. gendered value\n    - Also, what is grammatical gender?\n- About Equation 2, I have several questions\n    - What is z? Why do we need a $\\ell_2$ item on it?\n    - Missing a “)” somewhere in equation 2\n    - What is $P(o’|X’)$? The first two items in equations look similar to variational inference. In that case, $P(o’|X’)$ should be something similar to a prior distribution. However, I could not find its definition.\n- What is gender values metrics?  What is the relation between these V metrics and U?\n- Why $P$ in equation 5 is defined in that way, and how should we use it? I think it was explained in the paragraph right after equation 5, but I am not sure I understand it.\n- Figure 3 (b) seems to indicate that to reduce the bias, the cost is to get a much worse language model (based on the perplexity score), which is not exactly claimed in the paper.\n\nIn addition to my clarification question, one concern is about the technical novelty specifically the causal tracing idea is from prior work, and the linear project seems to be a straightforward idea in debiasing literature, so I am wondering what is the technical novelty of this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)