Conceptor-Aided Debiasing of Large Language Models
Authors: Li S. Yifei
Lyle Ungar
João Sedoc
Venue: EMNLP 2023 Main (2023)
Keywords: Natural Language Processing
Large Language Model
Fairness

Abstract: Pre-trained large language models (LLMs) reflect the inherent social biases of their training corpus. Many methods have been proposed to mitigate this issue, but they often fail to debias or they sacrifice model accuracy.  We use *conceptors*--a soft projection method--to identify and remove the bias subspace in LLMs such as BERT and GPT. We propose two methods of applying conceptors (1) bias subspace projection by post-processing by the conceptor NOT operation; and (2) a new architecture, conceptor-intervened BERT (CI-BERT), which explicitly incorporates the conceptor projection into all layers during training. We find that conceptor post-processing achieves state-of-the-art (SoTA) debiasing results while maintaining LLMs' performance on the GLUE benchmark. Further, it is robust in various scenarios and can mitigate intersectional bias efficiently by its AND operation on the existing bias subspaces. Although CI-BERT's training takes all layers' bias into account and can beat its post-processing counterpart in bias mitigation, CI-BERT reduces the language model accuracy. We also show the importance of carefully constructing the bias subspace. The best results are obtained by removing outliers from the list of biased words, combining them (via the OR operation), and computing their embeddings using the sentences from a cleaner corpus.

Conceptor-Aided Debiasing of Large Language Models
Li S. Yifei", Lyle Ungar", João Sedoc
Uniiersity of Pennsylvania, New York University (liyifei, ungar)@upenn. edu, jsedoc@sterm.nyu.ed
Abstract
Corpora Ruddi)
Pre-trained large language models (LLMs) re- flect the inherent social biases of their training corpus. Many methods have been proposed to mitigate this issue, but they often fail de- bias they sacrifice model accuracy. We use conceptors-a soft projection method-to iden- tify and remove bias subspace in LLMs such as BERT and GPT. We propose two meth- ods of applying conceptors (1) bias subspace projection by post-processing by the concep- for NOT operation; and (2) new architecture, conceptor-intervenedd intervened BERT (CI-BERT), which explicitly incorporates the conceptor projection into all layers during training. We find that con- ceptor post-processing achieves state-of-the- art (SoTA) debiasing results while maintaining LLMs' performance on the GL.UE benchmark. Further, it is robust in various scenarios and can mitigate intersectional bias efficiently by its AND operation on the existing bias sub- spaces. Although CI-BERT's training takes all layers' bias into account and can beat its post- processing counterpart bias mitigation, CI- BERT reduces the language model accuracy. We also show the importance of carefully con- structing the bias subspace. are obtained by removing outliers from the list of biased words, combining them (via the OR operation), and computing their embeddings using the sentences from cleaner corpus.'
Gentencaa
Outher 0.2..
Stack the wittle
(pronouns, exlended, propemouns,
entowliing
Irout comappoor 14tapase
Concaptar-Aided Deblasing
Post Peot-Procossing (P.F)
Continued Training via Conceplor Intervention
Negation
Method
Evaluate
Figure 1: The pipeline of the conceptor-aided debias- ing paradigm. We first use different settings (wordlists with outlier filter and corpora) to generate the best bias subspace (conceptor matrix), then apply them to two conceptor-aided debiasing methods and measure the debiasing performance by two evaluation metrics. The experiment conducted two LLMs: BERT and GPT.
et al., 2019). Therefore, many researchers have de- veloped debiasing techniques to improve the social fairness of NLP. However, such debiasing often fails to debias effectively and reduces language model performance in downstream tasks (Meade et al., 2022). Furthermore, most debiasing methods neither follow Bommasani et al. (2020)'s sugges- tion to reduce bias in all layers nor tackle intersec- tional bias in an efficient way (Lalor et al., 2022).
best results
1 Introduction
In this paper, we challenge Karve et al. (2019)'s conclusion that conceptor negation fails to debias BERT stably. Instead, we are the first ones to empir- ically find that as a soft shrinkage of the principal components of the subspace defined by the list of biased words (Liu et al., 2018) 8), conceptors is pow- erful tool to debias LLMs such as BERT and GPT using either post-processing or continued-training. In this process, demonstrate the effect on debi- asing performance of choosing different corpora, subspace removal methods, and criteria for select- ing the list of bias attribute words that are used to
LLMs such as BERT (Devlin et al., 2019) and GPT (Radford et al., 2019; Brown et al., 2020) are extremely successful in most natural language processing (NLP) tasks. However, since they are trained on texts written by humans, the social bias is inherited and represented in the parameters of LLMs (Bolukbasi et al., 2016; Caliskan et al., 2022). For example, gender bias has been found in contextualized embeddings (May et al., 2019; Zhao
Cooee link: https: / thub. com/realllyifei/ denceptor-debas--111

---

construct the bias subspace. Further, we unprece- dentedly show that the conceptor can tackle varied types biases (e.g. gender, race) intersectionally and efficiently by its unique logical operation.
the contextualized representations of LLMs like BERT and GPT (Bommasani et al., 2020); and Kaneko and Bollegala (2021) show that debias- ing all layers is more effective. Moreover, Lalor et al. (2022) indicates the importance of addressing varied biases in different dimensions. Thus, a new challenge is raised on how to adapt current methods or develop novel paradigms to mitigate the bias in each layer and across multiple social dimensions.
Specifically, the aftribute wordlists at the core of our method, and the methods we build on, are sets of attribute words related to bias. These typi- cally come in opposing pairs (e.g. 'man'/ "prince' / /"princess'). Bolukbasi et al. (2016), Liang al. (2020) and others use the first principal com- ponent (PC) to define the bias subspuce-which can be later subtracted entirely to debias. We similarly construct such subspaces, but use conceptors as the PC
2.2 Debiasing Techniques and Challenges
We collect the mainstream SoTA debiasing meth- ods (Overview: Meade et al. (2022); Xie and Lukasiewicz (2023)), each with typical examples: (1) Bias Subspace Projection (BSP): the clas- sic method of bias subspace subtraction is first capture the bias subspace determined by attribute words in the corpora and then project the bias direc- tion out from the language embeddings. This can be done by post- proscssccssing as either hard projection (Bolukbasi et al., 2016; SENTENCEDEBIAS, Liang et al.. 2020) or soft projection (Karve et al., 2019), Some variants attain similar goal by training a linear classifier (INLP, Ravfogel et al., 2020) fine-tuning LLMs (Kaneko and Bollegala, 2021).
a 'soft' way to remove adjusted by regularized identity map. When gen- crating such wordlists, it may be more representa- tive of bias by removing outliers in the embedding space. Considering the embeddings are contextual- ized, we select the contextualized token-level word embeddiigs using sentences from a specific corpus. Then we stack them to generate a bias subspace in a form of conceptor matrix for the debiasing in the next step. The pipeline is shown in Figure 1.
This work contributes the following:
Employs conceptor negation post-prin post-procsssing to debias LLMs such as BERT and GPT, beat- ing most SoTA while retaining useful seman- ties and robustness in multiple scenarios Explores conceptor-inntervened BERT (CI- BERT)-a novel model architecture that con- tinues training BERT after incorporating con- ceptors within all of BERT's s layers Illustrates how different corpora, bias attribute wordlists, and outlier removal criteria impact debiasing performance
(2) Counterfactual Data Augmentation (CDA): swapping the bi-polar bias attribute words (e.g.
her/him) to rebalance the training dataset and there- fore decrease the gender bias (Webster et al., 2020; Barikeri al., 2021).
(3) Dropout Regularization (DroPOUT): in combination with an additional pre-training, in- creasing the dropout components inside the transformer-baset language models can lead to lower bias (Webster et al., 2020).
Demonstrates conceptor-aides methods can be generalized to different layers of LLMs and various types of biases and can mitigate them intersectionally by its unique logical operation
(4) SELF-DEBIAS: by using specific templates to encourage L.MSs to generate toxic output and then modifying the original output distribution of the model by a decoding algorithm, Schick et al. (2021) makes use of the internal knowledge of language model to debias in post-hoc manner.
2
Related Work
2.1 Bias Manifestation
Further, it is common to combine multiple such methods. For instance, Zhao et al. (2019) and Liang et al. (2020) combine the techniques of data aug- mentation and hard debiasing. However, per the discussion in Meade et al. (2022), the methods of- neither debias as well as they claim (e.g. CDA, DROPOUT, SENTENCEDEBIASS, nor do they main- tain the model's capability for downstream tasks (c.g. CDA, DROPOUT, INLP). Worse, some tech- niques like CDA and DROPOUT increase the bias
Multiple demographic biases are common in so- ciety. Among them, gender bias is the most well- studied in academia, given its omnipresence and bi-polarity (Bolukbasi et al., 2016; May et al., 2019; Kurita et al., 2019). Other social biases (e.g. racial, religious) also widespread in LLMs and at- tracting increasing attention (Nangia et al., 2020; Nadeem et al., 2021; Meade al., 2022).
Such social bias manifests itself in all layers of

---

measured SEAT-a test of language bias which we will describe in Section This dilemma chal- lenges us to develop new methods to further reduce bias while retaining meaningful semantics. Last, the majority of debiasing methods ground the bias by word list; different lists can lead to different debias performance (Antoniak and Mimno, 2021).
where | |e is the Frobenius norm and a= is a scalar nypeerparameter called aperture 3, can be shown that C has a closed-form solution:
-= c=-xx xx
x + 21 77
(2)
where X = [ trix whose i-th column is 24. Intuitively, C' is a soft projection matrix on the linear subspace where the typical components of z; samples lie so that can capture the components that all representa- tions roughly share. Therefore, different from PCA projection which removes the first several princi- pal components (PCs) completely, the conceptors method softly downscales the PCs adjusted by regularized identity map (Figure 2).
is data collection ma-
2.3 Conceptors in NLP
Conceptors-a soft projection method support- ing conceptual abstraction and logical opera- tions (Jaeger, 2014)-has been adapted to NLP do- mains such as debiasing (Liu et al., 2018; Sedoc and Ungar, 2019; Karve et al., 2019), continual learning (Liu et al., 2019a), and semantic infor- mation enrichment (Liu et al., 2019b). Conceptor negation is a soft shrinkage of the PCs of sub- space such as stop words or, in our case, of the target words defining the bias directions (Liu et al.. 20. Therefore has the potential to debias better than hard projection (e.g.. Bolukbasi et al., 2016) while retaining enough semantics. Mathematically, it can capture, conjoin, and negate the bias con- cepts by logical operation, and thus can deal with intersectional bias efficiently.
C2
C3
Figure 2: Geometry of three conceptors in the shape of ellipsoids (Jaeger, 2014).
Conceptors support Boolean operations such as NOT (-), AND (A) and OR (V). For two arbitrary conceptors C1 and C2, we have
Although Karve et al. (2019) showed that debias- ing conceptors successfully debias both static embeddings such Glove, Word2vec, and Fasttext, and contextual embeddings such ELMo (Peters al., 2018 8), they state that the performance in BERT is far less consistent and effective than other word representations. We discover that this is the result of their having selected the wrong set of at- tribute words, which leads to poor bias subspace." Another difference is that the BERT tokens of at- tribute words should be averaged if they contain multiple subwords after tokenization (Liang al., 2020; Kaneko and Bollegala, 2021).
-01 = - C1 A (C -
(3)
r)-
(4)
-C2) ((-
(5)
These logical operations are feasible if C₁ and C2 are created by the sets of equal sizes (Jaeger, 2014), as shown in Figure 3. This reveals the potential for debiasing by combining different conceptors learned from different bias subspaces. This is help- ful both in combining different wordlists for the same bias (e.g. gender) or different wordlists for different protected classes (e.g. gender and race).
3 The Mechanism of Conceptors
Let us take a closer look at the mathematics of con- ceptors: considering a set vectors { >x,)
4 Debiasing Sentence Representations
E N for all i E {1.-.-. .n). conceptor ma- trix C is regularized identity map that minimizes
4.1 Bias Subspace Setting
We explore the impact of different choices of at- tribute wordlists, the corpora used find their em- beddings, and how the wordlists are combined and filtered remove outliers, on the quality of bias
Caill + CCI
(1)
"The default value of a is I; we empirically find that grid- scarching is not helpful for debiasing so keep as default
We fixed the coding issues.

---

three corresponding conceptor matrices to generate what can be viewed as a union of the three bias subspaces (named or).
0.5
Unlike Karve et al. (2019). to study the effects of removing outliers from the wordlists, we first project the LLM's 's embeddings of the words in the wordlist to 2-dimensional UMAP cluster- ing (McInnes al., 2018) space, shown in Fig- ure 4, and then filter the outliers by percentile on their (x, y)-oorddinete. The outliers are defined as the points that fall outside of 1.5 times the inter- range (IR), which is the difference between p-th and (1-p)-th percentile. We iterate p from 0.1 to 1.0 with step size 0.1 to generate different wordlists and then test how well each debiases. Our goals are to detect the negative effect of outliers on debiasing performance and explore which percentile here optimal for debiasing.
-1
OS(C', (9)
AND(C),))
NOT())
Figure Visualizing the boolean operations on two conceptor matrices. The OR (AND) operator leads to a conceptor matrix (in pink color) with the smallest (largest) ellipsoid (He and Jaeger, 2018). In our case, it is then negated by the NOT operator to debias.
subspace, and hence the debiasing (Fig 1). Differ- ent procedures of bias subspace construction yield significantly different debiasing performances.
Corpora We compare three corpora: (1) the Brown Corpus (Francis and Kucera, 1979), a col- lection of text samples of mixed genres; (2) the Stanford Sentiment Treebank (SST; Socher et al., 2013), a polarized dataset of 10,662 movie reviews; and (3) a Reddit Corpus (Liang et al., 2020), a dataset collected from discussion forums about re- lationships, electronics, and politics. The reason is to see how the language formality and topic breadth of texts impact the debiasing, the Brown corpus is formal and contains 15 genres, the Reddit corpus is informal with 3 domains and the SST corpus is informal, has only one domain. They are used to provide embeddings for the attribute words.
10
prapemouns
extended
prenoume
(a) Wordlist
(b) Gender
Figure 2D UMAP BERT Embeddings of Words.
Combining and Filtering Wordlists We com- pare five ways of using three different wordlists to create conceptor bias subspaces.
4.2 Debiasing Methods
We propose and explore two kinds of conceptor- aided debiasing: conceptor post-procet post-processing. and conceptor-intervened continued training. They are abbreviated as P.P. and C.T. respectively in tables.
The three wordlists are gender words originating from different sources: the pronouns wordlist is a set of common terms that are specific to particular genders, such as 'daughter' or 'son'; the extended wordlist, an extension of the former, contains less frequent words such as 'cowgirls' or 'fiancees"; and propernouns wordlist is comprised of proper nouns like 'Tawsha', 'Emylee', and so on.
Conceptor Bias Subspace Construction We construct the conceptor negation matrix -C demonstrated in Algorithm where matrix X is stack of the within-sentence contextualized em- beddings of the words. The words are determined by attribute wordlists and the sentences are from the specified corpus as mentioned in Section 4.1. Note that we do not need the "difference space" of bipolar bias as the conceptor projection matrix is applied to the original space-in this way the con- ceptor method different from the so-called hard- debiasing (Bolukbasi et al., 2016). To ensure con- textualization we remove the less-than-fourr word sentences. Also, following Kaneko and Bollegala (2021)'s idea, if a word crosses multiple sub-tokens,
There are five methods of using these three wordlists to generate a bias subspace. We can use each of them individually (their subspaces are named the same as themselves: pronouns, ex- tended, and propernouns, respectively). We can also combine them in two ways: either by con- catenating them as single list generating a cor- responding subspace (named all); or by running the conceptor OR operation-8 Boolean operation of conceptors described in subsection on the

---

then its contextualized embedding is computed by averaging the contextualized embeddings of its con- stituent sub-tokens, which different than the pre- vious conceptor works.
trained weights) to the new architecture C1- BERT (Type III: CI-BERT x trained weights).
Quantifying Bias
5.1 Sentence Encoder Association Test
Conceptor Negation and Post- Post-Processing Next, we post-process the sentence embeddings t which contain attribute words and target words, by tak- ing the matrix product of -C to subtract the bias subspace, rendering debiased embeddings t", as demonstrated in the last part of Algorithm 1. Each BERT layer manifests different levels of bias (Bom- masani et al., 2020). To maximize the effectiveness of -C, we want -C to be generated from the corre- sponding layer. Therefore, we are the first ones to test the debiasing performance by using different conceptor matrices generated by different layers of the language model and to explore whether concep- tor post-processing generalizes well on each layer of LLMs and on different LLMs (BERT and GPT).
The Sentence Encoder Association Test (SEAT) (May et al., 2019) is extension of the Word Encoder Association Test (WEAT) (Caliskan al., 2017). It can measure the bias at the sentence level in different kinds of bias (Meade et al., 2022).
SEAT uses two types of words: attribute words Wa (e.g. he/she) and larget words Wi (e.g. occupa- tions), which we expect to be gender-neutral. That is, the associations between Wa/w's C Wa and Wy E W. should be no difference in the sentence- template representations of LLMs.
Denote the sentence sets of attribute words as A and A', and of target words as T and T', we have:
<AA,A'.T,T) Σ A(..
Σ ((''A,A) PET
Intersectioanl Debiasing Importantly, not only can conceptors mitigate different types of biases such as gender and race respectively, but it can also conjoin and negate these biased concepts together due to its magical logical operations. It is natural that socictal biases co-exist in such "African Male" rather than "African" and "Male". Therefore, it is efficient that conceptors can tackle them intersectionally by utilizing the previously constructed bias subspaces via its OR operation to construct the new mixed conceptors.
where for each sentence s, have c(s, A'). the difference of the mean cosine similarity of s concerning sentences from between A and A'; as c(8, A. A')
A Σ cos(s, a)
Σ cos(s, a'EA''
The amount of bias is given by the effect size d ((fc(t, 1, A, A'))eT) p (fa(t', A, A))ver)) a (fc(a, TTT) Ja)
where μ and a denote the mean and standard devia- tion, respectively. The smaller the absolute value of is, the less bias has been detected. The one-sided p-value measures the likelihood that a random re- sampling of the sentence set that contains attribute words would generate the observed test statistic.
Conceptor Intervention and Continued Training The varying levels bias across BERT layers sug- gest the possible utility of an alternate approach to mitigate the bias. Accordingly, we construct a new architecture, Conceptor-Interve Intervened BERT (CI- BERT), by placing the corresponding conceptor matrix after each layer of BERT (Figure 5). We then continue training the entire model incor- porate the model weights with the bias negation captured by the conceptors in each layer. Thus we can take the biases in all layers into account so that we can mitigate the layerwise bias simultancously. CI-BERT architecture can be used in three ways. We can load the original pre-trained weights to CI-BERT and directly render the language embed- dings (Type CI-BERT original weights), Alter- natively, we can continue training the model using CI-BERT to get newly trained weights; then we can load these weights back to either the original off-the-shelf BERT architecture (Type II; BERT
5.2 Gender Co-Reference Resolution
As described by Gonen and Goldberg (2019). SEAT can detect only the presence but not the absence of bias. To further understand how the conceptor- aided methods work on debiasing, we adopt an end-task: gender co-reference resolution.
WinoBias (Zhao et al., 2018) provides gender- balanced co-reference tests to evaluate LLMs' neu- trality towards pronouns referring to occupations.
The tests include pro-stereotypical (PRO) scenarios, where gender pronoums match gender-conforming occupations (e.g.. her/nurse), and anti-stereotypical (ANTI) scenarios, where gender pronouns apply disfavored occupations. The bias measured by the average and absolute difference in Fl scores between the PRO and ANTI subsets.

---

Algorithm| SENCEPTOR-DERIASE a conceptor-aided post-process algorithm for debiasing LLMs.
Require: large language model Mo (with parumeters 6), hias attrihute wordlist and corpus 8.
X [-[]
each word w 3: for each sentence E S
4:
if us inside then We 4- the embedding of w inside Mo(s) x we
get contextualized word embedding If stack as a matrix
7:
if end for
9: end for
10: C + XX XXXX + +1)
If construct conceptor bias subspace
// note that different X, yields different C, for arbitrary
11: C + (CT
11-
If cross bias subspaces by AND operator (if intersectional debias)
12: ---
1)
// unite bias subspaces by OR operator (for robust debias)
13: -C' -C-I-C
// make negation conceptor matrix by NOT operator
14: for each new sentence do
15: t" f- - (t)
// debias sentence by projection
16: end for
processing mitigate bias and beat SoTA?
Original BERT
Does embedding conceptors into LLMs via continued training beat post-processing!
Layer
Conceptan-1
Layer
Comceptar-2
What roles can conceptors operators-NOT, OR, AND-play in the debiasing pipeline?
To help comparison, the SoTA debiasing results from Meade et al. (2022) is included in the tables.
Layer
Comceptor-12
Continue Training
6.1 Models
To investigate the generalization of conceptor de- biasing, explored different scales of typical LLM families, which are: BERT- (bert-tiny), BERT (berr-base-uncased)) BERT-L (bert-large- uncased), GPT2 (gpt2), GPT-L (gpt2-large), and GPT-J (gpt-j). We did not test on GPT3 and Chat- GPT since their embedding models (e.g. text- embedding-adea
Figure 5: Conceptor-Intervened Intervened BERT (CI-BERT), Each model's layer takes the matrix product (blue circle) of the conceptor-X X generated from the corresponding layer X. It can be used directly continually trained.
Based on this, de Vassimon Manela et al. (2021) develop two intuitive metrics, skew and stereotype, to better probe model fairness. In formula,
do not support the contextual- ized embedding on token level. However, due to the similar modeling, once have such embedding, conceptor techniques can be transferred.
F1 M
F1 +
F1
F1F
pro
pro
and
anñ
MSkeew
2
M pro
F1M
M nnti
FIF
F1F
pro
anti
MSttecoo
2
6.2 Bias Subspace Construction with Robustness Boosted via OR Operator
where superscripts M and F denote male and fe- male respectively and F1 stands for the Fl-score. It is shown that there is an approximate trade-off between these two biases. The authors argue that the T2 test set of WinoBias is better than the T' test set at revealing bias, as the latter is less ambiguous to LLMs. Therefore, we only report T2 here.
We construct the conceptor bias subspaces upon the different combinations of corpora, wordlist se- lections, and outlier removal.
To evaluate corpora, by testing on the last layer of the BERT, we compare the debiasing result of three different corpora: Brown, SST, and Reddit on SEAT. Table 8 shows that Brown stably provides the best debiasing result even if using different wordlist subspaces. The SST corpus is a close sec- ond, while Reddit by far the worst. The style of the Reddit corpus is most likely least similar to that of the SEAT evaluations.
6 Debiasing Results
This section aims answer these questions:
What is the best setting for bias subspace gen- debiasing?
cration within conceptor-aie
Given the best setting, can the conceptor post-

---

To evaluate alternate methods of constructing the bias wordlist subspace, we use the five sub- spaces described Section 4.1. Among them, the or subspace is the most robust; see Table 9, 10 and 11. Combining the pronouns, extended and propernouns subspaces with or represents the dis- tinct yet union concepts (and hence subspaces) of each of the wordlists, thus both outperforming in- dividual wordlists and outperforming the all sub- space, which simply concatenates all the wordlists, giving a less precise subspace definition.
is consistent in different scales of BERT and GPT models. Note that the settings of BERT-L, GPT2-L and GPT-J are not tuned (i.e. default setting), which means that they can likely reach much lower SEAT scores. Moreover, conceptors can mitigate the bias in almost all scenarios, no matter using which cor- pus, bias subspace, or wordlist threshold (Table 9, 11 and 10); matter which LLMs (Table 1, 15, 17, 16 and ,19) no matter in which layer (Table 12, 13 and 18); and no matter which type of biases (Table 3, Table 21 and 22).
To evaluate wordlist outlier removal, we define the outliers by the UMAP filter as discussed in sec- tion 4.1 and generate different percentages of the words that are used to capture bias. For example, the all subspace has 2071 words within 0.5- 1.0 percentile, 2061 in the 0.4 percentile, 1601 in the 0.3 percentile, 130 in the 0.2 percentile, and 82 in the 0.1 percentile (Table 6). We observe that including fewer words often leads to higher debias- ing performance, presumably due to the removal of outliers. However, extremely small percentile, say 0.1, would harm the effectiveness of debiasing because of the inadequate loss being left (Table 9, 10 and 10). Similar results are obtained if using T-SNE (Van der Maaten and Hinton, 2008).
Model
Fi Male
Female
Bias
Pro
Anti
Steren
Skow
BERT
66.4
58.9
318
17.0
11.2
38.3
Conceptor
69.5
52.8
20.1
22.3
Conceptor C.T.
41.0
39.3
57.6
56.6
4.1
17.0
Table Fl of skew and stereotype biases in WinoBias.
6.4 Intersectional Debias via AND Operator
Table 3 empirically shows that conceptors not only can mitigate the different type of biases, but also can intersect the existing bias subspaces (e.g. gen- der and race) to create a mixed conceptor matrix in an efficient way and to debias gender and race re- spectively. Furthermore, for assessing the intersec- tional debiasing, we employ the I1-15 intersectional bias test introduced by Tan and Celis (2019). They adapt the SEAT to examine the privilege associated with the combination of being African/European American and being male or female. The results demonstrate that such intersected conceptor formed via the AND operator can effectively reduce multi- dimensional bias, lowering the SEAT score from 0.673 to 0.434, while its conceptor counterparts focused solely on linle-dimensional bias can only reduce the score to 0.613 and 0.635 respectively.
In conclusion, the optimal setting for BERT-T is "sst-0.5-or" (SST; percentile 0.5; or subspace); similarly, for BERT "brown-0.4-or" (Brown; per- centile 0.4; or subspace). For other models, not Hence-
mentioned, it is default as "brown-1 wrrown-1.0-or".. .0-or". forth, these settings are held for the conceptor de- biasing on the models respectively.
6.3 Post-Processing Debias via NOT Operator
For general debiasing via conceptor negation post- processing, the performance is excellent. The SEAT score of BERT decreases from 0.620 to around 0.350-0.400 in Brown Corpus (Table 9), and can be as low as 0.311 if using the setting "brown-0.4. or". outperforming the debiasing result of CDA, DROPOUT and SENTENCEDEBIASE (Table 1). The success of debiasing is further verified by Wino- Bias (Table 2), where the skew bias drops from 38.3 to 22.3 without any additional fine-tuning. Although the stereotype bias increases, is not only expected since these two biases are trade-offss but also acceptable, as they now reach good bal- ance (de Vassimon Manela et al., 2021).
6.5
Debias
We use CI-BERT architecture to continue train the models to get the new weights. Then we demonstrate the combinations of architectures and weights as an ablation study (Type 1, I1, and III). Among them, Type III can outperform conceptor post-processing (Table 1), and Type I and II (Ta- ble 4). Compared to the SEAT score after post- processing, Type I can outperform it at each layer of BERT-T but underperform it at most layers of BERT (Table 13 and 18).
In short, using the CI-BERT with the newly trained weights could receive the lowest bias in the model and is promising to beat processng....ng. For example, when using the setting "brown-0.4-
The debiasing conceptors are robust and general- izable, as shown in Table 1, the debias performance

---

Model
SEAT-6
SEAT-6b
SEAT-7
SEAT-7h
SEAT-8
SEAT-8h
Gender (AAvg.)
BERT
0,931
0.090
-0.124
0,937
0.783
0.858
0.620
Conceptor P.P. (tuned)
0.388
-0.078
-0.292
0.179
0.594
0.335
10.309
0.311
Conceptor C.T.
0.227
0.426
-0.341
-0.253
-0.344
-0.088
10.340
0.280
CDA
0,846
0.186
-0.278
1.342
0,831
0.849
10.120
0.722
DROPOUT
1.136
0.317
0.138
1.179
0.879
0.939
10.144
0.765
INLP
0,317
-0.354
-0.258
0.105
0.187
-0.004
[0.416
0.204
SENTENCEDEBIAS
0.350
-0.298
-0.626
0.458
0.413
0.462
10.186
0.434
BERT-L
0.370
-0.015
0.418*
0.221
-0.258
0.711
0.332
Conceptor P.P. (default)
0.197
-02206
0.064
0.065
0.371
0.337
10.125
0.207
GPT2
-0,510
0.057
-0.274
-0.186
+0,369
-0,313
0.285
Conceptor P.P. (tuned)
0.092
0.316
-0,001
0.064
-0.05
-0.062
40.190
0.095
GPT2-L
1.093*
0.192
0.214
1.354
0.861
1.157"
0.812
Conceptor P.P. (default)
1.055'
0.008
-0.089
1.406
0,282
0.992
J0.173
0.639
GPT-J
1.299
0.300
0.962
1.434"
0.617"
1.031"
0.940
Conceptor P.P. (default)
1.184"
0,285
0.661
1.284
0.558
1.024"
10.117
0.833
Table I: SEAT effect size of gender debiased BERT and GPT model. Effect sizes closer to 0 indicate less biased sentence representations (bolded value). Statistically signifieant effect sizes at p 0.01 are denoted by * The final column is the average absolute SEAT score of the first six columns. Defauli means using the default setting: brown corpus, no wordlist filtering, and OR subspace; while tined means using the optimal combination corpus, wordlist percentile, and conceptor bias subspace. P.P. stands for post-processing, while C.T. stands for continued training. The full version is in Appendixes E and G.
Model
Gender (AAvg.)
Race (AAvg.), SEAT-II
SSEAT-I
SEAT-12
SEAT-I3
SEAT-14
SEAT-15
Intersect (AAvg.)
BERT
0.620
0,620, 0.389
-0.424
1.195"
0.525'
0.834*
0.673
Gender Conceptor
10.309
0.311
N/A 0.394"
-0.456
1.156"
0.413"
0.755'
J0.060
0.613
+ Race Conceptor
N/A
10.043
0.577 i 0.394"
-0.456
1.156
0.413"
0.755
J0.062
0.635
Intersect Conceptor
J0.029
0.591
10.016
0.604! 0.214
-0.474
0.872
0.207
0.403
J.0.399
0.434
Table 3: SEAT effect size race, gender, and intersectionally debiased BERT model, where the absolute average SEAT score gender, race, and intersect are across 6, 7, 5 tests, respectively. The full version is Appendix I.
It indicates the conceptor matrix generated by its negated AND operation of gender conceptor matrix and race conceptor matrix
Type (Ori.)) II III
CI-BERT (Arch.) Trained Weights
SEAT
post-processing counterparts in some steps during the early training stage, and then the bias fluctu- ates and increases again, perhaps due to the model relearning the bias during continued training, or oversaturating the conceptor bias projections into its weights.
0.620
0.336
0.592
0.280
Table 4: The ablation study of architecture and weights of CI-BERT evaluated by SEAT (the same as Table 1).
In comparison, the continual-trained CI-BERT can more stably lower the bias in smaller Bert model. We suspect this related to the model com- plexity. The debiasing projection of the last layer's conceptor matrix is upon the last hidden state and thus generated transitively from all the prior layers. Currently, we are embedding all layers' conceptor matrices, which may lead to overlapping and re- dundant debiasing projection from the prior layers.
or", the lowest SEAT score is 0.280, beating the post-processing result of 0.311 and more than half of the SoTA methods. This is verified again by gen- der co-reference resolution Table 2-in compar- ison to its post-processing counterpart, CI-BERT continued training lowers both stereotype bias by 22.9 and skew bias by 5.3 from Test Set 2 of Wino- Bias. This is non-trivial since these two biases are a tradeoff and thus generally hard to decrease si- multancously (de Vassimon Manela al., 2021).
6.6 Maintaining Meaningful Semantics
To understand how conceptor debiasing impacts the downstream natural language understanding (NLU) tasks, the GLUE benchmark (Wang et al., 201 c)-coppd of nine different tasks-is used evaluate the model after debiasing (Table 5). While there seems to be no consensus about the quanti-
To further study the feasibility and robustness of CI-BERT continued training concerning the model property. We experiment on both BERT-T and BERT and plot the average SEAT curve along with training steps (Figure 6). Both can beat their

---

bert-liny, layer
Model
Average
1,2
BERT
77.74
Conceptor P.P.
11.77
79.51
Conceptor
J1.03
76.71
1.0
Trained Weignts C-BERT
Post:Process
CDA
J0.22
77.52
Abx. SEAT
Origit
DROPOUT
J1.46
76.28
0.3
INLP
J0.99
76.76
SENTENCEDEBIAS
70.07
77.81
0.5
BERT-L
78.81
Conceptor P.P.
70.05
78.86
GPT2
73.01
0.4
Conceptor P.P.
J0.11
72.90
GPT2-L
75.84
Training Step
164
Conceptor P.P.
10.05
75.89
GPT-J Conceptor P.P.
78.22
SEAT Score Curve: bert-base-unassed layer 12
J0.14
78.06
Trained Weights x CHEERT
1.0
Post Process
Table 5: GLUE validation set results for gender debi- ased BERT and GPT model. The full version Ap- pendixes E and H.
Origin
Avc. Aps. SEAT Stre
0.8
0.6
Notice that even when trained the original BERT architecture, the average GLUE score still drops about 0.3 point. Thus, the lower GLUB score here is not completely caused by CI-BERT, though the actual reason is hard to determine due to train- ing randomness (McCoy et al., 2019).
0.4
0.7
Traiiing
Figure 6: SEAT score curve of CI-BERT continued train- ing. We compare the results with the original embed- dings and post-processed embeddings. We test the last layer of BERT-T (top) and BERT (bottom).
Conclusion and Future Work
tative threshold of the trade-off between language modeling capability and debiasing performance, a small decrease may be acceptable depending on the downstream tasks. We believe that, in an ideal scenario, the performance on the GLUE benchmark should not significantly decline after debiasing.
We have shown that conceptor-aided debiasing successfully mitigate bias from LLMs (c.g., BERT, GPT) by its NOT operation. Specifically, conceptor post-processing outperforms many state- of-the-art debiasing methods in both debiasing fectiveness and semantic retention. We also tested a new architecture, conceptor-intervent BERT), which in combination with continued train- ing, takes all layers' biases into account and shows the promise to outperform its post-processing coun- terpart. However, it might be the cost of in- creased instability and worse semantic retention. In all cases, the best conceptor matrices are gener- ally obtained when the bias subspace is constructed using (I) a cleaner corpus, (2) the union of different related wordlists (e.g. pronouns, roles, and names) by the conceptor OR operation, and (3) removal of outliers from the wordlists. We further show that cocneptor-aio aided debiasing is robust in differ- LLMs, various layers of models, and varied types of biases. Moreover, conceptors can utilize the current conceptor matrices to construct a new conceptor matrix to mitigate the intersectional bias in an efficient manner by AND operation.
The conceptor post-processing BERT can re- tain and even improve the useful semanties (in- crease the average GLUE score by 1.77) for down- stream tasks without damage to the model's abil- ity, outperforming any other listed SoTA debiasing methods. Even if scaling to BERT-L, the GLUE is still slightly higher. In comparison, the average GLUE score of conceptor continued-trainine BERT is relatively low, although it is not the worst among all the methods. This indicates that the continued- training method, while still capable of outperform- ing its post posseesoocssing counterpart under the same setting, may sacrifice NLU abilities.
BERT (CI-
Since GPT is an autoregressive model, we adopt the SequenceCClassifcaation counterpart the GLUE benchmark, following the method of Meade et al. (2022). The score of GPT2 and GPT-J are decreased slightly by 0.11-0.14 14, which is an affordable cost, while GPT2-L increases slightly by 0.05.
In future research we plan to make CI-BERT and intersectional conceptors more robust and effective.

---

Limitations
sometimes one wants see the bias inherent in corpus.
We list several limitations of our work below.
A Acknowledgements
1) We only test the binary bias. We only test the bias in pairs via SEAT and WinoBias, for ex- ample, 'male'/'fmaal" / or "young''oldd ''od''. However, it is widely recognized that terms in gender, race. etc. can be multi-polar.
First, we would like to thank the reviewers for their fruitful discussion with us. We would also like to thank Claire Daniele for her editorial support. Last but least, we appreciate PennNLP members for their helpful comments.
2) Our result is limited to English, and both corpora and wordlist tend towards North Amer- ican social biases. The whole of our experiment is conducted in English. In addition, Brown and SST Corpora are collected entirely in the North American environment. So the wordlists. There- fore, it is expected that they skew towards North American social biases. When such models are de- biased under the North American environment, is necessary to understand how effective they are when transferred to other cultures.
References
Maria Antoniak and David Mimno. 2021. Bad seeds: Evaluating lexical methods for bias measurement. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natural Lan- guage Processing (Volume I: Long Papers), pages 1994,1904, Online. Association for Computational Linguistics.
Soumya Barikeri, Anne Lauscher, Ivan Vulić, and Goran Glavaš. 2021. RedditBiast real-world resource for bias evaluation and debiasing of conversational lan- guage models. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joins Conference on Natural Language Processing (Volume 1: Long Papers), pages 1941-1955, Online, Association for Computational Linguistics.
3) The generalization of conceptor-aided debias- ing techniques can be tested more exhaustively. This work has tested it on gender and race, but can also be tested on other types of bias such as religious bias and hate speech.
Ethical Considerations
The definition and recognition of bias are subtle. For example, we have used simple traditional bi- nary definitions of male and female examine gender bias. This, of course, ignores much wider variety of gender identities, thus introducing an implicit bias to the analysis. Similarly, studies on racial bias rely on possibly problematic definitions race. Core to our debiasing method is the selec- tion of the wordlists. Each wordlist carries its own implicit definitions of gender, race, and other im- portant dimensions. Care should be used to ensure that they represent the desired categories. To this end, it is often useful involve people from the communities whose language is being debiased to better represent their values and belief systems.
Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to home- maker? debiasing word embeddings. Advances in neural information processing systems, 29.
Rishi Bommasani, Kelly Davis, and Claire Cardie. 2020. Interpreting pretrained contextualized representations via reductions to static embeddings. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4758 4781.
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. 2020. Language models few-shot learners. Advances in neural information processing systems, 33:1877-1901.
One should also be careful in the use of debi- asing. Removing signals about race or gender is often beneficial in reducing discrimination or in producing better translations. may also remove key features of models needed for analyses. For ex- ample, removing gender or race 'signal' from the model may severely hamper the use of that model in gender studies or work on critical race thcory. "White-washing models are not always a benefit;
Aylin Caliskan, Pimparkar Parth Ajay, Tessa F. S. Charlesworth, Robert Wolfe, and Mahzarin R. Banaji. 2022. Gender bias word embeddings: compre- hensive analysis of frequency, syntax, and semantics. ArXiv, abs/2206.03390.
Aylin Caliskan, Joanna J Bryson, and Arvind Narayaman. 2017. Semantics derived automatically from lan- guage corpora contain human biases. Science, 356(6334)) 183-166.

---

Daniel de Vassimon Manela, David Errington, Thomas Fisher, Boris van Breugel, and Pasquale Minervini. 2021. Stereotype and skew: Quantifying gender bias in pre-trained and fine-tuned language models. In EACL.
Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Saško, Gun- jan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matus- sière. Lysandre Debut, Stas Bekman, Pierric Cis- tac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. 2021. Datasets: A community library for natural language processing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing: System Demonstrations, pages 175-184, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
W Nelson Francis and Henry Kucera. 1979. Brown corpus manual. Letters the Editor, 5(2):7.
Paul Pu Liang, Irene Mengze Li. Emily Zheng, Yao Chong Lim, Ruslan Salakhutdinov, and Louis- Philippe Morency. 2020. Towards debiasing sentence representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics.
Hila Gonen and Yoav Goldberg, 2019. Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Compula- fional Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages (99-614, Minneapolis, Minnesota. Association for Computa- tional Linguistics.
Tianlin Liu, João Sedoc, and Lyle Ungar. 2018. Cor- recting the common discourse bias in linear represen- tation of sentences using conceptors. arXiv preprint arXiv: 1:1111811.111001
Xu He and Herbert Jaeger. 2018 Overcoming catas- trophic interference using conceptor-aided backprop- agation. In International Conference on Learning Representations.
Tianlin Liu, Lyle Ungar, and João Sedoc. 2019a. Con- tinual learning for sentence representations using con- ceptors. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume (Long and Short Papers), pages 3274-3279.
Herbert Jaeger. 2014. Controlling recurrent neu- ral networks by conceptors. arXiv preprint arXiv:1403.3369.
Masahiro Kaneko and Danushka Bollegala. 2021. De- biasing pre-trained contextualised embeddings. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Lin- guistics: Main Volume, pages 1256-1266.
Tianlin Liu, Lyle Ungar, and Joao Sedoc. 2019b. Unsu- pervised post-processing of word vectors via concep- tor negation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6778- 6785.
Saket Karve, Lyle Ungar, and João Sedoc. 2019. Con- ceptor debiasing of word representations evaluated on weat. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 40-48.
Chandler May, Alex Wang, Shikha Bordia, Samuel Bow- man, and Rachel Rudinger. 2019. On measuring so- cial biases in sentence encoders. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 622-628.
Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black,
and Yulia Tsvetkov. 2019. Mcasuring bias in con- textualized word representations. In Proceedings of
R Thomas McCoy, Junghyun Min, and Tal Linzen. 2019. Berts of feather do not generalize together: Large variability generalization across models with similar test set performance. arXiv preprint arXiv: 1911.0299..
the First Workshop on Gender Bias in Natural Lan- guage Processing. pages 166-172, Florence, Italy. Association for Computational Linguistics.
John Lalor, Yi Yang, Kendall Smith, Nicole Forsgren, and Ahmed Abbasi. 2022. Benchmarking intersec- tional biases in NLP. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 3598-3609, Seattle, United States. Association for Computational Lin- guistics.
Leland McInnes, John Healy, and James Melville. 2018. Umap: Uniform manifold approximation and pro- jection for dimension reduction. arXiv preprint 102.v:180..03426.
Nicholas Meade, Elinor Poole-Dayan, and Siva Reddy. 2022. An empirical survey of the effectiveness of

---

debiasing techniques for pre- pre-trained language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume Long Papers), pages 1878-1898, Dublin, Ireland. Association for Computational Linguistics.
Yi Chern Tan and Elisa Celis. 2019. Assessing so- cial and intersectional biases in contextualized word representations. Advances in neural information pro- cessing systems, 32.
Laurens Van der Maaten and Geoffrey Hinton, 2008.
Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. StercoSet: Measuring stereotypical bias in pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the I 1ith International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5356-5371, Online. Association for Computational Linguistics.
Visualizing data using t-sne. Journal of machine learning research, 9(11).
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman, 2018. GLUE: A multi-task benchmark and analysis plat- form for natural language understanding. ArXiv preprint 1804.07461.
Kellie Webster, Xuezhi Wang. Ian Tenney, Alex Beutel, Emily Pitler, Fillie Pavlick, Jilin Chen, Ed Chi, and Slav Petrov. 2020. Measuring and reducing gendered correlations pre-trained models. arXiv preprint arXiv:2010.0603..
Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020. CrowS-pairs: chal- lenge dataset for measuring social biases in masked language models. In Proceedings of the 2020 Con- ference on Empirical Methods Natural Language Processing (EMNLP), pages 1953-1967, Online. As- sociation for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi. Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, and Jamie Brew. 2019. Huggingface's 's transformers: State-of-the natural language processing. ArXiv, abs/ 10100911.
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke 7eulemoyer. 2018. Deep contextualized word repre- sentations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long Pupers), pages 2227-2237, New Orleans, Louisiana. Association for Computa- tional Linguisties.
Zhonghin Xie and Thomas Lukasiewiez. 2023. An em- pirical analysis of parameter-efficient methods for de- biasing pre-trained language models. arXiv preprint ar 2.062700004067.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Cotterell, Vicente Ordonez, and Kai-Wei Chang. 2019. Gender bias in contextualized word embeddings. In NAACI (short).
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg, 2020. Null it out: Guard- ing protected attributes by iterative nullspace projec- tion. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7237-7256, Online. Association for Computational Linguistics.
Jieyu Zhao, Tianlu Wang. Mark Yatskar, Vicente Or- donez, and Kai-Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. In NAACL (short).
Timo Schick, Sahana Udupa, and Hinrich Schüizze. 2021. Self-diagnosis and self-debiasing: A proposal for re- ducing corpus-based bias in nlp. Transactions of the Association for Computational Linguisties, 9:1408- 1424.
João Sedoc and Lyle Ungar. 2019. The role of protected class word lists in bias identification of contextual- ized word representations. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 55 61.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng. and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference empiri- cal methods in natural language processing, pages 163 11642.

---

A Attribute Wordlist
A.4 Race Wordlist
Words (in total 8); africa, african, america, asia, asian, caucasian, china, europe
The examples and sources of the attribute wordlists are given below. Due to space limitations, would only provide up to 50 words for each list.
A.5
Outlier Filtering
A.1 Pronouns Wordlist
Table 6 shows the number of remaining gender words per percentile after being filtered on UMAP- clustering space.
Words (in total 22): son, mother, daughter, him, brother, girl, uncle, hers, grandfather, his, boy, her, father, she, sister, man, female, aunt, woman, grand- mother, he, male.
Bias Subspace Type
Percentile
pronouns
extended
properouuns
all
0.5-1.0
22
388
7578
7988
They are the concatenation of W7_terms and W8 terres from WEAT list
0.4
22
388
5443
6902
0.3
11
372
4942
5140
0.2
7
364
3194
3289
A.2 Extended Wordlist
0.1
4
67
I067
1087
Words (randomly 50 of 388): paramour, abbesses, headmistress, stepson, gods, congressman, gents, uncle, hers, wizard, cowgirls, fiancees, adultress, sororal, ladies, sons, uncles, actors, beards, heiress, fellas, salesman, princess, empress, masters, chair- women, miss, horsewomen, actor, mr. strong- woman, barons, andrology, busboy, prince, hens, womb, masseuse, lady, testosterone, daughter, girl, stateswoman, businessmen, women, fraternities, aunts, boys, abbot, heroine,
Table 6: The number of remaining words per percentile after filtering UMAP- MMMP-clustering space. The model is "bert-base-uncasee""
B Model Checkpoints
We use the Hugging Face Transformers pack- age (Wolf et al., 2019) in our experiments. The models and checkpoint names are given in Table 7.
Model
Checkpoint
They are the concatenation of lists: WinoBias extra gendered words VN-GloVe male's name and female's name?.
BERT-T prajjwall/bbrr-tiing
BERT
bert-base-uncased
BERT-L
bert-large-uncased
GPT2
gpt2
A.3 Propernouns Wordlist
GPT2-L
gpt2-large
Words (randomly 50 of 7578): Broddie, Tony, Taw- sha, Emylee, Orelle, Gerrilee, Katusha, Georges, Reine, Hayley, Deloria, Richmond, Wilfrid, Neille, Florie, Riva, Sandro, Cooper, Thom, Pate, Niko- letta, Rodrique, Pat, Chuck, Theressa, Brett, Kas- par, Elric, Storm, Yule, Bubba, Thomasina, An- son, Margery, Abra, Benedict, Cy, Gertrud, Morly, Julina, Melly, Quinta, Paolo, Brynne, Maurene, Alexis, Ramsey, Sianna, Phebe, Alfred,
GPT-J
EleutherAI/got-j-66
Table 7: The package's model and checkpoint name in our experiment.
C Continued Training Details
The conceptor-intervened model trained for one epoch by setting prediction_los__oot as true and per_ terprice_trri_aaateeeeeee lowing the training procedure in Devlin et al. (2019), we train by tasks Masked Language Model (MLM) and Next Sentence Prediction (NSP) si- multancously. The training corpus is the Wikipedia dump from datasets library (Lhoest et al., 2021).
as 8. Fol-
They are the concatenation of lists: CMU male's name and female's name?.
'nttps://githubs wtw. /tcp:///c///ommpptrrebbllll///// aubbmmasterr//isttt/tt___LLSS
mastrr/ssteeiiniisss//ffreeeeeee txt
D GLUE Details
/
fitpseeememeeeeodddleeeieeeeeeee txt
Before being evaluated GLUE, each model is trained for three epochs with the following settings: batch_size m_ sawuuu_sequuc__eeeeeeee 128, and learning_rate 2e-5; the same as Meade
masttr//wordList//mlleee wil.f..feeeee
"https://www. cmu. edtu/rrouss/AT/aress//ll
corpora/namess/male. .xx
Phttps://wwil cw.ccc.cccmmmmmmm http::.ww.c.....erruup///////// a...c..ea///rrrps/////aaa//////
corpora/names/femal. txt
et al. (2022).

---

E Full Bert-Base-Uncasee Model Results
Table shows the gender debiasing result by different types of the corpus, using the last layer of "bert-baaseuncaseed as a benchmark.
Table 9, and 10, 11 show the post-processing gender debiasing result of different percentiles of wordlist three different corpora: Brown, SST, and Reddit, respectively.
Table 12 and 13 show the post-processing and conseptor-intervened gender debiasing result of each layer on two different corpora: Brown and SST, respectively.
Table 14 contains GLUE results for the gender debiased model.
F Full Bert-Tiny Model Results
Table 15, 16, and 17 show the post-processing gender debiasing result of different percentiles of wordlist three different corpora: Brown, SST, and Reddit, respectively.
Table 18 shows the post-processing and conseptor-intervennd gender debiasing result of each layer the SST corpus.
G Full GPT2 Model Dehiasing Results
Table 19 shows the post-processing gender debiasing result of different percentiles of wordlist on Brown corpus.
H Full Other LLMs' GLUE Results
Table 20 contains GLUE results for gender debiased model.
I Full Intersectional Debiasing Results
Table 21 and 22 show the post- processocssiiggg in- tersectional debiasing results.

---

Model
SEAT-6
SEAT-6b
SEAF7
SEAT-76
SEAT-8
SEAT-8b 0.858
Avg. Abs.
BERT ("bert-buse-uncasee""
0.931
0.090
-0.124
0.937
0.783
0.620
(Brown Corpus)
+ Conceptor- 12 (pronouns)
0.488
-0.091
-0.311
0.471
0.783"
0.621
J0.156
0.464
+ Conceptor- 12 (extended)
0.509
-0.109
-0.406
0.240
0.606
0.449
J0.234
0.386
+ Conceptor- 12 (propernouns)
0.581
-0.053
-0.258
0.187
0.585
0.659
J0.233
0.387
+ Conceptor-12 12 (all)
0.452"
-0.123
-0.277
0.258
0.662'
0.607
J0.224
0.396
+ Conceptor-12 (or)
0.440
-0.063
-0.136
0.251
0.640'
0.617
J0.262
0.358
(SST Corpus)
+ Conceptor-11 12 (pronouns)
0.627
-0.104
-0.416
0.520
0.636"
0.628
J0.132
0.488
+ Conceptor- 12 (extended)
0.626
-0.068
-0.365
0,280
0.556
0.429
J0.233
0.387
+ Conceptor-12 12 (propernouns)
0.680
-0.050
-0.405
0.614'
0,585
0.790
[0.099
0.521
+ Conceptor- 12 (all)
0.624
-0.093
-0.480
0,442'
0,538'
0.663"
J0.147
0.473
Conceptor- 12 (or)
0.606
10.100
-0.447
0.337
0,428
0.575
J0.204
0.416
(Reddit Corpus)
Conceptor-12 (pronouns)
0.619"
-0.092
-0.235
0.816'
0.756
0.962"
J0.040
0.580
Conceptor-12 12 (extended)
0.630"
-0,061
-0.157
0.676'
0.711
0.806
J0.113
0.507
Conceptor-12 12 (propernouns)
0.792
0.151
0.068
0.964
0.765
0.934
10.008
0.612
Conceptor- 12 (all)
0.613"
-0.010
0.004
0.803'
0.735'
0.917
J0.106
0.514
+ Conceptor- (or)
0.593
0.004
0.092
0.838
0.652
0.961
10.097
0.523
CDA
0.846
0.186
0.278
I.342
0.331
",,349
T6.130"
0.722
+ DROPOUT
1.136
0.317
0.138
0,879
0.939*
10.144
+ INI.
0.317
-0.354
-0.258
0.105
0.187
-0,004
J0.416
0.204
+ SENTENCEDEBIAS
0.350
-0,298
-0.626
0.458
0.413
0.462
10.186
0.434
Table 8: SEAT effect size of gender debising. The impact of different corpora bert-base uncased models. Effect
sizes closer to are indicative of less biased sentence representations (bolded value). Statistically significant effect sizes at p < 0.01 are denoted by Note that the "conceptor-X" X (subspace)" indicates the conceptor negation matrix is generated by the X X-layer of the language model in combinations with the subspace of the specific attribute wordlist. The top-3 best performance is colored orange.

---

Model
SEAT-6
SEAT-6b 0.090
SEATE7 -0.14
SEAT-76 0.937
SEAT-8 0.783"
SEAT-8b 0.858"
Avg. Abs.
BERT ("bert-base unsar--ees-ucssd"
0.931
0.620
(Wordlist Percentile 0.5-1.0)
+ Conceptor-12 12 (pronouns)
0.488
-0.091
-0.331
0.471
0.783
0.621
10.156
0.464
+ Conceptor-12 12 (extended)
0.509
-0.109
-0.406
0,240
0.606
0.449'
J0.234
0.386
+ Conceptor- 12 (propernouns)
0.581
-0.053
-0,258
0.187
0,585
0.659'
10.233
0.387
+ Conceptor- 12 (all)
0.452"
-0.123
-0,277
0.258
0.662"
0.607
J0.224
0.396
+ Conceptor- 12 (or)
0.440
-0.063
-0.136
0.251
0.640
0.617'
10,262
0.358
(Wordlist Percentíle 0.4)
Conceptor-12 12 (pronouns)
0.483"
-0.095
-0.385
0,435'
0.776
0.609"
J0.156
0.464
Conceptor- 12 (extended)
0.509
-0.110
-0.407
0.239
0.603"
0.447'
J0.234
0.386
Conceptor- 12 (propernouns)
0.451
-0.188
-0.005
-0.122
0.399
0.264
J0.298
0.322
Conceptor-12 12 (all)
0.466
-0.112
-0.260
0.267
0.697'
0.617'
J0.217
0.403
+ Conceptor- 12 (or)
0.388
-0.078
-0,992
0.179
0.594
0.335
J0.309
0.311
(Wordlist Percentile 0.3)
+ Conceptor- 12 (pronouns)
0.487
-0.016
-0.351
0.398*
0.807"
0.776'
J0.148
0.472
+ Conceptor- 12 (extended)
0.509
-0.109
-0.410
0.228
0.604
0.453'
J0.235
0.385
+ Conceptor-11 12 (propernouns)
0.495
-0.168
-0.478
-0.083
0,456
0.315
J0.288
0.332
+ 1onceptor-12 (all)
0.348
-0,236
-0.520
0.506'
J0.288
0.332
+ Conceptor- 12 (or)
0.407
-0.022
-0,247
0.331
0.677'
0.483
J0,259
0.361
(Wordlist Percentile 0.2)
+ Conceptor- 12 (pronouns)
0.570"
0.035
-0.378
0.334
0.708"
0.768
J0.154
0.466
+ Conceptor- 12 (extended)
0.508
-0.109
-0.416
0.219
0.602'
0.450'
J0.236
0.384
+ Conceptor-12 12 (propernouns)
0.548
-0.157
-0.397
0.270
0.483'
0.366
J0.250
0.370
+ Conceptor-12 12 (all)
0.357
-0,235
-0.598
0.110
0.455'
0.383
J0.264
0.356
+ Conceptor-12 (or)
0.476
-0.063
-0.385
0.296
0.558'
0.500
J0.240
0.380
(Wordlist Percentile 0.1)
+ Conceptor-12 12 (pronouns)
0.800"
0.204
-0.314
0.273
0.764
0.965"
J0.067
0.553
+ Conceptor-12 12 (extended)
0.869"
0.162
-0.265
0,266
0.861
0.635'
J0.110
0.510
+ Conceptor-12 (propernouns
0.613
-0.084
-0.582
0.190
0.579
0.740*
J0.155
0.465
+ Conceptor-12 12 (all)
0.603"
-0.102
-0.612
0.182
0.566
0.712"
J0.157
0.463
1onceptor-12 (or)
0.614
0.197
-0.401
-0.132
0.624
0.699
10.176
0.444
CDA
0.846
T..86
-0.778
T.342
0.331
003497
10.1200
0.722
DROPOUT
1.136
0.317
0.138
1.179*
0.879
0.939*
10.144
0.765
INLP
0.317
-0.354
-0.258
0.105
0.187
-0.004
J0.416
0.204
SENTENCEDEBIAS
0.350
-0.298
-0.626
0.458
0.413
0.462
10.186
0.434
Table 9: SEAT effect size of gender debising. The impact of different percentiles of wordlist (using UMAP clustering) Brown Corpus, bert-base- uncsedd models. top-3 best performance colored orange.

---

Model
SEAT-6
SEAT-6b 0.090
SEATE7 -0114
SEAT-76 0.937
SEAT-8 0.783"
SEAT-8b 0.858"
Avg. Abs.
BERT ("bert-base unstr-beed-ucssd""
0.931
0.620
(Wordlist Percentile 0.5-1.0)
+ Conceptor-12 12 (pronouns)
0.627
-0.104
-0.416
0.520
0.636
0.628
J0.132
0.488
+ Conceptor-12 12 (extended)
0.688
0.024
-0.293
0.138
0.559'
0.375
J0.274
0.346
+ Conceptor- 12 (propernouns)
0.680
-0.050
-0.405
0,614
0,585
0.790
10.099
0,521
+ Conceptor- 12 (all)
0.624"
-0.093
-0.480
0,442"
0,538'
0.663"
J0.147
0,473
+ Conceptor- 12 (or)
0.619
-0,069
-0.288
0.280
0.414
0,539
10,229
0.391
(Wordlist Percentíle 0.4)
Conceptor-12 12 (pronouns)
0.619
-0.113
-0.526
0,449*
0.606
0,584"
J0.137
0.483
Conceptor- 12 (extended)
0.688
0.024
-0.293
0.138
0,559
0.375
J0.274
0.346
Conceptor- 12 (propernouns)
0.704
-0,086
-0.227
0.590
0.682'
0.716
J0.119
0.501
Conceptor-12 12 (all)
0.622
-0.087
-0.508
0.277
0.519
0.578
J0.188
0.432
+ Conceptor- 12 (or)
0.646
-0.034
-0.427
0.200
0.438'
0.401
J0,262
0.358
(Wordlist Percentile 0.3)
+ Conceptor- 12 (pronouns)
0.550
-0.035
-0.396
0.344
0.682"
0.744"
J0.161
0.459
+ Conceptor- 12 (extended)
0.687
0.023
-0,299
0.129
0.559'
0.382
J0.273
0.347
+ Conceptor-11 12 (propernouns)
0.706
-0.088
-0,230
0.602
0.683"
0.720
[0.115
0.505
+ Conceptor-12 (all)
0.652
-0.117
-0.378
0.504
0.536'
0.657'
J0.146
0.474
+ Conceptor- 12 (or)
0.595
0.027
-0.375
0.171
0.519*
0.600
J0,239
0.381
(Wordlist Percentile 0.2)
+ Conceptor- 12 (pronouns)
0.730"
0.090
-01100
0.523
0.714"
0.758
J0.132
0.488
+ Conceptor- 12 (extended)
0.687"
0.023
-0,299
0.129
0,559*
0.382
J0.273
0.347
+ Conceptor-12 12 (propernouns)
0.755
-0.057
-0.346
0.584
0.659*
0.733*
J0.098
0.522
+ Conceptor-12 12 (all)
0.699
-0.094
-0.492
0.456
0.620'
0.688
J0.112
0.508
+ Conceptor-12 (or)
0.579
-0,004
-0.261
0.270
0.503'
0.634
J0.245
0.375
(Wordlist Percentile 0.1)
+ Conceptor-12 12 (pronouns)
0.903"
0.198
-0.464
0.030
0.434
0.492
J0.200
0.420
+ Conceptor-12 12 (extended)
0.631"
-0.107
-0.012
1,008"
0.615'
0.794"
J0.092
0.528
+ Conceptor-12 (propernouns
0.655
-0.130
-0.166
0.895
0.641'
0.877
[0.059
0.561
+ Conceptor-12 12 (all)
0.597
-0.164
-0.223
0.791
0.698
0.909"
10.056
0.564
12 (or)
0.542
-0.039
-0.184
0.589*
0.457'
0.929*
10.163
0.457
CDA
0.846
T.186
-0.778
T.34.
0.331
0,3497
10.120"
0.722
DROPOUT
1.136
0.317
0.138
1.179*
0.879
0.939*
10.144
0.765
INLP
0.317
-0.354
-0.258
0.105
0.187
-0.004
J0.416
0.204
SENTENCEDEBIAS
0.350
-0.298
-0.626
0.458
0.413
0.462
10.186
0.434
Table 10: SEAT effect size of gender debising. The impact of different percentiles of wordlist (using UMAP clustering) on SST Corpus, port-base-uncasee models. The top-3 best performance colored in orange.

---

Model
SEAT-6
SEAT-6b 0.090
SEATE7 -0114
SEAT-76 0.937
SEAT-8
SEAT-8b 0.858"
Avg. Abs.
BERT ("bert-basee unsare-eed-ucssd"
0.931
0.783"
0.620
(Wordlist Percentile 0.5-1.0)
+ Conceptor-12 12 (pronouns)
0.619
-0.092
-0,235
0.816
0.756
0.962
10.040
0.580
+ Conceptor-12 12 (extended)
0.630
-0,061
-0.157
0.676
0.711
0.806
10.113
0.507
+ Conceptor- 12 (propernouns)
0.792
0.151
0.068
0.964
0.765'
0.934
10.008
0.612
+ Conceptor- 12 (all)
0.613"
-0.010
0.004
0.803'
0.735'
0.917"
10.106
0.514
+ Conceptor- 12 (or)
0.593
0.004
0.092
0.838'
0.652"
0,961
10.097
0.523
(Wordlist Percentíle 0.4)
+ Conceptor-12 12 (pronouns)
0.618
-0.095
-0.297
0.769
0.728
0.944
J0.045
0.575
Conceptor- 12 (extended)
0.732"
0.047
-0.121
0.601
0.748
0.719"
J0.125
0.495
Conceptor- 12 (propernouns)
0.691
-0.063
0.315
1.028'
0.667'
0.635'
J0.054
0.566
Conceptor-12 12 (all)
0.651
-0.010
-0.004
0.786'
0.732
0.940"
10.100
0.520
+ Conceptor- 12 (or)
0.526
-0.057
0.054
0.596'
0.655'
0.724
J0.185
0.435
(Wordlist Percentile 0.3)
+ Conceptor- 12 (pronouns)
0.596
-0.068
-0.286
0.772*
0.790
0.983"
J0.038
0.582
+ Conceptor- 12 (extended)
0.740
0.043
-0.119
0,593'
0.758
0.725'
J0.124
0.496
+ Conceptor-11 12 (propernouns)
0.825
0.007
0.512
1.180"
0.761"
0.705'
10.045
0.665
+ 1onceptor-12 (all)
0.689
0.007
0.180
0.924
0.627'
0.653'
J0.107
0.513
+ Conceptor-12 12 (or)
0.612
-0.018
0.046
0.745'
0.778
0.916
J0.101
0.519
(Wordlist Percentile 0.2)
+ Conceptor- 12 (pronouns)
0.801
0.104
-0.072
0.757'
0.873"
0.997"
J0.019
0.601
+ Conceptor- 12 (extended)
0.740
0.043
-0.119
0.593'
0.758"
0.725'
J0.124
0.496
+ Conceptor-12 12 (propernouns)
0.837
0.037
0.532"
1.170'
0.785
0.722
10.060
0.680
+ Conceptor-12 (all)
0.694"
-0,042
0,356
1.044
0.608
0.554
J0.070
0.550
+ Conceptor-12 (or)
0.665
0.096
0.130
0.607
0.852*
0.842*
J0.088
0.532
(Wordlist Percentile 0.1)
+ Conceptor-12 12 (pronouns)
0.949
0.121
-0.472
0.102
0.568"
0.707'
J0.134
0.486
+ Conceptor-12 12 (extended)
0.736
-0.035
-0.291
0.780'
0.812"
1.095'
10.005
0.625
+ Conceptor-12 (propernouns
0.948
0.107
-0.094
0,949*
0.783*
0.842*
0.620
+ Conceptor-12 12 (all)
0.949*
0.105
-0.061
0,899
0.782'
0.846
J0.013
0.607
12 (or)
0.936"
0.130
0.579
0.098
0.591
0.914
10.079
0.541
CDA
0.846
T..86
-0.778
T.34.
0,831
003499
10.1200
0.722
DROPOUT
1.136
0.317
0.138
1.179*
0.879
0.939*
10.144
0.765
INLP
0.317
-0.354
-0.258
0.105
0.187
-0.004
J0.416
0.204
SENTENCEDEBIAS
0.350
-0.298
-0.626
0.458
0.413
0.462
10.186
0.434
Table 11: SEAT effect size of gender debising. The impact of different percentiles of wordlist (using UMAP clustering) on Reddi Corpus, hert-base-uncutt models. The top-3 best performance colored in orange.

---

Model
SEAT-6
SEAT-66
SEAT-7
SEAT-7b
SEAT-8
SEAT-8b
Avg. Abs.
(Layer 0)
BERT ("bert-baseuueuseed""
0
0.194
0.251
-0.122
-0.110
0.366
0.336
+ Conceptor-O (or)
0.147
-0.087
-0.266
-0.633
-0.405
-0.324
10.022
0.314
+ Conceptor-Intervenee
0.147
-0.087
-0.266
-0.653
-0.405
-0.324
J0.022
0.314
(Layer 1)
BERT "bert-hase-unnased""
1.245*
0.292
0.469
1.101"
0.110
1,261
0.746
+ Conceptor-1 (or)
0.473
0.205
-0.210
-0.093
-0,095
0.396
10.501
0.245
+ Conceptor-Intervened
0.241
-0,038
-0.274
0.291
-0.751
-0.107
10,462
0.284
(Läyer 2)
BERT ("bert-hasee ""hert-hasc-uncssed")
1.149"
0.216
0.431
1.021
0.474
1,231
0.754
Conceptor-2 (or)
0.180
-0.047
00450
-0.055
0.133
0.133
10.579
0.175
Conceptor-Intereened
-0.108
0.115
0.965
1,388"
-1.146
0.329
10.079
0.675
([ayer3)
BERT ("bert-bas-uuccsedd")
1.186
0.214
0.152
0.770
0.262
1.049
0.606
+ Conceptor-3 (or)
0.404
-0.046
-0.755
0.102
-0,325
-00024
10.343
0.263
+ Conceptor-Intervenee
0.158
0.081
-0.599
1.348"
-1.093
0.409
10.069
0.675
()ayer
BERT ""bert-bae-unnasedd"
0.975
0.106
0.552*
0.890
0.542
0.724
0.632
+ Conceptor-4 (or)
0.597
0.068
-0.249
0.251
-0,016
-0.315
J0.383
0.249
+ Conceptor-Interveen
0.060
0.121
00.86
1.676"
-0840
0,943
10,139
0.771
(Layer
BERT ("bert-base-uncaset
1.002*
0.184
0.628
0.914
0.376
1.053*
0.693
+ Conceptor-5 (or)
0.634
0.064
0.118
0.225
-0.160
0.429
J0.421
0.272
+
0.046
0.043
-1003
1.378
-0.79
0.659
10.034
0.659
(Läyer
BERT ("bert-base-uncuseed)
0.753*
0.118
0.539
1.048*
0.597
1.042*
0.683
+ Conceptor-6 (or)
0.327
0.041
0.176
0.104
0.150
0.174
J0.521
0.162
+ Conneptor-Intervened
-0.210
0.004
-..65
1,242
-0.739
0.475
10.077
0.606
[Layer?)
BERT ("bert-base-uncusedd")
0.719
0.155
0.341
0.935
0.562
0.721
0.572
Conceptor-7 (or)
0.235
-0,064
-0,038
0.206
0.173
0.223
J0.416
0.156
Conceptor-Intervened
-0,246
-0.082
-0.821
1.112
-0.671
0.248
J0.042
0.530
(Layer 8)
BERT ('bettbase-uncusedd))
0.983"
0.163
0.313
1.157'
0.766'
0.789*
0.695
+ Conceptor-8 (or)
0.235
0.005
-0.136
0.389
0.379
J0.482
0.213
+ Sanceptor-Intervenee
-0.125
-0.193
-0.940
0.796
-0.606
0.084
J0.238
0.457
(Layer 9)
BERT "bert-base-unccueed")
0,922
0.224
0.503*
1.293"
0.780
0.996
0.786
+ Conceptor-9 (or)
0.234
0.019
-0.005
0.485
0.694
0.686
J0.432
0.354
+ Sanceptor-Intervenedd
-0.151
-0,246
-0.599
0.836
0.455
-0.95
J0.389
0.397
(Layer 10)
BERT ""bettbass-unccused")
0.686
0.082
0.226
0.894
0.904
0.965
0.626
+ Conceptor- 10 (ur)
0.294
-0.091
-0.153
0.078
0.703
0.545
10.315
0.311
+ lentepeor-Intervveeed
-0,253
-0.298
-0.569
0.753'
-0,462
-0.099
10.221
0.405
(Layer 11)
BERT ("bert- ("bert-base-unee -useb-dd
0.665
-00155
00.44
0.602"
0,919*
0.891
0.573
Conceptor-11 11 (or)
0.197
-0.114
-0.399
-0.157
0.557'
0.277
10.289
0.284
Conceptor-Interveneed
-0.314
-0.269
-0.635
0.769*
-0.430
0.096
10.154
0.419
(Layer
BERT Sheet-base-uncaser
0.931
0.090
0..14
0.937'
0.783'
0.858*
0.620
+ Conceptor-12 (or)
0.388
-0.078
-02922
0.179
0.594
0.335
10.309
0.311
+
-0.334
-0.117
0.459
-0.230
0.178
10.284
0.336
+ CDA
0.846
0.186
002778
T.342
0.831
0.8499
10.120"
0.722
+ DROPOUT
1.136*
0.317
0.138
1.179*
0,879*
0.939*
10.144
0.765
+ INLP
0.317
-0.354
-0.258
0.105
0.187
-0.004
10.416
0.204
+ SENTENCEDEBIAS
0.350
-0.298
-0.626
0.458
0.413
0.462
10.186
0.434
Table 12: SEAT fffect size of gender debising from CI-BERT, Type I. The conceptor-intervenne performance of
different layer's 's conceptors on SST Corpus, port-base-uncased models. The setting is "brown-0 The layer(s) CI-BERT that outperform the conceptor post-processing same layer(s) are colored orange.

---

Model
SEAT-6
SEAT-6b
SEAT-7
SEAT-7b
SEAT-8
SEAT-8b
Avg. Abs.
(Layer 0)
BERT ("bert-base-ure uncased))
0.921
0.194
0.251
-0.122
-0.110
0.366
0.336
+ Conceptor-O (extended)
0.497
-0.095
-0.412
-0.760
-0,001
-0.276
10.004
0.340
+ Conceptor-Intervee Intervennd
0.497
-0.095
-0.412
-0.760
-0.001
-0.276
10.004
0.340
(Layer I)
BERT "berrtbhase-uncseed
1.245*
0.292
0.469
1.101"
0.110
1,261"
0.746
+ Conceptor-1 (extended)
0.897"
0.156
-0.084
0.208
0.099
0,558
10.412
0.334
+ Conceptor-Intervened
0.813"
0.029
-0.961
-0,513
-0.211
-0.292
10,276
0.470
(Layer
BERT ("bert- "hertt-has--unesedd")
1.149
0.216
0.431
1.021
0.474
1,231
0.754
Conceptor-2 (extended)
0.542
0.059
-0.146
0.112
0,515
0.428
10.454
0.300
Conceptor-Intervenne
0.366
0.088
-.342
-0.249
-0,408
-0.634
10.239
0.515
(]ayer3)
BERT "'bert-bas-unccsedd")
1.186*
0.214
0.152
0.770
0,262
1.049
0.606
+ Conceptor-3 (extended)
0.849
-0.034
-0.516
0.154
0.205
0.356
10.254
0.352
+ Conceptor-Intervenne
0.329
0.048
-1.240
-0,520
-04299
-0.499
10.095
0.511
()ayer
BERT "bhrrtbase-unccssdd")
0.975
0.106
0.552
0.890
0.542
0.724
0.632
+ Conceptor-4 (extended)
0.789
0.109
-0.014
0.254
0.515
0.009
J0.350
0.282
+ Conceptor-Intervent
0.248
0.068
-13667
-0.040
-0.401
-0.100
10,261
0.371
(Layer
BERT "'bert-base-uncssadf
1.002*
0.184
0.628
0.914
0.376
1.053*
0.693
+ Conceptor-5 (extended)
0.695
0.122
-0.007
0.075
0.158
0.472
J0.438
0.255
+ Sonceetor-Inneeveene
0.105
0.066
-1099
-0.170
-0,173
0.036
10.419
0.274
(Läyer
BERT ("bert-base-uneessed")
0.753*
0.118
0.539
1,048
0.597
1.042*
0.683
+ Conceptor-6 (extended)
0.372
0.084
0.150
0.033
0.467
0.209
J0.464
0.219
Conneptor-Intervenee []ayer?)
0.004
-0.023
0.866
-0.312
-0,234
-0.106
10.425
0.258
BERT ("bert-base-uncusedd"
0.719
0.155
0.341
0.935
0.562
0.721
0.572
Conceptor-7 (extended)
0.451
0.082
0.051
0.196
0.326
0.185
J0.357
0.215
Conceptor-Intervened
0.041
-0.066
-0.697
-0.509
-0.381
-0.015
J0.287
0.285
(Layer 8)
BERT "bert-base-uneused")
0.983
0.163
0.313
1.157"
0.766
0.789*
0.695
+ Conceptor-8 (extended)
0.597
0.051
-0.023
0.639
0.503'
0.200
J0.359
0.336
+ Conceptor-Intervent
0.110
-0.095
-0.392
-0.702
-0,287
10.399
0.296
(Layer 9)
BERT bbett-bas-uuncueed")
0.922
0.224
0.503"
1,293"
0.780
0.996
0.786
+ Conceptor-9 (extended)
0.397
0.146
0.333
0.903
0.764
0.722
J0.208
0.578
+ Sanceptor-Intervened
0.148
-0.024
0.162
-0.669
0.224
0.487
10.500
0.286
(Layer 10)
BERT "bert-base-unncseed"
0.686
0.082
0.226
0.894
0.904
0.965
0.626
+ Conceptor- 10 (extended)
0.639
0.099
-0.034
0.044
0.605
0.322
10.335
0.291
+ Conceptor-Intervenne
0.557
-0.165
-0.149
-1,046
0.142
0.522
10.196
0.430
(Layer 11)
BERT ("bert- ("bert-base-uncee base-uncased")
0.
0.665*
00015
-0.44
0.602"
0.919*
0.891
0.573
Conceptor-11 (extended)
0.565
-0.45
-0.511
-0.406
0.523
0.198
10.198
0.375
Conceptor-Intervenn
0.602'
-0.199
0.143
-1,219
-0.006
0.205
10.179
0.394
(Layer
BERT ("bert-base-unes uncasdd"
0.931
0.090
-0.124
0.937'
0.783'
0.858*
0.620
+ Conceptor-12 (extended)
0.688
-0.293
0.138
0.559*
0.375
10.274
0.346
+ Contepeor-Interveeeed
0.384
-0.261
0.144
-1,256
-0.148
0.398
10.188
0.432
+ CDA
0.846
0.186
-02278
T.342
0.831
0.849"
10.120"
0.722
+ DROPOUT
1.136*
0.317
0.138
1.179*
0,879*
0.939*
10.144
0.765
+ INLP
0.317
-0.354
-0.258
0.105
0.187
-0.004
10.416
0.204
+ SENTENCEDEBIAS
0.350
-0.298
-0.626
0.458
0.413
0.462
10.186
0.434
Table 13: SEAT effect size of gender debising from CI-BERT, Type T. The conceptor-intervened performance
of different layer's conceptors on SST Corpus, layer(s) CI-BERT that outperform the conceptor post-processing same layer(s) are colored orange.
models. The setting is "sst-0.9 1att-..--xttee D-exxended". The

---

Model
CoLA
MNLI
MRPC
QNLI
QQP
RIE
SST
STS-B
WNLI
Average
BERT
$5.89
84.50
88.59
91.38
91.03
63.54
92.58
88.51
43,66
77.74
Conceptor P.P
57.54
84.66
89.30
91.03
91.05
65.34
92.66
89.07
54.93
11.77
79.51
Conceptor C.T.
47.06
83.46
87.20
90.73
90.97
58.98
91.67
88.21
$2.11
J1.03
76.71
CDA
55.90
84.73
88.76
91.36
91.01
66.31
92.43
89.14
38.03
.0.222
77.52
DROPOUT
49.83
84.67
88.20
91.27
90.36
64.02
92.58
88.47
37.09
J1.46
76.28
INLP
56.06
84.81
88.61
91.34
90.92
64.98
92.51
88.70
32.86
.0.99
76.76
SENTENCEDEBIAS
56.41
84.80
88.70
91.48
90.98
63.06
92.32
88.45
44.13
10.07
77.81
Table 14: GLUE validation set results for gender debiased BERT model. We use the F1 score for MRPC, the Spearman correlation for STS-B, and Matthew's correlation for CoL.A. For all other tasks, report accuracy. All scores are averaged among three runs. The model is "hert-base-uneased" The top-3 best performance is colored in orange.
Model
SEAT-6
SEAT-6b 0.797
SEAF-7
SEAT-7b
SEAT-8 0.837
SEAT-8b 1,293
Avg. Abs.
BERT-T
1,735"
1.294
1.243"
1.200
Pencentlee 1.0)
Conceptor-2 (pronouns)
1.657
0.449
1,185'
0.936
0.453
0.833'
J0.281
0.919
Conceptor-2 (extended)
1,570
0.353
1.094
0.991
0.176
0.775'
J0.373
0.827
Conceptor-2 (propernouns)
1.641
0.655
1.142"
1.121"
0.203
0.781
J0.276
0.924
Conceptor-2 (all)
1,587
0.377
1.188"
1.077'
0.128
0.735'
J0.351
0.849
Conceptor-2 (or)
1.464"
0.257
1.005"
0.944
-0.114
0,503"
10,486
0.714
(Wordlist Percentile 0.5-0.9)
+ Conceptor-2 (pronouss)
1.657*
0.449
1.185"
0.936
0.453
0.833*
J0.281
0.919
2 (extended)
1.296
0.255
1.014"
1.194"
-0.274
0.502*
[0.444
0.756
Conceptor-2 (propernouns)
1.641*
0.655"
1.142"
1.121"
0.203
0.781
[0.276
0.924
Conceptor-2 (all)
1.587
0.377
1,188"
1.077"
0.128
0.735'
[0.35]
0.849
Conceptor-2 (or)
1.323"
0.186
0.947
1.027"
-0.289
0,403
[0,504
0.696
(Wordlist Percentile 0.4)
Conceptor-2 (pronouns)
1.657*
0.443"
1,181"
0.937"
0.448"
0.821
J0.286
0.914
Conceptor-2 (extended)
1.294
0.254
1.014"
1.194"
-0.274
0.502"
J0.445
0.755
Conceptor-2 (propernouns)
1.589*
0.722"
1.130'
1.084"
0.494
0.991'
[0.198
1.002
Conceptor-2 (all)
1.585
0.396
1.192"
1.067'
0.159
0.726
[0.346
0.854
Conceptor-2 (or)
1.278
0.233
0.852
0.910
-0.265
0.346
J0.553
0.647
(Wordlist Percentile 0.3)
Conceptor-2 (pronouns)
1.691*
0,573"
1.227'
1.158"
0.573"
1.009"
J0.161
1.039
Conceptor-2 (extended)
1.295*
0.260
1.010"
1.192"
-0.286
0.490'
[0.444
0.756
Conceptor-2 (propernouns)
1.597'
0.746
1.162"
1.147"
0.551"
1.034"
[0.160
1.040
Conceptor-2 (all)
1,536'
0.436
1.143'
1.181'
0.140
0.849'
J0.319
0.881
Conceptor-2 (or)
1.277*
0,235
-0.090
0.542*
J0.472
0.728
(Wordlist Percentile 0.2)
Conceptor-2 (pronouns)
1.656*
0.543
1.253"
1.175'
0.569*
1.038"
J0.161
1.039
Conceptor-2 (extended)
1.296'
0.260
1.011'
1.189
-0.290
0.478'
[0.446
0.754
Conceptor-2 (propernouns)
1,577"
0.723
1.231'
1.193"
0.541
1.067'
[0.145
1.055
Conceptor-2 (all)
1.490'
0.341
1.116'
1.182'
-0.018
0.849*
J0.367
0.833
Conceptor-2 (or)
1.178
0.141
1.022*
1.887'
-0.252
0.502*
[0.496
0.704
(Wordlist Percentile 0.1)
Conceptor-2 (pronouns)
1.677'
0.643"
1.317"
1.342"
0.696
1.191
10.056
1.144
Conceptor-2 (extended)
1.547'
0.700
1.305"
1.333'
0.464
0.997
J0.142
1.058
Conceptor-2 (propernouns)
1.722*
0.836"
1.256
1.213"
0.956*
1.316*
10.017
1.217
Conceptor-2 (all)
1.771*
0.882"
1.189
1.160'
0.996
1,277'
10.012
1.212
2 (or)
1.579*
0.560
1.278
1.301
0.422
0.881
J0.196
1.004
Table 15: SEAT effect size gender debising. The impact of different percentiles of wordlist (using UMAP clustering) Brown Corpus, bert-tiny models. The top-3 best performance is colored in orange.

---

Model
SEAT-6
SEAT-6b
SEAT-7
SEAT-7b
SEAT-8
SEAT-8b 1,293
Avg.
BERT-T
1,735"
0.797
1,994
1.243"
0.837
1.200
(Wordlist Percentile 1.0)
+ Conceptor-2 (pronouns)
1.703"
0.403
0.958
0.706
0.254
0.679
J0.416
0.784
2onceptor-2 (extended)
1.608
0.473
0.870
1.118"
-0.209
0.732"
[0.365
0.835
Conceptor-2 (propernoums)
1.704"
0.582
1.012"
1.111"
-0.069
0.730'
10.332
0.868
2 (all)
1.680'
0.377
1.028'
1.047
-0.175
0.669
[0.371
0.829
Conceptor-2 (or)
1.489
0.163
0.539*
0.937
-0.612
0.314
J0.524
0.676
(Wordlist Percentile 0.5-0.9)
Conceptor-2 (pronouns)
1.703"
0.403
0.958
0.706
0.254
0,679
J0.416
0.784
Conceptor-2 (extended)
1.647*
0.391
0.806
0.815"
-0.136
0.637'
[0.461
0.739
Conceptor-2 (propernouns)
1.704"
0.582
1.012"
1.111'
-0.069
0.730'
[0.332
0.868
Conceptor-2 (all)
1.680'
0.377
1.028"
1.047'
-0.175
0.669
[0.371
0.829
Conceptor-2
1.542'
0.148
0.486
0.806
-0.549
0.245
J0.571
0.629
(Wordlist Percentile 0.4)
Conceptor-2 (pronouns)
1.703"
0.375
0,999*
0.720'
0.235
0.669*
J0.416
0.784
Conceptor-2 (extended)
1.608
0.473
0.870
1.118'
-0,209
0.732'
[0.365
0.835
Conceptor-2 (propernouns)
1.765'
0.954
1.027'
0.457
1.028'
J0.156
1.044
Conceptor-2 (all)
1.694
0.444
1,057'
1.054'
-0,090
0.709'
J0.359
0.841
Conceptor-2 (or)
1,587
0.344
0.881
0.489*
J0.497
0.703
(Wordlist Percentile 0.3)
Conceptor-2 (pronouns)
1.786"
0.843"
1.121"
1.028"
0.617"
1.064"
J0.124
1.076
Conceptor-2 (extended)
1.610'
0.479
0.866
1.119"
-0.215
0.727'
[0.364
0.836
Conceptor-2 (propernouns)
1.749*
0.945"
1.039"
1.067'
0.474
1,041'
[0.148
1.052
Conceptor-2 (all)
1.751*
0.813"
1.102"
1.063'
0.522
1.092'
(0.143
1.057
Conceptor-2 (or)
1.652*
0.414
0.784
1.047"
-0.170
0.725
[0.401
0.799
(Wordlist Percentile 0.2)
Conceptor-2 (pronouns)
1.862"
0.983"
1.178"
0.982
0.737'
1.080"
J0.063
1.137
2 (extended)
1.610'
0.479'
0.866
1.119'
-0.215
0.727
J0.364
0.836
Conceptor-2 (propernouns)
1.751*
0.893"
1.082.
1.125
0.581
1.167'
10.100
1.100
Conceptor-2 (all)
1.773*
0.862"
1.101
1.120'
0.628
1.191
10.088
1.112
Conceptor-2 (or)
1.736
0,582
0.702
0.946
-0.103
0.763'
10.395
0.805
(Wordfist Percentile 0.I)
Conceptor-2 (pronouns)
1.828*
0.971"
1.185'
1.065'
0.755
1,123'
J0.046
1.154
Conceptor-2 (extended)
1.638
0.511
1.167"
I.114"
0.265
1.017'
J0.248
0.952
Conceptor-2 (propernouns)
1.777*
0.941"
1.121
1.169"
0.885
1.332"
10.004
1.204
Conceptor-2 (all)
1.795*
0.952"
1.070*
1.129
0.785
1.269*
J0.033
1.167
Conceptor-2 (or)
1.706*
0.726
0.990
0.972
0.455
0.978
10.229
0.971
Table 16: SEAT effect size of gender debising. The impact of different percentiles of wordlist (using UMAP clustering) on SST Corpus, bert-riny models. The top-3 best performance colored in orange.

---

Model
SEAT-6
SEAT-ob
SEAT-7
SEAT-7b
SEAT-8
SEAT-8D
Avg. Abs.
BERT-
1,735'
0.797
1.294
1.243
0.837
1,293
1.200
(Wordlist Percentile 1.0)
Conceptor-2 (pronouns)
1.676'
0.389
1.218
1.095"
0.557
1.008"
10.210
0.990
2onceptor-2 (extended)
1.578'
0.507
1.248"
1.220'
0.656
1.351"
J0.107
1.093
Conceptor-2 (propernouns)
1.713"
0.776
1.184"
1.315"
0.538
1.193'
[0,080
1.120
Conceptor-2 (all)
1.660'
0.379
1.248'
1.185'
0.486
1.125'
[0.186
1.014
Conceptor-2 (or)
1.550'
0.180
1.000'
1.146
0.197
1.088
[0.338
0.862
(Wordlist Percentile 0.5-0.9)
Conceptor-2 (pronouns)
1.676'
0.389
1.218"
1.095"
0.557
1.008"
[0,210
0.990
Conceptor-2 (extended)
1.684"
0.374
1.204'
1.065"
0.616
10.186
1.014
Conceptor-2 (propernouns)
1.713'
0.776
1.184*
1.315*
0.538
1.193'
[0,080
1.120
2onceptor-2 (all)
1.660'
0.379
1.185'
0.486
J0.186
1.014
2 (or)
1,573
0.963
1.117
0.103
0.963
J0.384
0.816
(Wordlist Percentile 0.4)
Conceptor-2 (pronouss)
1.677'
0.382
1.218
1.095"
0.561
1.008"
10.210
0.990
Conceptor-2 (cxtended)
1.578"
0.507
1.248"
1.220'
0.656
1.351"
[0.107
1.093
Conceptor-2 (propernouns)
1.622*
0.745
0.946
0.937
0.717
1.104
[0.188
1.012
Conceptor-2 (all)
1.656'
0.388
1.198"
1.114"
0.490
1.083'
J0.212
0.988
+ 2onceptor-2 (or)
1,544
0.270
0.899
1.034"
0.497
1,033'
L0,316
0.884
WWordfistPennenttil 0.3)
Conceptor-2 (pronouns)
1,554
0.447
1.116'
1.198
0.973
1.429*
10.080
1.120
Conceptor-2 (extended)
1,584
0.523"
1.248
1.223'
0.636
1.345'
J0.107
1.093
Conceptor-2 (propernoums)
1.648*
0.764
1.087
1.134
1.084
1.349*
J0.022
1.178
Conceptor-2 (all)
1,588
0.783"
1.125'
1.132"
0.989
1.325*
J0.043
1.157
Conceptor-2 (or)
1,549
0.402
1.236
1.186'
1,024"
1,430"
J0,062
1.138
(Wordlist Percentile 0.2)
Conceptor-2 (pronouns)
1.653*
0.588
1.124
1.084
0.863
1.257
J0.105
1.095
Conceptor-2 (extended)
1.584
0.523
1.248
1.223'
0.636
1.345"
J0.107
1.093
2onneptor-2 (propernouns)
1.623*
0.753"
1.184"
1.224
1.001
1.336'
J0.013
1.187
Conceptor-2 (all)
1.600
0.747"
1.190"
1.195"
0.986
1.314
J0.028
1.172
2 (or)
1.476'
0.364
1.089"
1.026"
0.407
1.162"
J0,279
0.921
(Wordlist Percentile T.T)
Conceptor-2 (pronouns)
1.668*
0.775
1.154
0.989
0.813
J0.105
1.095
Conceptor-2 (extended)
1.689*
0.762
1.345"
1.206
0.991
1.260"
10.009
1.209
Conceptor-2 (propernouns)
1.705*
0.863*
1.257
1.214
0.812
1.282
J0.011
1.189
Conceptor-2 (all)
1.709"
0.870"
1.224
1.203"
0.858
1.287'
10.008
1.192
Conceptor-2 (or)
1.630"
0.739
1.069
0.928
0.753
1.152"
[0.155
1.045
Table 17: SEAT effect size of gender debising. The impact of different percentiles of wordlist (using UMAP clustering) on Reddit Corpus, bert-tiny models. The top-3 best performance is colored in orange.
Model
SEAT-6
SEAT-66
SEAT-7
SEAT-7b
SEAT-8
SEAT-Sh
Avg. Abs.
(Layer 0)
BERT-T
1.536'
0.640*
0.959*
1.307
0.263
0.814
0.920
Conceptor (or)
0.803*
0.103
0.249
0.825
0.039
0.568
10.489
0.431
Conceptor-
centepper-Interveened Intervened
0.803*
0.103
0.249
0.825
0.039
0.568
10.489
0.431
(Layer 1)
BERT-T
1.702*
1.019*
1.102*
1.250"
0.313
1.094
1.080
Conceptor-1 (or)
1.241
-0.067
0.588
0.939
-0.340
0.477*
J0.471
0.609
conceptor-Intervence
0.928
0.022
-0.427
0.708
-0.753
0.542"
J0.517
0.563
2)
BERT-T
1,735*
0.797*
1.294
1.243
0.837
1.293*
1.200
2 (or)
1.542*
0.148
0.486
0.806
-0.549
0.245
10.571
0.629
Conceptor- -Coneeptor-nntevvend Intervened
1.026
-0.079
-0.264
0.862
-0.500
0.239
J0.705
0.495
Table 18: SEAT effect size of gender debising from C1- CI-BERT, Type I. The conceptor-intervence performance of different layer's conceptor matrix on SST Corpus, bert-tiny models. The layer(s) of CI-BERT that outperform the conceptor post-processing of the same layer(s) are colored in orange.

---

Model
SEAT-6
SEAT-6b
SEAT-
SEAT-7b
SEAT-8
SEAT-88 8b
Avg. Abs.
GPT2
-0,5T0
0.057
-0.274
10.186
-0.369
-0.313
0.285
(Wordfist Percentile 1.0)
Conceptor- 12 (pronouns)
0.030
0.269
-0.137
0.044
-0.129
-0.076
10.171
0.114
Conceptor-12 (extended)
0.053
0.293
-0.163
0.057
-0.114
-0.054
J0.163
0.122
Conceptor-12 (propernouns)
0.713
0.430
0.029
0.222
0.045
0.196
J0.013
0.272
1onceptor-12 (all)
0.443
0.333
-0.107
0.102
-0,088
0.065
J0.095
0.190
Conceptor- (or)
0.494
0.269
0.065
0.236
-0.044
0.308
J0.049
0.236
WWordfist Percentile 0.9)
+ Conceptor-12 (pronouns)
0.030
0.269
-0.137
0.044
-0.129
-00076
J0.171
0.114
+ Conceptor-12 (extended)
0.312
0.448*
-0.352
0.032
-0.067
0.048
J0.075
0.210
+ Conceptor-12 12 (propernouns)
0.713
0.430"
0.029
0.222
0.045
0.196
J0.013
0.272
+ Conceptor- 12 (all)
0.443"
0.333
-0.107
0.102
-0.088
0.065
J0.095
0.190
+ Conceptor- (or)
0.537"
0.268
0.051
0,260
-0.043
0.361
J0.032
0.253
(Wordlist Percentile 0.8)
+ 12 (pronouns)
0.030
0.269
-0.137
0.044
-0.129
-0.076
J0.171
0.114
+ 1onceptor-12 (extended)
0.312
0.448*
-0.352
0.032
-0,067
0.048
J0.075
0.210
+ Conceptor-12 (propernouns)
0.713
0.430*
0.029
0.222
0.045
0.196
J0.013
0.272
+ Conceptor- 12 (all)
0.443
0.333
-0.107
0.102
-0.088
0.065
10.095
0.190
+ Conceptor-122 12 (or)
0.537"
0.268
0.051
0,260
-0.043
0.361
10.032
0.253
(Wordfist Percentile 0.7)
Conceptor- (pronouns)
0.031
0.269
10.136
0,045
-0.128
-0.076
J0.171
0.114
Conceptor-12 12 (extended)
0.312
0.448*
-0.352
0,032
-0.067
0.048
J0.075
0.210
Conceptor-12 12 (propernouns)
0.713"
0.430"
0.029
0.222
0.045
0.196
J0.013
0.272
Conceptor- 12 (all)
0.443"
0.333
-0.107
0.102
-0,088
0.065
J0.095
0.190
Conceptor- (or)
0.538
0.269
0.051
0.260
-0,043
0.361
10.031
0.254
(Wordiist Percentile 0.6)
+ Conceptor- (pronouns)
0.031
0.269
-0.136
0.045
-0.128
-0.076
J0.171
0.114
+ Conceptor- 12 (extended)
0.304
0.449*
-0.381
-0.002
-0.106
0.016
10.075
0.210
+ Conceptor- 12 (propernouns)
0.713"
0.430'
0.029
0.222
0.045
0.196
J0.013
0.272
+ Conceptor-11 12 (all)
0.443"
0.333
-0.107
0.102
-0,088
0.065
[0.095
0.190
+ Conceptor-122 (or)
0.519
0,255
0.036
0.244
-0.554
0.348
10.042
0.243
(Wordlist Percentíle 0.3)
+ Conceptor- 12 (pronouns)
0.031
0,269
-0.136
0.045
-0.128
.00766
J0.171
0.114
+ Conceptor- 12 (extended)
0,269
0,411"
-0.258
-0.038
0.114
0.027
J0.099
0.186
+ Conceptor- 12 (propernouns)
0.713"
0,430"
0.029
0,222
0.045
0.196
J0.013
0.272
+ Conceptor-12 12 (all)
0,565*
0.445*
-0.110
0.125
-0.063
0.127
J0.046
0.239
+ Conceptor-12 (or)
0.478"
0.248
0.029
0.188
-0,016
0.317
J0.072
0.213
(Wordlist Percentile 0.4)
+ Conceptor- 12 (pronouns)
0.060
0.271
-0.130
0.045
-0.131
-0.074
J0.167
0.118
+ Conceptor-12 (extended)
0.276
0.410"
-0.255
-0,038
0.110
0.027
J0.099
0.186
+ Conceptor-12 (propernouns)
0.730"
0.393'
0.033
0.190
0.028
0.127
J0.035
0.250
+ 1onceptor-12 (all)
0.648
0.374
-0.036
0.154
0.014
0.134
10.058
0.227
+ Conceptor-12 (or)
0.564
0.258
0.057
0.196
0.014
0.287
10.056
0.229
(Wordfist Percentile 0.3)
Conceptor-12 12 (pronouns)
0.092
0.316
-0.001
0.064
-0.035
-0.062
J0.190
0.095
Conceptor-12 (extended)
0.264
0.369
-0.261
-0,043
0.115
0.015
J0.107
0.178
Conceptor-12 (propernouns)
0.729*
0.391*
0.030
0.187
0.028
0.125
J0.037
0.248
1o (all)
0.682
0.397*
-0.021
0.162
0.018
0.140
J0.048
0.237
Conceptor- (or)
0.545
0.214
0.072
0.198
0.006
0.279
J0.066
0.219
WWordfist Percentile 0.2)
+ Conceptor- (pronouns)
0.094
0.316
-0.001
0.064
-0.033
-0,622
J0.190
0.095
+ Conceptor-12 (extended)
0.249
0.356
-0.238
-0,048
0.016
J0.116
+ Conceptor- 12 (propernouns)
0.699
0.403*
0.042
0.176
0.017
0.092
J0.047
0.238
+ Conceptor- 12 (all)
0.699
0.437*
0.056
0.164
0.063
0.098
J0.032
0.253
+ Conceptor- (or)
0.519
0.223
0.086
0.191
-0.004
0.255
J0.072
0.213
(Wordfist Percentile 0.T)
+ 12 (pronouns)
0.446
0.599*
0.035
0.064
0.064
-0.037
J0.077
0.208
+ 1o (extended)
0.497
0.544*
0.019
0.097
0.171
0.074
10.051
0.234
+ Conceptor-12 12 (propernouns)
0.753*
0.495*
-0.038
0.095
-0.010
0.062
J0.043
0.242
+ Conceptor- 12 (all)
0.730"
0,508*
-0,009
0.086
0.102
0.070
10.034
0.251
+ Conceptor- 12 (or)
0.914
0.568*
0.258
0.247
0.021
0.264
10.094
0.379
Table 19: SEAT effect size gender debising. The impact of different percentiles of wordlist (using UMAP clustering) Brown Corpus, gpt-2 models. The top-3 best performance is colored in orange.

---

Model
CoLA
MNLI
MRPC
QNLI
QQP
RIE
SST
STS-B
WNLI
Average
BERT-L
62.82
86.13
88.32
92.15
91.56
69.31
93.81
90.00
35.21
78.81
Conceptor P.P.
62.34
86.16
89.44
91.71
91.59
74.73
93.58
90.06
30.17
10.05
78.86
GPT2
29.10
82.43
84.51
87.71
89.18
64.74
91.97
84.26
43.19
73.01
+ Conceptor
35.47
82.39
84.08
88.30
67.15
92.09
83.67
33.80
J0.11
72.90
GPT2-L
12.48
88.80
82.98
80.30
87.61
51.26
81.77
78.87
46.48
75.84
+ Conceptor
20.03
88.92
79.64
87.65
50.54
82.34
78.26
10.04
GPT-J
59.73
82.49
87.95
87.93
87.54
75.81
94.50
88.60
39.44
78.22
+ Conceptor 59.48
82.89
87.69
91.56
89.90
76.17
95.07
88.81
30.99
J0.14 78.06
Table 20: GLUE validation results for other LLMs. We use the FI score for MRPC, the Spearman correlation for -nnd Matthew's 's correlation for CoLA. For all other tasks,we we report the accuracy.
Model
SEAT-6
SEAT-6b
SEAT-7
SEAT-7b
SEAT-8
SEAT-8h
Avg. Abs.
BERT
0.931
0.090
-0.124
0.937
0.783
0.858
0.620
Gender Conceptor
0.388
-0.078
-0.292
0.179
0.594'
0.335
J0.309
0.311
Intersected Conceptor
0.916
0.026
-0.180
0.840
0.749*
0.832
J0.029
0.591
Table 21: BERT intersectional gender debiasing, where intersected conceptor indicates the conceptor matrix generated by its negated AND operation of gender conceptor matrix race conceptor matrix
Model
ABW-1
ABW-2
SEAT-3
SEAT-3b
SEAT
SEAT-5
SEAT-Sb
Avg. Abs.
BERT
-0.079
0.690
0.778
0.469*
0.901
0.887'
0.539*
0.620
+
Race Conceptor
-0.063
0.682"
0.803"
0.209
0.949
0.946
0.390
40.043
0.577
+
Intersected Conceptor
-0,045
0.685
0.799*
0.361
0.926
0.931*
0.484
10.016
0.604
Table 22: BERT intersectional race debiasing, where intersected conceptor indicates the conceptor matrix generated by its negated AND operation of gender conceptor matrix and race conceptor matrix


Reviews:
[{"review_id": "r08irA5FDv", "reviewer": "Reviewer_ZbQ9", "summary": "", "questions": "See reasons to reject\n1. What is the difference between Conceptor P.P. (default) and Conceptor P.P. (tuned) in table 1? Did not find any explanation on this.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and valuable comments.\n\nRE Reject #1: The conceptors post-processing method is shown by Table 1 that it can be generalizable to different types and scales of language models (LMs). The post-processing technique is cheap since it doesn’t require any retraining nor finetuning, and can be plugged-in anywhere. Furthermore, we have two settings, as mentioned in line 444-447 and 466-468: (1) *default*: the LM uses the default pipeline setting—Brown corpus, no outlier filtering, and the AND subspace. (2) *tuned*: the LM adopts the strongest pipeline setting after grid-searching different corpora, wordlist percentiles, and subspaces. Per Table 1, even if we use the *default* setting, the conceptor post-processing method can already stably lower the bais. The conceptor continued-training method--which leads to lower bias than the post-processing method--is indeed more computationally expensive. However, the continued training step only requires hundreds of training steps. For example, in the case of bert-base-uncased, it takes less than a half hour on the NVIDIA V100 GPU on Colab. That is, the users can use either the cheap post-processing method under the default setting to get a good debiasing result, or a slightly more expensive (but still the cost is reasonably priced) continued-training method if they want to debias further. \n\nRE Reject #2: Different bias subspaces do lead to different *magnitude* of debiasing performance, but almost all of them debias *successfully*. As said in line 165-168 of the paper, even different choices of bias attribute wordlist would impact the debiasing, and commonly exist in lots of debiasing methods [1]. Therefore, our paper specifically shows the robustness of conceptor methods. For example, the conceptor methods can successfully debias in almost any case no matter which bias subspace, corpora, or wordlist are chosen (Tables 9-11, highlighted in green). In addition, we compute that the overlap of words between SEAT benchmark and the wordlist used for subspace constructed is 0.64%, similar to that of the ‘unseen’ test setting.\n\nRE Q1: See the first response. We will make the definition of *default* and *tuned* more obvious in the table in the camera-ready version of our paper.\n\n---\n[1] [Bad Seeds: Evaluating Lexical Methods for Bias Measurement](https://aclanthology.org/2021.acl-long.148) (Antoniak & Mimno, ACL-IJCNLP 2021)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. State-of-the-Art De-Biasing performance\n2. Bias subspace projection improves the debiasing performance while maintaining the models’ accuracy.\n3. The introduction of a new architecture (CI-BERT) that incorporates bias reduction during training shows an innovative and proactive approach to the problem, potentially leading to more effective bias reduction in the long term.\n4. Addressing biases in language models is an ethical imperative.", "reasons_to_reject": "1.  The conceptors method and CI-BERT architecture could introduce complexity and implementation challenges. If the method requires extensive modifications to existing models or intricate procedures, it might hinder its adoption and integration into real-world applications.\n2. The focus on constructing a specific bias subspace might lead to overfitting, where the method becomes too tailored to the specific training data and fails to generalize well to new or unseen data.", "questions_to_authors": "See reasons to reject\n1. What is the difference between Conceptor P.P. (default) and Conceptor P.P. (tuned) in table 1? Did not find any explanation on this.", "paper_topic_and_main_contributions": "The paper use conceptors to reduce social biases in large language models (LLMs) like BERT and GPT, including two approaches: 1) post-processing to remove bias, maintaining model performance; 2) a new model architecture called CI-BERT that includes bias reduction during training, but with a slight accuracy trade-off. Conceptors effectively handle intersectional bias and maintain robustness. The abstract also highlights the importance of carefully defining the bias subspace for optimal results, involving steps like outlier removal and embedding computation from a cleaner data source.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wKqkMl8HEQ", "reviewer": "Reviewer_ogLa", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 1, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and valuable comments. We will fix the mentioned typo in the camera-ready version.\n\nRE Reject #1: We acknowledge and appreciate your point regarding the scope of the recent LMs being multilingual. However, the primary focus of our work was to shed light on the methodology and robustness of the conceptor debiasing technique, and to explore intersectional debiasing, within the context of English LMs. It's worth noting that the specificity of the study (i.e., focusing on English and North American biases) was purposefully chosen to provide a more in-depth analysis of conceptor debiasing pipeline and intersectional biases in this particular context. Broadening the scope to multilingual models would have made the study much more extensive and might have diluted the depth of analysis we intended to provide. That said, we recognize the potential importance of studying such methods in multilingual contexts, and have indeed mentioned it in the limitations section. We believe this offers a direction for future research, where researchers can expand upon our methodology to a multilingual setting.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well written and the proposed approach appears promising given the improvement performance across multiple LMs and on multiple aspects. ", "reasons_to_reject": "As points out by the authors also, the proposed approach is restricted to only English while all recent LMs are all multilingual. Also, the focus has been on debiasing biases in North America and more extensive study would have been more helpful. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper focuses on the problem of debiasing existing language models for improving soclal fairness of NLP. The authors challenges the existing conclusion that conceptor negation fails to debias BERT stably. They employ conceptor negation post processing to debias LMS such as GPT while retaining useful semantics and robustness. The authors also explore conceptor-intervened BERT (CIBERT) where they continue training BERT after incorporating conceptors with all BERT layers. With different corpora, bias attribute wordlists and outlier removal criteria they observe different debiasing performance. The results demonstrate that conceptor post-processing outperformsn several SOTA debiasing approaches on both debiasing effectiveness and semantic retention. Proposed CI-BERT outperforms post-processing approach. However, with increased instability and worse semantic retention. ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Typos:\n1. In line 529, replace “Bert” with “BERT”\n"}, "extra_text": {}}, {"review_id": "khOf8VwjWp", "reviewer": "Reviewer_cvA6", "summary": "", "questions": "A. You stated that conceptor post-processing achieved SOTA debiasing results in the paper (e.g., lines14-17), but in Table1 , it seems INLP's average absolute SEAT score (0.204) is the lowest. Any comment on this? ", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and valuable comments.\n\nRE Reject - Conceptor mechanism: As mentioned in Appendix A, the word matrix $X$ is the data collection of the embedded attribute words. Via formula 2, we can compute the $C$–-a soft projection matrix on the linear subspace where the word embeddings $X$ have the highest variance. Once $C$ has been learned, it can be ‘negated’ by subtracting it from the identity matrix and then applied to any word embeddings to shrink the bias directions. Please further note that we do not want to compute the conceptor on the “diff space” as the projection matrix is applied to the original space. In this way the conceptor method is different from the so-called hard-debiasing of Bolukbasi et al. We will add this explanation and move some mechanism introduction from Appendix A to the main body in the camera-ready version.\n\nRE Reject - section 3.2: We find the conceptor matrices by the original BERT’s weights, then use them to continually train the BERT model in a reasonable amount of iterations. We don’t recalculate the word vectors and the conceptor matrix after each iteration, thus it is not computationally expensive.\n\nRE Reject - layout: The main results (e.g. tuned conceptor result, robustness and extendability) are self-contained in the main body (e.g. Table 1-5), and Table 8-11 are the extension of the tuned result in Table 1, and the further proof of the robustness and extendability. However, we understand the confusion of the crowded contents and will adjust the layout when we have an additional page in the camera-ready version, following your suggestion!\n\nRE Question A: What we said is that the conceptor method achieves SoTA debiasing performance *while maintaining the semantic meaning* (line 14-17). As shown in Table 5, INLP drops the GLUE of BERT by 0.99 [1], which means INLP significantly harms the language capacity. In this case, it is suspicious that INLP successfully debiases since the damage of language capacity can also manifest as the decrease of bias in the fairness benchmark. A reductio ad absurdum would be to zero out or randomize all embeddings. This is perfect debiasing; however, has little to no utility. Thus it is critical to find  bias mitigation  methods that do not impair the language capacity in the final target application.\n\n---\n[1] [An Empirical Survey of the Effectiveness of Debiasing Techniques for Pre-trained Language Models](https://aclanthology.org/2022.acl-long.132) (Meade et al., ACL 2022)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors proposed two novel methods based on conceptors: one post-processing-based method and one continuing training-based method. Both of them debias the representations in every layer of a transformer and seem to be effective in mitigating bias.\n\nIt is novel to use the conceptors' logical semantics (AND and OR) to study the effect of combining different bias attribute word lists, and mitigating intersectional bias. The problems themselves are also of research importance.\n\nThe authors conducted extensive experiments and provide many insights on the practice of bias mitigation, such as how to combine different word lists and filter out outliers, the impact of different debiasing corpora, their methods' performance on different kinds and sizes of language models, etc.", "reasons_to_reject": "I think the authors left out some details for their Algorithm 1 (at least for me). In the original HardDebias [Bolukbasi, etal, 2016][Liang, etal, 2020], the bias subspace is created by the first n (n=1 or 2) principle components of the set of difference vectors between each pair of bias attribute words. But in algorithm 1, it seems the authors simply stack all the word vectors in matrix X, not the difference vectors. I'm wondering why this can mitigate bias. Perhaps the authors could provide more details on the application of conceptors for debiasing, in the related works section or background section.\n\nI'm also a bit confused with the \"Conceptor Intervention and Continued Training\" method in section 3.2. The authors may want to apply the conceptors to BERT's weights before calculating the loss function and the gradients. But since the word vectors will change after BERT's weights have changed, I'm not sure if the authors re-calculate all the word vectors and the conceptor matrix after every update step of BERT's weight? If so, the method may be computationally expensive.\n\nSome of the main results (such as Table 8-11) are in the appendix. The authors may re-organize the paper and include at least part of these results in the main text for better presentation.", "questions_to_authors": "A. You stated that conceptor post-processing achieved SOTA debiasing results in the paper (e.g., lines14-17), but in Table1 , it seems INLP's average absolute SEAT score (0.204) is the lowest. Any comment on this? ", "paper_topic_and_main_contributions": "This paper propose to use conceptor to mitigate bias in language models, specifically, one post-processing-based method and one continuing training-based method. The authors cleverly use the conceptors' logical semantics (AND and OR) to study the effect of combining different bias attribute word lists, and mitigating intersectional bias. The authors also conducted comprehensive experiments to study the impact of different debiasing corpora, bias word lists and their combination and filtering methods, the proposed conceptor methods' performance on different kinds and sizes of language models, as well as the impact on language understanding.  ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper delineates an approach that employs conceptors for the purpose of debiasing Large Language Models (LLMs). The paper introduces a novel architecture capable of diminishing bias while simultaneously preserving high accuracy. The architecture is characterized as cost-effective and potentially integrable into various architectures, both of which are highly desirable attributes for debiasing methods.

All reviewers unanimously concurred on the novelty of the proposed methods, as well as the rigor and the favorable outcomes of the experiments.

I urge the authors to refine the paper, taking into careful consideration the feedback provided by all the reviewers, with a particular emphasis on addressing the insights from Reviewer 3.
Decision: Accept-Main