Collapsed Language Models Promote Fairness
Authors: Jingxuan Xu
Wuyang Chen
Linyi Li
Yao Zhao
Yunchao Wei
Venue: ICLR 2025 Poster (2025)
Keywords: Neural Collapse
Fairness
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: To mitigate societal biases implicitly encoded in recent successful pretrained language models, a diverse array of approaches have been proposed to encourage model fairness, focusing on prompting, data augmentation, regularized fine-tuning, and more. Despite the development, it is nontrivial to reach a principled understanding of fairness and an effective algorithm that can consistently debias language models. In this work, by rigorous evaluations of Neural Collapse -- a learning phenomenon happen in last-layer representations and classifiers in deep networks -- on fairness-related words, we find that debiased language models exhibit collapsed alignment between token representations and word embeddings. More importantly, this observation inspires us to design a principled fine-tuning method that can effectively improve fairness in a wide range of debiasing methods, while still preserving the performance of language models on standard natural language understanding tasks. We attach our code at https://github.com/Xujxyang/Fairness-NC-main

Published as a conference paper at ICLR 2025
COLLAPSED LANGUAGE MODELS PROMOTE FAIRNESS
Jingxuan Xu Wuyang Chen?*, Linyi Li', Yao Zhao', Yunchao Wei
Beijing Jiaotong University 2Simon Fraser University
ABSTRACT
To mitigate societal biases implicitly encoded in recent successful pretrained language models, a diverse array of approaches have been proposed to encour- age model fairness, focusing on prompting, data augmentation, regularized fine- tuning, and more. Despite the development, it is nontrivial reach a princi- pled understanding of fairness and an effective algorithm that can consistently debias language models. In this work, by rigorous evaluations of Neural Col- lapse learning phenomenon happen in last-layer representations and classi- fiers in deep networks - on fairness-related words, we find that debiased lan- guage models exhibit collapsed alignment between token representations and word embeddings. More importantly, this observation inspires us to design a princi- pled fine-tuning method that can effectively improve fairness in a wide range of debiasing methods, while still preserving the performance of language mod- els standard natural language understanding tasks. We attach our code com www.///ujwaaggg/aaeeeeeeaeeeeeee -CCmmann
INTRODUCTION
The rise of pre-trained language models (PLMs) has revolutionized natural language processing, greatly enhancing tasks like reasoning and prediction by harnessing the semantic richness of language data. Despite their effectiveness, these models, trained on extensive corpora, often reflect and even intensify societal biases in their training datasets. Such biases manifest in the association of demographic groups with specific roles or capabilities, affecting fairness in applications ranging from legal analytics to hiring processes (Peters et al. 2018 Devlin 2018 Liu 2019 Blodgett et al. 202T Rabelo et al. [022 Bolukbasi et al. 20166; Caliskan et al.) 2017). Thus, is crucial to address and
mitigate these biases to prevent discriminatory practices in downstream applications (Zhao et al.
2019 Webster et al. 2020 Nadeem al. 2020).
To mitigate societal biases in language models, a substantial array of fairness algorithms has been proposed. On the one hand, people target different learning stages: making language models fair via balanced and augmented training data (Bartl et al. (020),, fine-tuning with regularizations or auxiliary objectives (He al., 2022 Park et al. (023). or carefully-tunee prompts (Yang al. 2023). On the other hand, these debiasing approaches can also be categorized by their awareness of downstream tasks. Task-specific methods fine-tune language models with sensitive annotations (Han) ct al. 202lb Shee et al. 2021 Ravfogel et al.) (2022). while task-agnoste approaches directly debias word embeddings or representations during pretraining (Cheng al.) 2021 Kaneko Bollegala 2021; Guo et al. 2022. He et al. (022). Despite this multitude of eftorts, it is challenging to find
common ground among these methods and shared properties of debiased language models. We are thus motivated to ask: Can we understand and improve the fairness of LMs in principle?
The recent development of deep learning theory pro- vides fruitful frameworks and tools for us to understand deep neural networks (DNNs). Among them, neural col- lapse (Papyan et al. 2020) is first observed for classifi- cation tasks, and is then analyzed to understand the opti- mization (Han et al. 2021c| Zhou al. 2022a) and gen- eralization (Hui et al. 2022 Gao et al. 2023) of DNNs. Meanwhile, the training of generative language models,
o00 Blased μ ooe De-bissed l
W;
W2
Wa
Equal contribution Corresponding author
Figure 1: Debiased LMs show more col- lapsed alignment between classifiers (w) and class means (H).. See (U)) Table

---

Published as a conference paper at ICLR 2025
typically via the next token prediction task, is essentially classification problem. We thus study the fairness of LMs through the lens of neural collapse.
We ask two specific questions:
Q1: Do debiased LMs commonly exhibit greater collapse?
Q2: Can we leverage this inductive bias to improve LM fairness principle?
Motivated by these two questions, we try to find connections between neural collapse and fairness of language models. We observed, as expected, that fairer language models show more collapsed representations of sender-sensttive words, indicated by greater alignment between classifiers (word embeddings) and class means (token representations) (see Figure D This behavior is consistently and implicitly exhibited across wide range of popular debiasing methods, suggesting a commonly shared perspective analyzing fairness in language models. This observation further inspires us to explicitly enforce neural collapse to promote fairness in pretrained language models. Since this explicit collapse is principled and does not introduce any customization in fine-tuning data augmentation, it can be universally applied to enhance many existing fairness approaches, with little implementation or computation overhead. We summarize our contributions below:
We for the first time comprehensively analyze the relations between neural collapse and fairness in language models.
Our empirical analysis motivates us to introduce regularization based on neural collapse, which is extremely simple, and agnostic to any fine-tuning method or augmentation of language data.
Comprehensive experiments on both intrinsic and extrinsic evaluations demonstrate that our regularization can consistently debias language models. It orthogonal wide range of highly tailored fairness algorithms, and thus can be plug-and-ple and-play adopted without sacrificing the models' performance on typical downstream language tasks.
RELATED WORKS
2.1
NEURAL COLLAPSE
A learning phenomenon called neural collapse (NC) arises during the terminal phase of training neural networks with cross-entropy (CE) loss for classification tasks (Papyan et al., 2020). was initially defined by the simultaneous emergence of multiple properties in the model's 's top-layer features (also referred to as last-layer representations or embeddings) and classifiers, including the variability and geometry of class-wise averaged features, the alignment between features and classifier weights, and the collapse to nearest-neighbor classifier. Since then, .VC has been theoretically analyzed to understand the optimization (Han et al.) [221cc] Zhou al. 2022a) and generalization (Hui et al.) 2022
Gao et al. 2023) of DNNs. NC inspires numerous applications and empirical studies, including 2024 2023b), data imbalance (Fang et al. 2021 Yang et al.) 2022), and outlier detection (Liu & Qin 2023 Wang et al. 2024b
transfer learning (Galanti et al., 2021), privacy (Li et al.
Meanwhile, DNN behaviors related to ACC are also observed in various settings. Several studies
have explored NCC under various loss functions. For example, Han et al. (2021c) examined WC in the context of Mean Squared Error (MSE) loss, while /Zhou al. (2022b) demonstrated that NCC also arises with label smoothing and focal loss. Additionally, researchers like He & Sul (2023) and Rangamanii al. (2023) have investigated A'C properties in intermediate layers as well. More recently, Wu & Papyan (2024) observed the linearity of A/C in language models and provided some quantitative explanations for this phenomenon.
2.2 FAIRNESS IN LANGUAGE MODELS
Language Models (LMs), extensively studied in academic literature and widely adopted across various applications, have recently raised concerns regarding fairness (Li et al. 2023a Gallegos et al. 2024). For medium-sized LMs, such BERT (Devlin 2018) and Roberta (Liu 2019). faimess elimination methods have been developed from both data and training perspectives. Given that label imbalance across different demographic groups in the training data is significant source of bias, a common data processing technique is to balance labels using Counterfactual Data Augmentation (CDA) (Lu et al. 2020 Zmigrod et al. 2019). Some research bridges robustness and fairness by
2

---

Published as a conference paper at ICLR 2025
augmenting robust training set with techniques such robust word substitution (Pruksachatkun iet al., 2021) and counterfactual logit pairing (Garg et al. 2019). For parameter-efiie effficienn methods, GEEP (Fatemi et al. 2021) and Adept (Yang et al. 2023) incorporated gender equality prompts into
LLMs using trainable embeddings of occupation names. In addition to retraining, FairBERTa (Qian al.
2022 showed that fine-tuning language models on the demographic perturbation dataset PANDA can enhance fairness in downstream tasks, For large LMs, such as Llama (Touvron et al. 2023 and GPT (Brown 2020), recent research not only focuses on fine-tuning the model itself but also emphasizes lightweight pest-processing techniques to address fairness concerns, including instruction ane-tuning (Wei et 2021 Chung al., 2024) and prompt engineering (Bubeck et al.,
2023
Tamkin et al. 2023)
Representative Debiased Language Models. In this paper, we choose to focus on studying the following three representative works, mainly because: 1) They debiased language models from orthogonal perspectives: 2) They all targeted debiasing the BERT model so they can be more fairly compared to each other.
Data Preparation: BEC (Bartl et al. 2020) focused on developing customized Bias Evaluation Corpus (BEC) via counterfactual data substitution, and studied associations between gender- denoting target words and names of professions (Webster et al. (0188).
Fine-tuning Method: Mabel (He et al. 2022) augmented premises and hypotheses from the natural language inference (NI.I) dataset with counterfacts, and applied contrastive learning objective on gender-balanced balanced entailment pairs fine-tune BERT.
Regularization: ASE (Park et al. 2023) proposed incorporating the fairness objective into the training process of downstream tasks through two regularization terms (stereotype neutralization and prevention of catastrophic forgetting) beyond the task objective to encourage the fairness.
DEBIASED LANGUAGE MODELS ARE MORE COLLAPSED
3.1 PRELIMINARY: WHY Do LANGUAGE MODELS COLLAPSE?
Suppose the whole vocabulary is the set of word indices V 21,2.... 2...... CJ. In language models, sequence of tokens (indices) Ti: E yt are embedded by the word embedding layer E and are forwarded through layers. In the context of next-token prediction for language models, the penultimate token representations h EEEa ((x1a)) E Rdd is used to predict the next token .,+1 E V by comparing with classifier weights (w (we E R -= 1,................... ,C} and also the bias term:
ž+ạ1 argmax (w.,, h (E (II=)) + b cEV
(e) We further denote the mean token representation phe Rdd whose ground-truth next token is x We also focus on the centered means:
T-1 He :=== ト h (B(x(1))
(E(ol()))) ( (c)) 1
Σ
ee
(2)
c=1
where No is the number of samples of class I is the (binary) indicator function, S is total number of sentences in the dataset, and T is the total number tokens in a sentence.
We also accumulate the unbiased sample variances:
n (B (x(tt)
(2t1 =c)
N.
Neural collapse (NC) Papyan al. 2020 is originally observed when neural networks are solving classification problems, where the representations from the penultimate layer and the final classifier weights collapse into certain geometric structures.
Neural Collapse in Language Models. From the perspective of token predictions (Eq. D. we can see that the pretraining and fine-tuning of language model are actually classification tasks, and the word embedding layer E {w E Rd 1.. C} is typically the classifier. Therefore, can expect neural collapse also happens in language models, between the mean embedding μ, and the word embedding layer E.
3

---

Published as a conference paper at ICLR 2025
Specifically, inherited from Papyan et al. (2020), the neural collapse behavior in Language Models can also be defined from four perspectives (Wu & Papyan 2024):
NC1, measures the separability between classes via intra-class variability over inter-class distance:
NC1 == Ee,e' o.+a?
f'). A less NC₁ indicates more collapsed classifier.
(g)WC2 measures the separability from the geometric (g) perspective, where the class means tend to become equinorm and equiangular vectors and formulate simplex known as equiangular tight frame (ETF). This can further be relaxed to account for noises and imbalances in practice:
(G)NC2 == Be,er log
Me | Ilp.. ml
1
(Ve + d')- A less (9)NC2 indicates more
expanded and collapsed class means.
(UNCss quantifies the alignment between classifiers and class means, termed as "uniform (U)
we Pe- ||walla" Iltt--lll/ for each class, and then calculate their standard deviations std(-) :) across classes. A less (2))VC3 indicates class means are more consistently correlated with the classifier, i.e., more collapsed.
duality". We first measure
NC4 simplified the linear projection in Eq. [ into nearest-class center (NCC) classifier argmax (w. h)) + b - argmin h Pellz and further measure the token prediction accuracy by eEV
cEV
nearest neighbors. A greater NC₁, indicates stronger (more collapsed) NCC classifier.
3.2 DEBIASED LMS ARE MORE COLLAPSED THAN BIASED LMS
In the context of fairesss of language models, the main idea pursue dehiased word embeddings. For example, people measure the geometry of gender-related tokens in the embedding space (Caliskan al. 2017: May al., 2019; Guo & Caliskan (021) and try to remove gender-related information of stereotypical words and mitigate biases in the word embedding space. To counteract artifacts from training that leads to the encoding of stereotypes, people even adopt static embeddings (Mikolov
et al. 2013 Bolukbasi et al. 2016 Caliskan et al. 2017 Manzini et al. 2019).
Inspired by these previous works, we are expecting to see more collapse between the word embedding layer and token representations in debiased language models.
3.2.1 NC METRICS IN DEBIASED LMs
We first evaluate NC metries in popular debiased language models and compare them with their biased baselines.
Settings. Following previous works, we measure NC metrics based on different training datasets used in each work, concerning the subset of the whole vocabulary of only gender-related words, dubbed V gender (see Appendix [A for the full list of words). The detailed datasets used by these works are listed as follows: Mabel on SNLI (Bowman et al. 2015) and MNLI (Williams et al.) 2017); ASE on OntoNotes (Hovy et al. 2006); BEC TinyStories (Eldan & Li 2023); Among these, only BEC's training dataset (Webster et al. (2018) is relatively small, while the datasets in other works exceed 100K sentences. To ensure meaningful comparisons, we evaluated BEC on larger dataset
TinyStories (Eldan & Li 2023).
We show this result in Table All methods start from the same pretrained BERT model. However, each work studied the BERT model on its own training data, leading to different NC measurements for the same BERT model. From Table we can see that debiased language models exhibit neural col- lapse in certain perspectives: N/3 is consistently improved (minimized) in debiased models, whereas NC1/2 /2/2 are diverging. This indicates that the alignments between token representations ("class means") and debiased word embeddings ("classifier weights") are more consistent, as illustrated in For evaluations of additional models, please refer to Appendix
Figure [
Our explanations are as follows. The neural collapse behavior manifests under certain conditions (Pa- pyan et al. 2020), including: 1) models are trained towards zero training loss; 2) clean labels with balanced classes; 3) the number of classes is not greater than the model's 's hidden dimension. How- ever, these conditions are commonly violated in practice: 1) The training loss is difficult to be minimized to zero due to the complexity of language data; 2) The occurrence of tokens in V is
4

---

Published as a conference paper at ICLR 2025
Table 1: (U)NC3 consistently improved (minimized) on all debiasing methods. Metrics are measured on the subset of the whole vocabulary of only gender-related words V genter (Appendix Ap In these three groups, each pair of "(BERT, debiased model)" is tested on customized dataset as detailed in Section 3.2. leading to different measurements for the same pretrained BERT model.
Model
NC1 +
(gWW22 +
(4))C's +
NC.1
Nc
(gWW))
BERT
0.967
0.148
0.096
2.799
827.8
0.337
MAREL.
0.786 0.181t
0.145 0.003t
0.070
0.0261
2.235 0.664]
724.3 103.5
0.331 0.000t
BERT
2.430
0.051
0.063
1.113
1134.7
0.364
ASE
3.008 0.578)
0,500
0.449)
0.056
0.0071
0.023 1.0004
382.6 762.1
0.372 0.008]
BERT
2.358
0.152
0.062
10.44
1032.7
0.359
BEC
2.509 0.1514
0.185
0.033L
0.056
0.0061
7.442
2 3.9981
1015.6
17.11
0.335
0.0241
highly unbalance due to the nature of languages; 3) Not all language models have greater hidden sizes than the vocabulary size Therefore, token representations and word embeddings are typically balanced and well-trained in practical language models, and thus not all perspectives of neural collapse (NC1/2/33 /2/33/ 4) can be consistently observed in debiased models. Instead, in the next section, we study how to calibrate these metrics to be more consistent with fairness encoded in models.
3.2.2 CALIBRATIONS NC1 AND NC2
When we further analyze Table D to understand why only NC3 is reduced debiased LMs as we expected, we find that only NC3 involves the classifier weights we, but NC1/2// all consider class means pec- We hypothesize that noises in language data and the complexity of different fine-tuning methods make measurements of NC1 /2/4 unstable and inconsistent.
To calibrate these representation-base based collapse metrics, we further study replacing class means with classifier weights in /21/22
Nc( == Ee,e'
2/1 (Ve d').
+0
(G) Ee,e' log ww well"
(vc cc).
Note that cannot calibrate NC4 in this way since that will trivially reduce NC4 back to the standard token prediction (Eq. [
We show the results of these calibrated /C1 /C1/2
2 in Table (right columns). Debiased language models exhibit either comparable or more collapsed measurements on AC((w) and (G), showing greater consistency than the uncalibrated NC,, and (G)NC2 metrics. This suggests that these calibrated metrics more reliably capture neural collapse in debiased language models.
3.3 DEBIASED LMS ARE MORE COLLAPSED IN FAIRNESS- SENSITIVE WORDS
Beyond using different versions of metries quantify neural collapse language models, we further study the impact of different choices of faimess-sensitive words on neural collapse. Our core questions are:
1. Do debiased language models collapse more across the whole vocabulary, only on faimness- sensitive words?
Will the size of subsets of words affect the comparison of NC metrics?
To answer the above two questions, calculate NC metrics on both the whole vocabulary set, and also a random vocabulary subset of the same size as that we used Table (Appendix A). Our gender word list includes 210 female-related words and 215 male-related words.
From Table 2 and Table 3 it is evident that gaps of NCC metrics between BERT and debiased BERT models much smaller than gaps in Table [ This indicates that debiased BERT models exhibit more different token representations and word embeddings only on gender-related vocabulary, and they perform much more similarly with BERT the whole vocabulary. This also explains why
'Fg. The hidden size of BERT is 768, which smaller than the vocabulary size of 30,522.
5

---

Published as a conference paper at ICLR 2025
Table 2: NC metrics of different debiased language models on the whole vocabulary V. Gaps of NC metrics between BERT and debiased BERT models are much smaller than gaps in Table
Model
NC1
(g)NC2 4
(UNCC
NC
NC +
(g)NC)) 4
BERT
0.340
0.248
0.056
0.712
446.8
0,407
MABEL
0.293 0.0471
0.240
0.0061
0.064
0.006L
0.865
0.1531
383.7 63.11
0.397 0.010t
BERT
0.922
0.212
0.062
0.125
806.9
0,413
ASE
1.321 0.3291
0,417
0.206]
0.049 0.0131
0.002 0.1233
193.1 613.8*
0.418 0.0054
BERT
1.308
0.009
0.050
0.734
815.9
0,415
BEC
1,334
0.028L
0.011
0.0024
0.047
0,0031
0.516 0.2181
798.4
17.51
0.408
0.0071
Table 3: NC metrics of different debiased language models on the same number of words as used in Table (V gender) but words are randomly selected. Gaps of NCC metrics between BERT and debiased
BERT models are much smaller than gaps in Table !
Model
NC,
(GINC2 4
(ONC33
NC.1
Nc( +
(G)NC)!)) 1
BERT
0.320
0.248
0.057
0.407
434.3
0,400
MABEL
0.314 0.0005
0.237
0.0111
0.060 0.003L
1.910 1.003
406.4 27.91
0.392 0.00st
BERT
1.086
0.197
0.059
0.235
859.7
0.406
ASE
1.158 0.0725
0.404
0.2074
0.046 0.0131
0.000
0.235]
167.1 692.6
0.436 0.030]
BERT
1.555
0.021
0.047
0.145
866.7
0.411
BEC
1.501
0.0540
0.031
0.0104
0.044
0.0031
1.079
0.9347
822.5
44.21
0.405
0.000t
neural collapse cannot determine if language model is debiased or not across the whole vocabulary, as none of the original NCC1/23344 metrics show consistent improvement in Table Moreover, as shown in T this observation also holds on a random subset of vocabulary with matched size with gender-related words in Table Meanwhile, in this context, NCC measurements are generally smaller than those measured on only gender words, which implies that is generally challenging to learn more separable representations of gender words compared with others insensitive to fairness.
BIAS MITIGATION VIA ENFORCING EXPLICIT COLLAPSE IN LMS
Motivated by our observations in Section g] we further ask: can we enforce explicit neural collapse in language models and thus improve their fairness?
We propose minimizing the regularization of language models using (U)NC's as an auxiliary objective during fine-tuning:
LNC, = std ((1 We
(1erall [4c
He 12
- ))
E
V
gender.
(4)
Although sounds straightforward. this principled approach could potentially be very important in improving the fairness of language models from two perspectives:
1. Our method is simple, principled, and is agnostic to any pretraining or fine-tuning methods for fairness. As we will show in our results, it can be adopted in a wide range of faimess algorithms plug-and-play fashion.
2. Our method can avoid manual filtration or augmentation of the underlying language training data.
In the following experiments, demonstrate that our regularization consistently de-biases language models across different fairness metrics (Sec. I.]), while still preserving models' language modeling performance (Sec. 4.2). We also provide ablation studies on the strength of this regularization in
Appendix B
4.1
ENFORCING NC PROMOTES FAIRNESS, BOTH INTRINSICALLY AND EXTRINSICALLY
4.1.1 IMPLEMENTATION DETAILS
We conduct experiments following the original settings of each work. M2bel+(2))VVs We im- plement Mabel with a batch size of 24, a learning rate of x 10 10-5, and use the Adam optimizer, training it for two epochs. ASE+(4))Ccs ASE is trained for 50 epochs with the Adam optimizer.

---

Published as a conference paper at ICLR 2025
The learning rate is set to 2 10-5, dropout probability of 0.1 is used, and batch size of is chosen. CCc+ BEC is trained for three epochs using the Adam optimizer, with a learning rate of 2 x 10 and batch size of 16. Due to the short fine-tuning duration, we directly accumulate He for regularization when computing (33)NC3.
The evaluation of the fairnsss of language models be categorized into addressing intrinsic and extrinsic biases (Li et al., 2023a), which we detail in the following subsections. We also select Sent-Debias (Liang al. 2020), Context-Debias (Kaneko Bollegala (021), and FairFil (Cheng et al. 2021> as our primary baselines, all of which introduce general-purpose methods for generating debiased representations.
4.1.2 INTRINSIC METRICS
The goal of intrinsic debiasing reduce bias within model representations before they are applied to downstream tasks, making it task- agnostic. Following previous works (He et al.) 2022, Bartl et al., 2020), we consider two pop- ular datasets for intrinsic metrics.
Table 4: Results on StereoSet. *; results are re- ported in He et al.] (2022); We follow previous works to evaluate 50, the better. LM: language modeling score, SS: Steoreotype score, ICAT: combined score, defined as L.M (min(SS, 100 S50)
their datasets, o: the closer to
StereoSet (Nadeem et al., 2020) evaluates the language model by testing for stereotypi- cal associations. Following He et al. (2022), we
StereoSet
Model
LM
SS o
ICAT 1
concentrate on examples within sentences that pertain to the gender domain. When encounter- ing a partial context sentence, this task presents a -te--hh--bblnk challenge where the model must choose from an unrelated word, anti- stereotypical word, or stereotypical word. The Language Modeling Score (LM) quantifies the percentage of times the model correctly identi- fies relevant word, whether it stereotypical or anti-stereotypical, or an irrelevant one. Mean- while, the Stereotype Score (SS) is a fairness- sensitive metric, revealing how often the model shows a preference for the stereotype compared to the anti-stereoe anti-stereotype. Lastly, the Idealized Con- text Association Test (ICAT) score integrates both LM and SS into single metric, quantifying the balance between language modeling and fairness.
BERT
84.17
60.28
66.86
BERT+DROPOUT"
83.04
60.66
65.34
BERT+CDA*
83.08
59.61
67.11
SENT-DEBIAS"
84.20
59.37
68.42
CONTEXT-DEBIAS
85.42
59.35
69.45
FAIRFIL
44.85
50.93
44.01
ADEPT
86.37
58.70
71.34
MABEL
84.80
56.92
73.07
MABEL+(4)
83.55
55,38
74.55
ASE
83.83
57.33
71.54
ASE+(U)NC
84.06
56.36
73.37
BEC
86.02
58.30
71.73
BEC+(UNC
85.95
57.89
72.38
As shown in Table our method consistently improves both SS and ICAT all fairness methods. With our assistance, Mabel's SS score improved by 1.54 and ICAT increased by 1.48, reaching value of 74.55. Similarly, ASE achieved significant ICAT improvement of 1.83.
BEC-Pro [Bartl et al. 2020 create a template- based corpus in two languages, English and Ger- man, to measure bias in BERT. The sentence templates include a gender-denoting noun phrase, or <person word>, along with <profession>. Ta- ble [S] presents the average association scores between person-related terms (targets) and pro- fessions (attributes) before and after applying (U)NN'3 to the debiased model on the GAP cor- pus [Webster et al.) 2018), with Counterfactual Data Substitution (CDS) (Maudslay al., 2019) applied, highlighting the difference between fe-
Table Results on BEC-Pro. We show the mean association scores between gender words and pro- fessions. "Diff" represents the score difference between female and male (smaller the better).
Model
Female
Male
Diff +
BERT
-0.0931
-0.3388
0.2457
ADEPT
-0.0056
0.0476
0.0532
MABLE
-0.0641
-0.0237
0.0404
MABLE+ (UNCC
0.0039
-0.0038
0.0077
ASE
40.7534
-0.5125
0.2409
male and male associations.
ASE+WANGs
-1.0093
-0.9613
0.0480
BEC
0,0841
0.1349
0.0508
We observe a significant reduction in the mean association score difference between female and
BEC+(UWNG
0.1145
0.1435
0.0290
7

---

Published as a conference paper at ICLR 2025
male after applying method. This indicates that the model perceives less distinction between these two concepts, implying a reduction bias. Specifically, Mabel and ASE show a significant order-of-maaee maggnitdde decrease in the score difference with method, and BEC also shows over 40% reduction.
4.1.3 EXTRINSIC METRICS
Extrinsic metrics (Huang et al. 2019; Smith et al. 2022, Lai et al. 2017) focus on enhancing fairness in downstream tasks, such as sentiment analysis (Mohammad al. 2018) and machine translation (Levy et al. 2021 by ensuring that models produce consistent outputs across different demographic groups.
WinoBias Zhao et 2018 is an intra-sentence coreference resolution task designed assess system's ability to correctly associate gendered pronoun with an occupation in both pro-stereotypical and anti-stereotypical contexts. Coreference can be inferred using syntactic cues in Type sentences, or more challenging semantic cues in Type sentences. We fine-tune models on the OntoNotes 5.0 dataset (Hovy et al., 2006) and then evaluate on the WinoBias benchmark. We report the average Fl-scores for pro-stereotypical and stti-stereotypical examples, along with two fairness metrics: TPR-1 (Type I: pro pro-stcreotypies minus auti-stereotypcca) and TPR-2 (Type 2: pro-stereotypical minus anti-stereotypicaa),, measured by average Fl-scores.
As shown Table after adding (2)NVC3 as the regularization, all the models exhibit significant improvements on both fairness metrics. Especially. Mabel equipped with our (U))C3s improves 6.8 on TPR-1 and more than a 40% improvement on TPR-2 over the Mabel baseline, achieving the best results for tested models.
Table 6: Average Fl-scores on WinoBias, and TPR scores across Winobias categories. = Type 1; Type 2. A anti-stereotypical; pro- pro-stereotypical TPR-1 IP 1A; TPR-2 2P 2A.
Model
IA f
IP 1
2A 1
2P f
TPR-1 +
TPR-2
BERT
53.96
86.57
82.20
94.67
32.79
12.48
SENT-DEBIAS
54.11
85.09
83.29
94.73
30.98
11.44
CONTEXT-DEBLAS
59.40
85.54
83.63
93.20
26.14
9.57
FAIRFIL
53.24
85.77
77.37
91.40
32.43
14.03
ADEPT
62.50
84.04
87.66
91.51
21.54
3.85
MABEL
61.21
84,93
92.78
96.20
23.73
3.41
MABEL+
64.15
81.08
93.55
95.51
16.93
1.97
ASE
56.00
87.02
76.44
91.06
31.02
14.62
ASE+(ONCC
58.57
85.71
84.38
92.54
27.14
8.16
BEC
60.32
84.04
86.86
93.98
23.72
7.12
BEC+(UNC
62.98
84.88
87.50
94.40
21.91
6.90
Bias-in-Bios (De-Arteaga et al. 2019) is third-person biography dataset labeled by both occupa- tion and gender. We fine-tune the encoder with the linear classification layer predict individual's occupation from their biography. During evaluation, as well as the accuracy segmented by gender. Additionally, we use two widely adopted fairness metrics De-Arteaga et al. (019); Ravfogel et al. (2020): 1) GAPrPRs which captures the disparity true positive rates (TPR) between male- and female-labeled instances; and 2) GA GAPRMS. which represents the root-mean-squaa (RMS) of TPR gaps across different occupation categories (C), the closer their score is O, the better. Their formula is as follows:
present the overall accuracy for the task,
GAPTPR (TPR - TPR
GAPRMS VIa Σ (GAPTPR.) VEC
(5)
In Table observe that introducing (U)NCss during the fine-tuning process improves the model's performance on these two TPR metrics. Additionally, enhances the model's accuracy in predicting female-related occupations, helping BEC achieve an accuracy of 85.17 for female predictions.
8

---

Published as a conference paper at ICLR 2025
Furthermore, also conducted validation on the Bias-NLI (Dev al. 2020). and the results can be found in the Appendix D
Table 7: Fine-tuning results on Bias-in-Bios.
Ace.
Acc. (M)
Acc.
GAP
GAP
Model
(AIl) f
(
TPR 4
RMS 4
BERT
84.14
84.69
83.50
1.189
0.144
SENT- BENT-DEBIAS
83,56
84.10
82,92
1.180
0.144
CONTEXT-DEBIAS
83.67
84.08
83.18
0.931
0.137
FAIRFIL
83.18
83.52
82.78
0,746
0,142
ADEPT
84.07
83.64
84.58
0.945
0.121
MARLE
84.85
84.92
84.34
0.599
0.132
MABLE+ (4NL
84.30
84.09
84.54
0.455
0.126
ASE
84,63
84.19
85,14
0.949
0.127
ASE+(ONC
84.55
84,21
84,95
0.740
0.126
BEC
84,43
83,99
84,95
0.954
0.124
BEC+
84.66
84.23
85.17
0.938
0.128
4.2 ENFORCING NC PRESERVES LM PERFORMANCE ON NLU TASKS
We also evaluate language models fine-tuned with method general natural language under- standing (NLU) tasks. As shown in Table g fine-ture tne-ttunng with Eq. 4 preserves the performance of language models general tasks with minimal difference from each baseline model. This indicates that our method can debias language models without catastrophically forgetting and sacrificing their pretrained knowledge.
ABLATION STUDIES AND VISUALIZATIONS
Fairness Can Be Improved Solely via NC. As have that debiased language models can implicitly become more collapsed, here we further provide an ablation study to verify that the effectiveness of our regularization is not because of any external debiasing algorithm, namely, by only adopting our NC-based regularization (Eq. 4 we can already encourage fairness in language models. We perform this ablation study in the standard language model fine-tuning for masked language modeling (MLM), where pronouns in the input sentence are replaced with [MASK] tokens. The model is then trained predict the correct gendered pronoun for each [MASK] token in the masked sentence. The training objective is to minimize the cross-entropy loss between the original pronouns and the predicted logits corresponding to the [MASK] tokens. The MLL loss denoted LMLM:
LMLM
Σ
CE(W hmm Fm)
mEmasked
where C' denotes the cross entropy loss, and hm E IRd is the last hidden state of the masked token Im. W IdC.C is linear layer for the MLM task.
We fine-tune the language model using (MLM alone, and compare against the fine-tuning with the addition of LNCg- We fine-tune models for three epochs. This allows more clearly observe the gains brought by (II)NCs, We evaluate the model on StereoSet using intrinsic metrics, and as shown Table 9) it evident that not only does it help to reduce bias in the model, but also enhances the language capabilities. Notably, we achieve a significant improvement of 4.29 on the ICAT metric.
Visualizations Debiased Token Representations. In addition to comparing quantitative results, we also conducted t-SNE visualizations of models' logits to analyze the two training methods. Specifically, we visualize two opposing word pairs: ("Herself"' "Himself") and ("Mother", "Father"). As shown in Figure with the original training method, words of the same gender tend cluster together (e.g., "Himself" and "Father"). However, after introducing (U)NC3, we can find that: 1) token representations from different classes (words) are more clearly separated each other;

---

Published as a conference paper at ICLR 2025
Table 8: Fine-tuning results on the GLUE benchmark: (44)CC3 preserves LM performance while encouraging fairness. BERT-NLI: BERT fine-tuned on NLI data then fine-tuned on GLUE. Bold numbers indicate the best for different metries.
CoLA (mcc.)
SST-21 (acc.)
MRPC ((I/acc.)
QQP (ace//11
MNLI (acc.)
QNLI 1 (acc.)
RTE
STS-B
Model
(acc.)
(pears//spear)
BERT
56,5
92.3
89.5/85.3
90.7/87.5
84.3
92.2
65.0
88.4/88.7
BERT-NLI
58.6
93.6
89.4/85.1
90.4/86.8
83.3
89.0
69.0
88. 88.3/87.9
SENT-DEBIAS
50.5
89.1
87.5/81.6
87.5/90.7
83.9
91.4
63.2
88. 88.1/87.9
CONTEXT-DEBIAS
55.2
92.0
85,1/77.5
90.7/87,4 4
84.6
89.9
57.0
88 88.4/88,
FAIRFIL
55.5
92.4
87.5/804
91 91.2/88.1
84.8
91.3
63.2
88 4/88./8
ADEPT
57.3
92.4
88.0/82.4
-----
84.1
91.0
61.4
85.8/85.7
MABEL
57.8
92.2
89.5/85.0
91 01.2/88.1
84.5
91.6
64.3
89.6 89.6/8992
MABEL+(OWC
56.3
92.1
90.5/86.5
91.0/88.0
84.5
91.1
60.7
1/8.7
ASE
54.9
92.9
87.3/80.9
84.4
91.2
59.3
88 88.4/88.J
ASB+
55.3
92.8
87.5/81.4
84.6
91.4
61.7
88.2/87.9
BEC
55.5
92.2
88.8/83.8
84.9
91.5
89 99.2/88.9
BBC+(U)NCs
55.7
92.5
88.7/83.6
84.9
91.6
66.4
89 19.2/88.9
Table Fine-tuning BERT with only mask-token predictions (Eq. 6 and (L)WCss (Eq. 4 Metrics are consistent with Table 4 "LM" for language modeling score, "SS for Steoreotype score, "ICAT"
combined score, defined as LM (min(SS, 100 /50)
StereoSet
Model
LM f
SS
ICAT
BERT+CMLM
72.85
58.65
60.24
BERT+CMEM+CNG,
75.43
57.23
64.53
2) tokens from each word are more clustered. This indicates that intra-class variances have been significantly reduced, thus gender-related words are more collapsed their class means.
BERT + Lpii
BERT +La + LIND
40
Father
20
Himself
0-
Mother
-20
-20
-40
Herself
-40
Figure 2: t-SNE plots of logits of two models in Table We collect 15 samples each of "Herself," "Himself." "Mother," and "Father."
CONCLUSION
Aiming to provide a principled understanding and improvement of debiasing language models, we try find connections between language models and neural collapse for fairness purposes. We demonstrated that debiased models exhibit more collapsed token representations, especially for fairness-sensitive words, leading to better alignment with word embeddings. Leveraging this insight, we introduced a principled regularization method based neural collapse that can consistently improve fairness in language models across various tasks without sacrificing performance. Our method is simple, effective, and applicable to wide range of debiasing techniques, providing robust tool for enhancing fairness in language models. We expect our understanding and fine-tuning to become principled method for debiasing language models.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Marion Bartl, Malvina Nissim, and Albert Gatt. Unmasking contextual stereotypes: Measuring and mitigating bert's gender bias. arXiv preprint arXiv:2010. 14534, 2020.
Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, and Hanna Wallach. Stereotyping norwegian salmon: An inventory of pitfalls in fairness benchmark datasets. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguisties and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1004-1015, 2021.
Tolga Bolukbasi, Kai-We Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. Man is to computer programmer as woman to homemaker? debiasing word embeddings. Advances in neural information processing systems, 29, 2016.
Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. A large annotated corpus for learning natural language inference. arXiv preprint arXiv: 053266 2015.
Tom B Brown, Language models are few-shot learners. arXiv preprint arXiv:2005.1416: 2020.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Bric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.122711 2023.
Aylin Caliskan, Joanna Bryson, and Arvind Narayanan. Semantics derived automatically from language corpora contain human- like biases. Science, 356(6334):183--888 2017.
Pengyu Cheng, Weituo Hao, Siyang Yuan, Shijing Si, and Lawrence Carin. Fairfil: Contrastive neural debiasing method for pretrained text encoders. arXiv preprint arXiv:2103. 13..06413, 2021.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction- finetuned language models. Journal of Machine Learning Research, 25(70):1-53, 2024.
Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. Bias in bios: A case study of semantic representation bias in high-stakes setting. In proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 120-128, 2019.
Sunipa Dev, Tao Li, Jeff M Phillips, and Vivek Srikumar. On measuring and mitigating biased inferences of word embeddings. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 7659-7666 2020.
Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.0488 04805, 2018
Ronen Eldan and Yuanzhi Li. Tinystories: How small can language models be and still speak coherent english? arXiv preprint arXiv:2305.0773: 2023.
Cong Fang, Hangfeng He, Qi Long, and Weijie J Su. Exploring deep neural networks via layer-peeled model: Minority collapse in imbalanced training. Proceedings of the National Academy of Sciences, 118(43)e21030911 18, 2021.
Zahra Fatemi, Chen Xing, Wenhao Liu, and Caiming Xiong. Improving gender faimess of pre-trained language models without catastrophic forgetting. arXiv preprint vrXiv221000536)) 2021.
Tomer Galanti, András György, and Marcus Hutter. On the role neural collapse in transfer learning. arXiv preprint 2xXiv:21/..///22 2021.
Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mchrab Tanjim, Sungchul Kim, Franck Dernon- court, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed. Bias and fairness in large language models: A survey. Computational Linguistics, pp. 1-79, 2024.
Peifeng Gao, Qiangian Xu, Yibo Yang, Peisong Wen, Huiyang Shao, Zhiyong Yang, Bernard Ghanem, and Qingming Huang. Towards demystifying the generalization behaviors when neural collapse emerges. arXiv preprint arXiv:2310.0885 2023.

---

Published as a conference paper at ICLR 2025
Sahaj Garg, Vincent Perot, Nicole Limtiaco, Ankur Taly, Ed H Chi, and Alex Beutel. Counterfactual fairness in text classification through robustness. In Proceedings of the 2019 AAAI/ACM Conference on Al. Ethics, and Society, pp. 219-226, 2019.
Wei Guo and Aylin Caliskan. Detecting emergent intersectional biases: Contextualized word embeddings contain distribution of human-like biases. In Proceedliggs of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 2021.
Yue Guo, Yi Yang, and Ahmed Abbasi. Auto-debias: Debiasing masked language models with automated biased prompts. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1012-1023, 2022.
Xudong Han, Timothy Baldwin, and Trevor Cohn. Balancing out bias: Achieving fairesss through training reweighting, arXiv preprint arXiv:2109.08253, 2021a.
Xudong Han, Timothy Baldwin, and Trevor Cohn. Diverse adversaries for mitigating bias in training. arXiv preprini arXiv:2101. 100000,,01,0001,,, 2021b.
XY Han, Vardan Papyan, and David L Donoho. Neural collapse under mse loss: Proximity and dynamics on the central path. arXiv preprint arXiv:2106.02003, 2021c
Hangfeng He and Weijie J Su. A law of data separation in deep learning. Proceedings of the National Academy of Sciences, 120(36):e221 (etpee22221111122 120, 2023.
Jacqueline He, Mengzhou Xia, Christiane Fellbaum, and Danqi Chen. Mabel: Attenuating gender bias using textual entailment data, arXiv preprint arXiv:22 14975, 2022.
Eduard Hovy, Mitch Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. Ontonotes: the 90% solution. In Proceedings of the human language technology conference of the NAACL, Companion Volume: Short Papers, pp. 57-60, 2006.
Po-Sen Huang, Huan Zhang, Ray Jiang, Robert Stanforth, Johannes Welbl, Jack Rae, Vishal Maini, Dani Yogatama, and Pushmeet Kohli. Reducing sentiment bias in language models via counterfac- tual evaluation. arXiv preprint 1rXiv:1911.03064 2019.
Like Hui, Mikhail Belkin, and Preetum Nakkiran. Limitations of neural collapse for understanding generalization in deep learning. arXiv preprint arXiv:2202.08384, 2022.
Masahiro Kaneko and Danushka Bollegala. Debiasing pre- pre-trained contextualised embeddings. arXiv preprint arXiv:2 10l.0 2021.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading comprehension dataset from examinations. arXiv preprint 1rXiv:1704.44680 2017.
Shahar Levy, Koren Lazar, and Gabriel Stanovsky. Collecting large-scale gender bias dataset for coreference resolution and machine translation. arXiv preprint 1rXiv:2109.03854. 2021.
Donghao Li, Yang Cao, and Yuan Yao. Neuromixgdp: A neural collapse-inspires random mixup for private data release. In Conference on Parsimony and Learning, pp. 480-514. PMLR, 2024.
Yingji Li, Mengnan Du, Rui Song. Xin Wang, and Ying Wang. A survey on fairness in large language models. arXiv preprint arXiv:2308. 10149, 2023a.
Zexi Li, Xinyi Shang, Rui He, Tao Lin, and Chao Wu. No fear of classifier biases: Neural collapse inspired federated learning with synthetic and fixed classifier. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 5319-5329, 2023b.
Paul Pu Liang, Irene Mengze Li, Emily Zheng, Yao Chong Lim, Ruslan Salakhutdinov, and Louis- Philippe Morency. Towards debiasing sentence representations. arXiv preprint arXiv:2007.08100 2020.
Litian Liu and Yao Qin. Detecting didtof-distribution through the lens neural collapse. arXiv preprint 1:Xiv:2311.01473 2023.
12

---

Published as a conference paper at ICLR 2025
Yinhan Liu. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:190..1169. 2019.
Kaiji Lu, Piotr Mardziel, Fangjing Wu, Prectam Amancharla, and Anupam Datta. Gender bias in neural natural language processing. Logic, language, and security: essays dedicated to Andre Scedrov the occasion of his 65th birthday, pp. 189-202, 2020.
Thomas Manzini, Yao Chong Lim, Yulia Tsvetkov, and Alan W Black. Black is to criminal as caucasian is to police: Detecting and removing multiclass bias in word embeddings. arXiv preprint WVXiv:/904.04047, 2019.
Rowan Hall Maudslay, Hila Gonen, Ryan Cotterell, and Simone Teufel. It's all in the name: Mitigating gender bias with name-based counterfactual data substitution. arXiv preprint (00.vv/9990007777 2019.
Chandler May, Alex Wang, Shikha Bordia, Samuel R Bowman, and Rachel Rudinger. On measuring social biases in sentence encoders. arXiv preprint arXiv: /903. 10561, 2019.
Tomáš Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in contimuous space word representations. Proceedings of the 2013 conference of the north american chapter of the association for computational linguistics: Human language technologies, pp. 746-751, 2013.
Saif Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. Semeval- 2018 task I: Affect in tweets. In Proceedings of the 12th international workshop on semantic evaluation, pp. 1-17, 2018.
Moin Nadeem, Anna Bethke, and Siva Reddy. Stereoset: Measuring stereotypical bias in pretrained language models. arXiv preprint arXiv:2004. 09456,, 2020.
Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40): 24652-24663 2020.
Sun Young Park, Kyuri Choi, Haeun Yu, and Youngjoong Ko. Never too late to learn: Regulariz- ing gender bias in coreference resolution. In Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining, pp. 15 15-23, 2023.
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations, 2018. URL http:::/aa/tiiv.orrr
abs/1802.0530 05365
Yada Pruksachatkun, Satyapriya Krishma, Jwala Dhamala, Rahul Gupta, and Kai-Wei Chang. Does robustness improve fairness? approaching fairness with word substitution robustness methods for text classification. arXiv preprint arXiv:2106. 10826, 2021.
Rebecca Qian, Candace Ross, Jude Fernandes, Eric Smith, Douwe Kiela, and Adina Williams. Perturbation augmentation for fairer nip. arXiv preprint arXiv:2205. 12586, 2022.
Juliano Rabelo, Randy Goebel, Mi- Mi-Young Kim, Yoshinobu Kano, Masaharu Yoshioka, and Ken Satoh. Overview and discussion of the competition on legal information extraction/entailment (colice) 2021. The Review of Socionetwork Strategies, 16(1): 111)::11-1333 2022.
Akshay Rangamani, Marius Lindegaard, Tomer Galanti, and Tomaso A Poggio. Feature learning in deep classifiers through intermediate neural collapse. In International Conference Machine Learning, pp. 18729-28744 PMLR, 2023.
Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. Null it out: Guarding protected attributes by iterative nullspace projection. arXiv preprint wxXv:2004.00667,, 2020.
Shauli Ravfogel, Michael Twiton, Yoav Goldberg, and Ryan Cotterell. Linear adversarial concept erasure. In International Conference Machine Learning, pp. 18400-18421. PMLR, 2022.
13

---

Published as a conference paper at ICLR 2025
Aili Shen, Xudong Han, Trevor Cohn, Timothy Baldwin, and Lea Frermann. Contrastive learning for fair representations. arXiv preprint arXiv22109.10455, 2021.
Eric Michael Smith, Melissa Hall, Melanie Kambadur, Eleonora Presani, and Adina Williams. "i'm sorry to hear that": Finding new biases in language models with a holistic descriptor dataset. arXiv preprint arXhv:2205.009: 2929099 2022.
Alex Tamkin, Amanda Askell, Liane Lovitt, Esin Durmus, Nicholas Joseph, Shauna Kravec, Karina Nguyen, Jared Kaplan, and Deep Ganguli. Evaluating and mitigating discrimination in language model decisions. arXiv preprint arXiv:23 2023.
Jörg Tiedemann. Parallel data, tools and interfaces in opus. In Lrec, volume 2012, pp. 2214-2218. Citeseer, 2012.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
Chendi Wang, Yuqing Zhu, Weijie Su, and Yu-Xiang Wang. Neural collapse meets differential privacy: Curious behaviors of noisygd with near-perfect representation learning, arXiv preprint arXiv:2405.08920, 2024a.
Yining Wang, Junjie Sun, Chenyue Wang, Mi Zhang, and Min Yang. Navigate beyond shortcuts: Debiased learning through the lens of neural collapse. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12322-12331, 2024b.
Kellie Webster, Marta Recasens, Vera Axelrod, and Jason Baldridge. Mind the gap: A balanced corpus of gendered ambiguous pronouns. Transactions of the Association for Computational Linguistics, 6:605-617, 2018.
2018.
Kellie Webster, Xuezhi Wang, Ian Tenney, Alex Beutel, Emily Pitler, Ellie Pavlick, Jilin Chen, Ed Chi, and Slav Petrov. Measuring and reducing gendered correlations in pre-trained models. arXiv preprint 2XXi:20000000:32, 2020.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.0166 2021.
Adina Williams, Nikita Nangia, and Samuel R Bowman, A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint urXiv: 2017.
Robert Wu and Vardan Papyan. Linguistic collapse: Neural collapse in (large) language models. arXiv preprint 2rXiv:2405.117710 2024.
Ke Yang, Charles Yu, Yi R Fung, Manling Li, and Heng Ji. Adept: A debiasing prompt framework. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 10780-10788, 2023.
Yibo Yang, Shixiang Chen, Xiangtai Li, Liang Xie, Zhouchen Lin, and Dacheng Tao. Inducing neural collapse in imbalanced learning: Do we really need a learnable classifier at the end deep neural network? Advances in neural information processing systems, 15:37991-38000 2022.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kei Chang. Gender bias in coreference resolution: Evaluation and debiasing methods. arXiv preprint 2018.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Cotterell, Vicente Ordonez, and Kai-Wei Chang. Gender bias in contextualized word embeddings. arXiv preprint arXiv:1904.03310 2019.
Jinxin Zhou, Xiao Li, Tianyu Ding, Chong You, Qing Qu, and Zhihui Zhu. On the optimization landscape of neural collapse under mse loss: Global optimality with unconstrained features. In International Conference Machine Learning, pp. 27179-27202. PMLR, 2022a.
14

---

Published as a conference paper at ICLR 2025
Jinxin Zhou, Chong You, Xiao Li, Kangning Liu, Sheng Liu, Qing Qu, and Zhihui Zhu. Are all losses created equal: A neural collapse perspective. Advances in Neural Information Processing Systems, 35:31697-31770,, 2022b.
Ran Zmigrod, Sabrina J Mielke, Hanna Wallach, and Ryan Cotterell. Counterfactual data aug- mentation for mitigating gender stereotypes in languages with rich morphology. arXiv preprini arXiv:1906.00457,, 2019.
15

---

Published as a conference paper at ICLR 2025
A
GENDER WORDS
countryman, fraternal, wizards, manservant, fathers, divo, actor, bachelor, papa, dukes, barman, countrymen, brideprice, hosts, airmen, andropause, penis, prince, governors, abbot, men, widower, gentlemen, sorcerers, sir, bridegrooms, baron, househusbands, gods, nephew, widowers, lord, brother, grooms, priest, adultors, andrology, bellboys, his, marquis, princes, emperors, stallion, chairman, monastery, priests, boyhood, fellas, king, dudes, daddies, manservant, semen, spokesman, tailor, cowboys, dude, bachelors, barbershop, emperor, daddy, masculism, guys, enchanter, guy, fatherhood, androgen, cameramen, godfather, strongman, god, patriarch, uncle, chairmen, sir, brotherhood, host, testosterone, husband, dad, steward, males, cialis, spokesmen, pa, beau, stud, bachelor, wizard, sir, nephews, fathered, bull, beaus, councilmen, landlords, grandson, fiances, stepfathers, horsemen, grandfathers, adultor, schoolboy, rooster, grandsons, bachelor, cameraman, dads, him, master, lad, policeman, monk, actors, salesmen, boyfriend, councilman, fella, statesman, paternal, chap, landlord, brethren, lords, blokes, fraternity, bellboy, duke, balletdancer, dudes, fiance, colts, husbands, suitor, paternity, he, businessman, masseurs, hero, deer, busboys, boyfriends, kings, brothers, masters, stepfather, grooms, son, studs, cowboy, mentleman, sons, baritone, salesman, paramour, malehost, monks, menservants, mr, headmasters, lads, congressman, airman, houschusband, priest, barmen, barons, abbot, handyman, beard, fraternities, stewards, colt, czar, stepsons, himself, boys, lions, gentleman, penis, his, masseur, bulls, uncles, bloke, beards, hubby, lion, sorcerer, macho, father, gays, male, waiters, sperm, prostate, stepson, prostaticutricle, businessmen, heir, waiter, headmaster, man, governor, god, bridegroom, grandpa, groom, dude, gay, gents, boy, grandfather, gelding, paternity, roosters, prostaticutricle, priests, manservants, stailor, busboy, heros.
Male:
Female: countrywoman, sororal, witches, maidservant, mothers, diva, actress, spinster, mama, duchesses, barwoman, countrywomen, dowry, hostesses, airwomen, menopause, clitoris, princess, governesses, abbess. women, widow, ladies, sorceresses, madam, brides, baroness, housewives, goddesses, niece, widows, lady, sister, nun, adultresses, obstetrics, bellgirls, marchioness, princesses, empresses, mare, chairwoman, convent, priestesses, girlhood, gals, mommies, maid, female ejacula- tion, spokeswoman, seamstress, cowgirls, chick, hairsalon, empress, mommy, feminism, enchantress, gal, motherhood, estrogen, camerawomen, godmother, strongwoman, goddess, matriarch, aunt, chair- women, ma' am, sisterhood, hostess, estradiol, wife, mom, stewardess, females, viagra, spokeswomen, ma, belle, minx, maiden, witch, miss, nieces, mothered, cow, belles, councilwomen, landladies, granddaughter, fiancees, stepmothers, horsewomen, grandmothers, adultress, schoolgirl, hen, grand- daughters, bachelorette, camerawoman, moms, mistress, lass, policewoman, saleswomen, girlfriend, councilwoman, stateswoman, maternal, wenches, sorority, ballerina, chicks, fiancee, fillies, suit- ress, maternity, she, businesswoman, masseuses, heroine, doe, busgirls, girlfriends, queens, sisters, mistresses, stepmother, daughter, minxes, cowgirl, mezzo, saleswoman, nuns, maids, mrs., head- mistresses, lasses, congresswoman, airwoman, housewife, priestess, barwomen, baronesses, abbesses, handywoman toque, sororities, stewardesses, filly, czarina, stepdaughters, herself, girls, lionesses, vagina, hers, masseuse, aunts, wench, toques, heiress, waitress, headmistress, bride, grandma, lesbian, girl, grandmother, hens, uterus, maidservants, seamstress', busgirl, heroines.
B ABLATION STUDY ON NC, REGULARIZATION
Our method is simple and principled. adds our ACc constraint to the original training loss used in each work, as shown in the following formula:
Liotal == Coriginal + LNCC,
where Loriginat represents the original loss from the respective study, and LvCs denotes our regular- ization term.
From Table 10 it's clear that increases, the SS metric generally improves, indicating that the model becomes more biased. However, a more appropriate value of a can strike better balance between language quality and bias, resulting in higher ICAT score.
C NC EVALUATIONS FOR MORE LANGUAGE MODELS
In addition to the three debiasing methods tested Section 3 we supplement our analysis here with experiments from popular models: BERT base and ERaa base. The results from these models
16

---

(GWeC Published as a conference paper at ICLR 2025
Table 10: StereoSet results on different hyper-parameter settings of a. Metries are consistent with Table "LM"" for language modeling score, "SS" for Steoreotype score, "ICAT" for combined score, defined LM (min(SS, 100 /50)
ASE+
(1)
BEC+())NC;
LM
SS
ICAT f
LM
SS
ICAT
LM
SS
ICAT
a
83.78
57.38
71.41
a 10
83.57
55.53
74.33
86.02
58.07
72.12
3
84.06
56.36
73.37
a
30
83.54
55.63
73.88
85.95
57.89
72.38
a 5
82.96
55.84
73.27
a 50
83.55
55.38
74.55
85.70
58.92
71.44
(Table 1 12 and 13) are consistent with the observations made in Section 3 The three additional
model setups are as follows: Adept on Cews-Commyntary v15 (Tiedemann 2012); Roberta and FairBERTa PANDA (Qian et al. 2022); Mabel under Roberta as the backbone.
Table 11: The performance of more debiasing methods NCC metrics varies across datasets on the gender words (V genderr).
Model
NC,
(G)NC, 1
(WWCss
NC,1
(GWeC)
BERT
2.090
0.088
1.317
0.169
1197.6
0.359
ADEPT YANG ET AL.. 2023
1.941
0.072
0.074
0.118
1143.5
0.364
ROBERTA
0.868
0.010
0.066
0.513
11.53
1.614
FAIRBERTA (QIAN AI..
2022
1.287
0.661
0.038
0.000
5,883
1.698
ROBERTA
0.443
0.059
0.062
0.250
10.22
1.488
MABEL-ROBERTA QIAN
2022
0.513
0.013
0.058
0.250
9.878
1.487
Table 12: NC metrics of more debiased language models on the whole vocabulary (V).
Model
NC,
(G)NC, 4
(2ONCs
NC,
Acf")
(gWc)!
BERT
1.235
0.165
0.064
0.013
946.2
0.413
ADEPT YANG AL.. 2023
1.133
0.149
0.066
0.012
889.2
0.420
ROBERTA
0.405
0.116
0.065
0.647
8.334
1.617
FAIRBERTA QIAN
2022
0.766
0.520
0.036
0.000
4.545
1.732
ROBERTA
0.246
0.135
0.369
6.219
1.567
MABEL-ROBERTA QIAN
2022
0.242
0.061
0.061
0.568
5.363
1.567
Table 13: NCC metrics of more debiased language models the same number of words as used in Table 11 (% gender) but words are randomly selected.
Model
NC,
(G)NC, 4 (WWNCs
NC,
BERT
1.292
0.158
0.065
0.002
974.7
0.409
ADEPT YANG ET AI.. 2023
1.111
0.150
0.064
0.002
888.6
0.423
ROBERTA
0.407
0.116
0.067
0.811
8.416
1.618
FAIRBERTA (QIAN AI..
2022
0.728
0.511
0.036
0.000
4.414
1.733
ROBERTA
0.217
0.146
0.058
0.162
5.653
1.561
QIAN ET AL.
2022
0.237
0.063
0.064
0.280
5.267
1.572
D EXTRA EXTRINSIC METRICS
Bias-NLI Dev et al. (0200 is an NLI dataset composed of neutral sentence pairs, systematically generated by populating sentence templates with a gendered word and an occupation that has a strong gender association (e.g., "The woman ate bagel" "The murse ate bagel"). Bias is measured deviation from neutrality and evaluated using three metrics: Net Neutral (NN), Fraction Neutral (FN), and Threshold:T (T:r). Specifically, 1) NN represents the mean probability assigned to the
17

---

Published as a conference paper at ICLR 2025
neutral label across all entailment pairs; 2) FN calculates the proportion of sentence pairs classified as neutral; and 3) Threshold:7 (T:T) is hyperparameter that determines the proportion of entailment pairs for which the probability of being neutral exceeds given threshold. In this paper, is set to 0.5 and 0.7. And a bias-free model would achieve score as I across all three metrics. NN and FN defined in the following manner:
M
NN
FN
Σ n mar n4, Mi,cs, ci
(8)
M
M
We fine-tune the model on SNLI and assess its performance on Bias-NLI during inference. After incorporating (4))CC3 during fine-tuning, find that such rare-class classification scenario, the improvement relatively limited. However, generally helps the model maintain its performance, leading to some gains accuracy across various metries for ASE and BEC. Specific details can be found in Table
Table 14: Results on Bias-NLI. We fine-tune the models on SNLI and then evaluate on Bias-NLI.
Model
TN
FN
T:0.5 f
T:0.7
BERT
0.799
0.879
0.874
0.798
SENT-DEBLAS
0.793
0.911
0.897
0.788
CONTEXT-DEBIAS
0.858
0.906
0.902
0.857
FAIRFIL
0.829
0.883
0.846
0.845
ADEPT
0.841
0.937
0.934
0.866
MABLE
0.900
0.977
0.974
0,935
MABLE+(U)NG;
0.882
0.977
0.974
0.919
ASE
0.886
0.974
0.971
0.932
ASE+(UNGG
0.910
0.974
0.973
0.948
BEC
0.933
0.955
0.955
0.934
BEC+(U)NCs
0.883
0.980
0.978
0.943
NC EVALUATIONS FOR NEUTRAL WORDS
In addition to gender-related words such as "female" and "male," there are also neutral words. To assess the impact of these neutral words, conducted ablation experiments. The experimental setup as follows:
Gender words + Neutral words: including 425 gender-related words from Appendix A and 208 neutral words mentioned below.
Gender words Random words: including 425 gender-related words from Appendix A and 208 Random words.
aerobics, adventurer, apparel, aggressive, assistant, tycoon, baker, warrior, bathing, ambi- tious, beautiful, trucker, beauty, welder, blonde, strong, bookkeeper, terrorist, ca, soldier, cashier, astronomer, chatty, sniper, cheerleader, skipper, cheerleading, banker, clerk, baseball, cocktail, sergeant, cooking, bodyguard, counselor, boss, crafting, boxer, cute, cabbie, dancer, captain, educa- tor, cardiologist, emotional, carpenter, flirt, ceo. flirtatious, chairperson, flower, chancellor, gossip, chef, graceful, colonel, hairdresser, commander, hairdryer, conductor, homemaker, police, hooker, custodian, housekeeper, dentist, housekeepers, detective, housework, diplomat, hula, doctor, indoor, driving, jealousy, dewellr, economist, kawaii, electrician, laundering, engineer, librarian, engineering, librarians, entrepreneur, lotion, lovely, firefighter, marvelous, footballer, mirror, gambler, moisturizer, gamer, nanny, gangster, neat, geek, nurse, geeks, nursery, gentle, nurses, guitarist, nurtur- ing, industrialist, parenting, inventor, passive, investigator, pink, laborer, pretty, lawyer, receptionist, leader, ribbon, lieutenant, romance, lifeguard, romantic, magistrate, secretary, manager, selfie, mar- shal, server, mathematician, sew, mechanic, sewing, muscle, shopping, muscular, smoothie, owner,
Neutral:
18

---

Published as a conference paper at ICLR 2025
soft, philosopher, softball, physicist, stylist, pilot, submissive, plumber, sweet, politician, tailor, presi- dent, tall, professor, teacher, programmer, thin, rugby, violinist, sailor, waiter, science, weak, scientist, yoga, sculptor, hysterical, blue, makeup, football, executive, management, professional, corporation, salary, office, business, career, home, parents, children, family, cousins, marriage, wedding. relatives, math, algebra, geometry, calculus, equations, computation, numbers, addition, poetry, art, dance, literature, novel, symphony, drama, sculpture, science, technology, physics, chemistry, Einstein, NASA, experiment, astronomy, Shakespeare.
As shown in Table 15 and Table 16 results of the two different word combinations on the NC metrics are very similar, which further supports the rationale for considering only gender-related words in our method. However, upon closer inspection, we can observe that when using neutral words, the metric values tend to be closer to those obtained with only gender words (Table and Table [T]. whereas the values with random words are more aligned with those generated by random words (Table 3 and Table 13). This further supports the validity of the NC metric.
Table 15: NC metrics of different debiased language models on Gender words + Neutral words.
Model
NC₁
(G)VC2 4
(10NCs
NC2 +
NC
(G)NC(
BERT MABEL (HE ET AL..
0.894
0.139
0.066
1.634
790.3
0.337 0.330
2022
0.698
0.141
0.069
1.421
687.6
BERT
2.182
0.073
0.064
0.684
1155.1
0.354
ASE (PARK ET AL.
2023
2.975
0.566
0.054
0.013
324.3
0.363
BERT
1,988
0.145
0.060
6.260
1033.0
0.351
BEC BARTL ET AL.
2020
2.105
0.183
0.055
4,469
1014.7
0.346
BERT
2.275
0.071
0.070
0.099
1267.4
0.351
ADEPT YANG ET AL.
2023
2.104
0.053
0.070
0.070
1206.4
0.357
ROBERTA
0.737
0.018
0.065
0.693
10.48
1.621
FAIBBERTA (QIAN ET AL.
2022
1.138
0.632
0.035
0.000
5,459
1.706
ROBERTA
0.317
0.075
0.065
0.155
6.986
1.553
MABEL-ROBERTA QIAN ET AL.
2022
0.333
0.069
0.065
0.134
5.573
1.553
Table 16: NC metrics of different debiased language models on Gender words + Random words.
Model
ACC
(g)NCa 1 (UNNCs
NC,
Nc("")
(G)-e)
BERT
0.466
0.225
0.061
1.137
533.5
0.390
MABEL (E
2022
0.413
0.219
0.065
1.132
475.6
0.384
BERT
1.534
0.142
0.064
0.644
943.3
0,391
ASE (PARK AL.
2023
2.076
0.445
0.055
0.012
288.17
0.397
BERT
1.742
0.052
0.083
5.721
891.8
0.399
BEC (BARTL
2020
1.903
0.093
0.051
4,115
892.6
0.389
BERT
1.555
0.140
0.068
0.093
1043.0
0,394
ADEPT YANG ET AL..
2023
1.473
0.117
0.069
0.067
1010.9
0.397
ROBERTA
0.515
0,093
0.068
0.256
9.151
1.631
FAIRBERTA QQIAN ET AL.
2022
0.909
0.552
0.037
0.000
5.032
1.731
ROBERTA
0.255
0.135
0.063
0.591
6.508
1.564
QIAN ET AL.
2022
0.260
0.040
0.062
1.617
5.595
1.567
F DETAILED EXPLANATION OF BEC-PRO
The association score accurately measures the correlation between a <person word> and <profes- sion>, with smaller values indicating weaker correlation. However, it is important to note that the association score itself does not directly indicate gender bias. For instance, both male and female associations could be weak, but the difference between these scores can reveal gender bias, where the profession may show varying inclinations toward males and females. When examining the disparity in association scores between <female> and <male> with respect to a particular profession, this difference can be interpreted metric serves as a more relevant and intrinsic measure gender bias, specifically addressing how professions are biased in their associations with gender.
measure of gender bias in occupational associations. As such, this
19

---

Published as a conference paper at ICLR 2025
To aid in understanding, conducted an experiment in Table 17 which (4)WC3 optimized in BEC using different sets words. The results demonstrate that both gender and profession words reduce the association scores. However, regularizing (U)NCC using gender-specific words leads to a reduced distinction between female and male associations (i.e., a decrease the difference), whereas regularizing with profession words actually escalates the gender bias.
Table 17: More results on BEC-Pro. We train BEC by utilizing different word lists (gender/profession)
for (U)WC3, a 10 (Eqq 7
Model
Female
Male
Diff +
BEC
0.0841
0.1349
0.0508
(EC+(L))CC ON GENDER
0.0632
0.1103
0,0471
(EC+ ON PROFESSION
0.0429
0.1200 +
0.0771 f
G ABSOLUTE CHANGES OF NC AFTER DEBIASING
From Figure 3 is evident that NC is more sensitive to fairness-related words, such as gender.
20

---

Published as a conference paper at ICLR 2025
NC1
(G)NC2
0.7
0.5
0.6
0.4
0.5
0.4
0.3
0.25
0.15
0.05
0
Mabel
ASE
BЕC
Mabel
ASE
BЕC
(U)NC3
NC4
0.03
3.5
0.025
3
2.5
0.02
2
0.015
1.5
0.01
0.005
Mabel
ASE
BEC
Mabel
ASE
Вec
NC1-W
NC2-w
800
0.035
700
600
500
400
0.015
0.01
200
0.005
0
Mabel
BEC
Mabel
ASE
BЕC
Gender
Whole Vocabulary
Random
Figure 3: Absolute changes NCC metrics after debiasing Mabel, ASE, BEC, Values used in these
plots are based on Table Table 2 and Table
21


Reviews:
[{"review_id": "TeODoga2Wu", "reviewer": "Reviewer_ypXD", "summary": "This work explores fairness in language models from the perspective of Neural Collapse (NC), a phenomenon observed during the final stages of training neural networks on classification tasks (thus can also be applied to language modeling), where the variability and alignment of the penultimate layer and the final classifier reach a structured state. This work further applies NC to debiased LMs and demonstrates that  $\\mathcal{NC}3$, which measures the alignment between token representations (contextual class means) and word embeddings (classifiers), consistently improved with debiasing. The author argues that other NC perspectives are less evident in debiased LMs due to common training configurations that violate NC’s typical conditions—such as vocabulary sizes exceeding embedding dimensions and imbalanced token distributions. To address this, they propose calibrated versions of  $\\mathcal{NC}1$ and  $\\mathcal{NC}2$ by replacing class means with classifier weights. Debiased LM shows significant collapse on these two metrics, and fairness-sensitive words demonstrate more collapse.\n\nBased on the empirical analysis, the authors suggest using  $\\mathcal{NC}3$ as a regularization term during fine-tuning to enhance fairness in LMs. Experiments on four fairness datasets demonstrate consistent gains on fairness metrics over three prior debiasing approaches and standard masked language modeling and also doesn't hurt the model's NLU ability.", "questions": "1. In section 3.2.2, the authors present a calibrated version of $\\mathcal{NC}1$ and  $\\mathcal{NC}2$. While I can understand that the main reason to replace class means with classifier weights is because the authors find that $\\mathcal{NC}3$ (solely derived from classifier weights) is significant on the debiased models, could this modification compromise the original physical meaning of these metrics Specifically, \n    - $\\mathcal{NC}1$, as proposed in [1], measures the variability via an inverse signal-to-noise ratio. However, in the calibrated $\\mathcal{N}\\mathcal{C}_{1}^{(w)}$, the numerator is the variance of the _class means_, while the denominator measures the distance of the _classifier weight_, which seems to lack physical coherence.\n    - Why is $w_c$ not normalized in $\\mathcal{(G)NC_2^{(w)}}$ while $\\mu_c$ is normalized in $\\mathcal{(G)NC_2}$ to measure uniformity?\n    - I hope the authors can elaborate more on the reason they choose these calibrated metrics.\n2. In section 3.3, the authors argue that _\"debiased BERT models exhibit more different token representations and word embeddings only on gender-related vocabulary\"_ based on __numerical gaps__ in $\\mathcal{NC}$ metrics. However, the baseline level of the base BERT model differ across gender-related words, full vocabulary, and a randomly selected subset. By looking at the __ratios__ of the gap w.r.t. base BERT model's performance under $\\mathcal{NC_1^{(w)}}$, I cannot tell there are significant differences among Table 1-3. Similarly, with $\\mathcal{(U)NC_3}$, for the gap of ASE and BEC, it's still hard to tell the difference among these three tables.\n3. In section 4.1.2, could the authors clarify why the difference in association scores between female and male groups is measured? If the association score is defined as $\\log \\frac{P_T}{P_\\text{prior}}$, I feel it's more reasonable to measure the absolute value of the association score since if the absolute value is close to 0, it should suggest that <person word> predictions are less likely influenced by <profession>, which indicates a less biased LM.\n4. All experiments in this paper are conducted on masked language models (e.g, BERT and RoBERTa). Could the proposed method also generalize to casual language models, as explored in the Linguistic Collapse[1] paper?\n\n[1] Wu, Robert, and Vardan Papyan. \"Linguistic Collapse: Neural Collapse in (Large) Language Models.\" arXiv preprint arXiv:2405.17767 (2024).", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Analyzing the debiased language models from the perspective of neural collapse is novel. This offers valuable insights into the structure of the model's embedding space across different debiasing approaches.\n2. The proposed method is simple but effective and can be easily integrated with other approaches. It's interesting to see that by only adding a NC-based regularization term in the loss function, all the compared debiasing methods can be enhanced.\n3. The experiments are comprehensive and well-motivated, with the proposed method grounded in observations of $\\mathcal{NC}$ perspectives on both biased and debiased LMs, effectively establishing a strong rationale for the proposed method, making it clear and convincing.", "weaknesses": "1. **Clarify of Notations**: Some notations are not clearly defined, which could impact readability.\n    - The class embedding variances mentioned on line 161 and line 240 could benefit from a more explicit definition, similar to how it is presented in Eq. 3 of the Linguistic Collapse[1] paper. \n    - In Eq. 1, to align with the token representation defined at line 137, the notation should ideally be ${h}(E(x_{1:t}))$ rather than ${h}(x_{1:t})$ for clarity and consistency. \n2. **Evaluation Metrics**: I have concerns regarding the evaluation metrics used in section 3.3 and section 4.2.1. Please see question 2 and 3 in the following section.\n3. **Definition of Unfairness**: Some definition of unfairness were not explicitly defined, which could make the paper less accessible to readers unfamiliar with the fairness literature. For example, in the *BEC-Pro* subsection of section 4.1.2, the average association score is used to measure model unfairness. While this is defined in [2] as $\\log \\frac{P_T}{P_\\text{prior}}$, it's not explicitly explained here, which may cause confusion.\n\n[1] Wu, Robert, and Vardan Papyan. \"Linguistic Collapse: Neural Collapse in (Large) Language Models.\" arXiv preprint arXiv:2405.17767 (2024).    \n[2] Bartl, Marion, Malvina Nissim, and Albert Gatt. \"Unmasking contextual stereotypes: Measuring and mitigating BERT's gender bias.\" arXiv preprint arXiv:2010.14534 (2020).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dF9CslNVJg", "reviewer": "Reviewer_adpT", "summary": "### Summary\n\n- The paper studies the relationship between neural collapse and fairness and tries to answer two research questions\n\t- \"Do debiased LMs exhibit greater neural collapse\"\n\t- \"Can this inductive bias be leveraged to improve fairness\"\n- Background on neural collapse:\n\t- This is a phenomenon that happens in the last layer representations of classifiers.\n\t- Since the last layer of LM predicts the next word, it can be viewed as a classifier\n\t- The phenomenon of neural collapse causes the representations of certain gender sensitive words to collapse closer to their class means in debiased LMs\n- The paper presents 4 ways to quantify neural collapse borrowing it from prior work. These metrics measure separability between classes.\n\t- NC1: intra-class variance over inter class variance\n\t- NC2: Measures separability from a geometric perspective\n\t- NC3: Uses dot products\n\t- NC4: Uses a linear projection\n\nIn section 3.2.1 the paper provides a worry about diverging values for various NC metrics but convincingly provides a resolution. This is achieved by calibrating the NC metrics by replacing class means with classifier weights in the expressions.\n- The experimental results justify the claim of the presence of neural collapse for gender words.\n- The second part of the paper deals with proposing a bias mitigation strategy by explicitly enforcing Neural collapse. To this end, they use the NC3 formulation as an auxiliary objective during fine-tuning.\n- Both intrinsic and extrinsic metrics show improvements with this modified objective.\n\nPlease consider citing the following additional work in the field of LM fairness and evaluation of bias in LMs\n- https://arxiv.org/pdf/2208.01448#page=14.61\n- https://aclanthology.org/2022.findings-acl.55.pdf\n- https://aclanthology.org/2022.acl-long.401.pdf\n \nnits\n- 3.2.2 CALIBRATIONS -> CALIBRATION\n- Section 4.1.3\n\t- TPR-1 (Type 1: pro-stereotypical minus anti-stereotypical) and TPR-2 (Type 2: pro-stereotypical minus anti-stereotypical) -> TPR-1 (Type 1: pro-stereotypical minus anti-stereotypical) and TPR-2 (Type 2: pro-stereotypical minus anti-stereotypical) reads a bit odd. I think Type 1 are ambiguous sentences and Type 2 are non-ambiguous sentences. Explicitly call this out with a simple example.", "questions": "No questions, see weakness section above.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "### Strengths\n\n- The paper studies a missing link between neural collapse and fairness and proposes a bias mitigation strategy based on neural collapse objective that is agnostic to pre-training or fine-tuning methods. This also avoids manual data balancing or filtering.\n- Both intrinsic and extrinsic fairness metrics see improvements with this approach.", "weaknesses": "### Weakness\n\n- Table 9 does not have the best results bold which makes it reading a little harder. \n- While the paper is well-written, simple to follow and proposes a simple technique which works, all the experiments conducted are on BERT based MLM based models. I understand this is intentional to ensure fair comparison with prior work but this also limits understanding the degradation on non MLM tasks. For eg. in Table 9, the results reported are on the GLUE benchmark which are all multiple choice questions. I am curious if this technique can really prove to be a generic technique even for generation tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QoPqOqPGdB", "reviewer": "Reviewer_fRsu", "summary": "This paper proposes a method to enhance model fairness based on the phenomenon of Neural Collapse (NC). The authors first analyze NC in the context of fairness-related words and find that debiased language models exhibit more aligned token representations with word embeddings. Inspired by this observation, they introduce a regularization method based on NC to reduce bias in gender-sensitive words during fine-tuning. The experiments demonstrate that this method can improve fairness across various debiasing algorithms while maintaining the language model's performance on standard natural language understanding tasks.", "questions": "Clarifications Needed: \n1. The σ symbol on line 161. \n2. On line 235, the authors hypothesize that \"noises in language data and the complexity of different fine-tuning methods make measurements of NC1/2/4 unstable and inconsistent.\" Could the authors elaborate on why they consider these factors to be influential? \n3. Could the authors justify the rationale behind replacing class means with classifier weights in NC1/2?\n4. Besides gender-related words, have the authors explored other words that might cause the model to collapse more? This experiment would complement Section 3.3.\n5. I am curious about the effectiveness of the proposed method in fairness scenarios beyond gender. Firstly, what if it is challenging to obtain relevant word lists? Secondly, will the model’s general performance being affected by other word lists? I think the authors could counduct experiments by adding random words/high frequency words to the word list.\n6. Can the authors provide further insights into the results of the visualization in Section 5? Why do they consider this to be a better feature distribution?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper is logically structured and easy to follow, with a clear progression from problem statement to method proposal and experimental validation.", "weaknesses": "1. In Section 4, the details of the datasets and metrics can be more concisely presented, with some details moved to the appendix. Conversely, the discussion of results is somewhat inadequate. It would be beneficial to highlight the advantages of the proposed method in comparison to the baselines.\n2. Additional analysis should be included in section 4.2. From the results alone, it is observed that many outcomes with the addition of (U)NC3 have significantly decreased.\n3. It is unclear whether the proposed method is also effective for LLMs.\n4. It is uncertain whether the method remains effective for aspects of fairness beyond gender", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yW0pvmzpV2", "reviewer": "Reviewer_6ic8", "summary": "This paper investigates the relationship between the neural collapse phenomena and fairness in language models. The paper demonstrates that debiased language models naturally exhibit more \"collapsed\" representations for fairness-sensitive words, and leverage this insight to develop a neural collapse-based regularization method. The proposed approach enhances fairness across different debiasing methods while maintaining model performance on general language tasks.", "questions": "1. Can this approach generalize to non-binary protected attributes? If so, how?\n\n2. How does the method work for decoder-only models, like gpt and llama?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1.the paper draws the connection between neural collapse theory and fairness in language models in a principal way. It provides novel analytical tools for understanding debiasing mechanisms and offers new insights into how representation geometry relates to model fairness. \n\n2.  Comprehensive empirical experiments across multiple models and datasets, demonstrating the practical utility of the proposed approach.\n\n3. The proposed regularization method seems to be simple yet effective. It can be easily integrated with existing debiasing approaches without significant computational overhead, making it highly practical for real-world applications. \n\n3. The empirical evaluation is thorough, covering both intrinsic and extrinsic fairness metrics. The results clearly demonstrate improvements across various benchmarks while preserving model performance on downstream tasks. \n\n5. The authors also provide helpful visualizations and analyses that illuminate how their method affects model representations.", "weaknesses": "1. The primary weakness of the paper lies in its theoretical foundation. While the empirical results are promising, the authors don't provide sufficient theoretical justification for why NC3 and calibrated metrics work better than others. The paper would benefit from a formal analysis or proof connecting neural collapse to fairness improvements. \n\n2. The scope of the evaluation is somewhat limited, focusing primarily on **binary** gender bias. While this allows for deep analysis in one area, it leaves questions about the method's generalizability to other (possibly non-binary) demographic groups.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)