Collapsed Language Models Promote Fairness

## Abstract

Abstract To mitigate societal biases implicitly encoded in recent successful pretrained language models, a diverse array of approaches have been proposed to encourage model fairness, focusing on prompting, data augmentation, regularized fine-tuning, and more.
Despite the development, it is nontrivial to reach a principled understanding of fairness and an effective algorithm that can consistently debias language models.
In this work, by rigorous evaluations of Neural Collapse – a learning phenomenon happen in last-layer representations and classifiers in deep networks – on fairness-related words, we find that debiased language models exhibit collapsed alignment between token representations and word embeddings.
More importantly, this observation inspires us to design a principled fine-tuning method that can effectively improve fairness in a wide range of debiasing methods, while still preserving the performance of language models on standard natural language understanding tasks.
We attach our code at https://github.com/Xujxyang/Fairness-NC-main .

## 1 Introduction

The rise of pre-trained language models (PLMs) has revolutionized natural language processing, greatly enhancing tasks like reasoning and prediction by harnessing the semantic richness of language data. Despite their effectiveness, these models, trained on extensive corpora, often reflect and even intensify societal biases in their training datasets.
Such biases manifest in the association of demographic groups with specific roles or capabilities, affecting fairness in applications ranging from legal analytics to hiring processes [; ; ; ; ; ; ].
Thus, it is crucial to address and mitigate these biases to prevent discriminatory practices in downstream applications [; ; ].

To mitigate societal biases in language models, a substantial array of fairness algorithms has been proposed.
On the one hand, people target different learning stages: making language models fair via balanced and augmented training data [], fine-tuning with regularizations or auxiliary objectives [; ], or carefully-tuned prompts [].
On the other hand, these debiasing approaches can also be categorized by their awareness of downstream tasks. Task-specific methods fine-tune language models with sensitive annotations [; ; ; ], while task-agnostic approaches directly debias word embeddings or representations during pretraining [; ; ; ].
Despite this multitude of efforts, it is challenging to find common ground among these methods and shared properties of debiased language models.
We are thus motivated to ask: Can we understand and improve the fairness of LMs in principle?

Figure: Figure 1: Debiased LMs show more collapsed alignment between classifiers ($\bm{w}$) and class means ($\bm{\mu}$). See $(\mathcal{U})\mathcal{NC}_{3}$ in Table 1.
Refer to caption: x1.png

The recent development of deep learning theory provides fruitful frameworks and tools for us to understand deep neural networks (DNNs). Among them, neural collapse [] is first observed for classification tasks, and is then analyzed to understand the optimization [; ] and generalization [; ] of DNNs.
Meanwhile, the training of generative language models, typically via the next token prediction task, is essentially a classification problem.
We thus study the fairness of LMs through the lens of neural collapse.

We ask two specific questions:

Q1: Do debiased LMs commonly exhibit greater collapse?
Q2: Can we leverage this inductive bias to improve LM fairness in principle?

Motivated by these two questions, we try to find connections between neural collapse and fairness of language models.
We observed, as expected, that fairer language models show more collapsed representations of gender-sensitive words, indicated by greater alignment between classifiers (word embeddings) and class means (token representations) (see Figure 1).
This behavior is consistently and implicitly exhibited across a wide range of popular debiasing methods, suggesting a commonly shared perspective on analyzing fairness in language models.
This observation further inspires us to explicitly enforce neural collapse to promote fairness in pretrained language models.
Since this explicit collapse is principled and does not introduce any customization in fine-tuning or data augmentation, it can be universally applied to enhance many existing fairness approaches, with little implementation or computation overhead.
We summarize our contributions below:

- •
We for the first time comprehensively analyze the relations between neural collapse and fairness in language models.
- •
Our empirical analysis motivates us to introduce a regularization based on neural collapse, which is extremely simple, and agnostic to any fine-tuning method or augmentation of language data.
- •
Comprehensive experiments on both intrinsic and extrinsic evaluations demonstrate that our regularization can consistently debias language models. It is orthogonal to a wide range of highly tailored fairness algorithms, and thus can be plug-and-play adopted without sacrificing the models’ performance on typical downstream language tasks.

## 2 Related Works

### 2.1 Neural Collapse

A learning phenomenon called neural collapse ($\mathcal{NC}$) arises during the terminal phase of training neural networks with cross-entropy (CE) loss for classification tasks []. It was initially defined by the simultaneous emergence of multiple properties in the model’s top-layer features (also referred to as last-layer representations or embeddings) and classifiers, including the variability and geometry of class-wise averaged features, the alignment between features and classifier weights, and the collapse to nearest-neighbor classifier. Since then, $\mathcal{NC}$ has been theoretically analyzed to understand the optimization [; ] and generalization [; ] of DNNs.
$\mathcal{NC}$ inspires numerous applications and empirical studies, including transfer learning [], privacy [; ], data imbalance [; ], and outlier detection [; ; ]

Meanwhile, DNN behaviors related to $\mathcal{NC}$ are also observed in various settings. Several studies have explored $\mathcal{NC}$ under various loss functions. For example, [] examined $\mathcal{NC}$ in the context of Mean Squared Error (MSE) loss, while [] demonstrated that $\mathcal{NC}$ also arises with label smoothing and focal loss. Additionally, researchers like [] and  [] have investigated $\mathcal{NC}$ properties in intermediate layers as well. More recently, [] observed the linearity of $\mathcal{NC}$ in language models and provided some quantitative explanations for this phenomenon.

### 2.2 Fairness in Language Models

Language Models (LMs), extensively studied in academic literature and widely adopted across various applications, have recently raised concerns regarding fairness [; ].
For medium-sized LMs, such as BERT [] and Roberta [], fairness elimination methods have been developed from both data and training perspectives.
Given that label imbalance across different demographic groups in the training data is a significant source of bias, a common data processing technique is to balance labels using Counterfactual Data Augmentation (CDA) [; ].
Some research bridges robustness and fairness by augmenting a robust training set with techniques such as robust word substitution  [] and counterfactual logit pairing  [].
For parameter-efficient methods, GEEP [] and Adept [] incorporated gender equality prompts into LLMs using trainable embeddings of occupation names.
In addition to retraining, FairBERTa [] showed that fine-tuning language models on the demographic perturbation dataset PANDA can enhance fairness in downstream tasks.
For large LMs, such as Llama [] and GPT [],
recent research not only focuses on fine-tuning the model itself but also emphasizes lightweight post-processing techniques to address fairness concerns,
including instruction fine-tuning [; ]
and prompt engineering [; ].

##### Representative Debiased Language Models.

In this paper, we choose to focus on studying the following three representative works, mainly because: 1) They debiased language models from orthogonal perspectives; 2) They all targeted debiasing the BERT model so they can be more fairly compared to each other.

- •
Data Preparation: BEC [] focused on developing a customized Bias Evaluation Corpus (BEC) via counterfactual data substitution, and studied associations between gender-denoting target words and names of professions [].
- •
Fine-tuning Method: Mabel [] augmented premises and hypotheses from the natural language inference (NLI) dataset with counterfacts, and applied a contrastive learning objective on gender-balanced entailment pairs to fine-tune BERT.
- •
Regularization: ASE [] proposed incorporating the fairness objective into the training process of downstream tasks through two regularization terms (stereotype neutralization and prevention of catastrophic forgetting) beyond the task objective to encourage the fairness.

## 3 Debiased Language Models are More Collapsed

### 3.1 Preliminary: Why Do Language Models Collapse?

Suppose the whole vocabulary is the set of word indices $\mathbb{V}=[1,2,\cdots,C]$. In language models, a sequence of tokens (indices) $\bm{x}_{1:t}\in\mathbb{V}^{t}$ are embedded by the word embedding layer $\bm{E}$ and are forwarded through layers.
In the context of next-token prediction for language models, the penultimate token representations $\bm{h}\left(\bm{E}(\bm{x}_{1:t})\right)\in\mathbb{R}^{d}$ is used to predict the next token $\hat{x}_{t+1}\in\mathbb{V}$ by comparing with classifier weights $\{\bm{w}_{c}\in\mathbb{R}^{d}|c=1,\cdots,C\}$ and also the bias term:

$$ $\hat{x}_{t+1}:=\underset{c\in\mathbb{V}}{\operatorname{argmax}}\left\langle\bm {w}_{c},\bm{h}\left(\bm{x}_{1:t}\right)\right\rangle+\bm{b}_{c}$ (1) $$

We further denote the mean token representation $\bm{\mu}_{c}\in\mathbb{R}^{d}$
whose ground-truth next token is $x_{t+1}^{(s)}=c$.
We also focus on the centered means:

$$ $\bm{\mu}_{c}:=\frac{1}{N_{c}}\sum_{s=1}^{S}\sum_{t=1}^{T-1}\bm{h}\left(\bm{E}( \bm{x}_{1:t}^{(s)})\right)\mathbb{I}\left(x_{t+1}^{(s)}=c\right),\quad \overline{\bm{\mu}}:=\frac{1}{C}\sum_{c=1}^{C}\bm{\mu}_{c},$ (2) $$

where $N_{c}$ is the number of samples of class $c$, $\mathbb{I}$ is the (binary) indicator function, $S$ is total number of sentences in the dataset, and $T$ is the total number of tokens in a sentence.

Neural collapse ($\mathcal{NC}$) [] is originally observed when neural networks are solving classification problems, where the representations from the penultimate layer and the final classifier weights collapse into certain geometric structures.

##### Neural Collapse in Language Models.

From the perspective of token predictions (Eq. 1), we can see that the pretraining and fine-tuning of language model are actually classification tasks,
and the word embedding layer $\bm{E}=\{\bm{w}_{c}\in\mathbb{R}^{d}|c=1,\cdots,C\}$ is typically the classifier.
Therefore, we can expect neural collapse also happens in language models, between the mean embedding $\bm{\mu}_{c}$ and the word embedding layer $\bm{E}$.

Specifically, inherited from papyan2020prevalence [], the neural collapse behavior in Language Models can also be defined from four perspectives []:

- •
$\mathcal{N}\mathcal{C}_{1}$ measures the separability between classes via intra-class variability over inter-class distance: $\mathcal{N}\mathcal{C}_{1}:=\mathbb{E}_{c,c^{\prime}}\frac{\sigma_{c}^{2}+
\sigma_{c^{\prime}}^{2}}{2\left\|\bm{\mu}_{c}-\bm{\mu}_{c^{\prime}}\right\|_{2
}^{2}}\quad(\forall c\neq c^{\prime})$. A less $\mathcal{N}\mathcal{C}_{1}$ indicates a more collapsed classifier.
- •
$(\mathcal{G})\mathcal{NC}_{2}$ measures the separability from the geometric ($\mathcal{G}$) perspective, where the class means tend to become equinorm and equiangular vectors and formulate a simplex known as equiangular tight frame (ETF). This can further be relaxed to account for noises and imbalances in practice: $(\mathcal{G})\mathcal{NC}_{2}:=\mathbb{E}_{c,c^{\prime}}\log\left\|\frac{\bm{
\mu}_{c}-\overline{\bm{\mu}}}{\left\|\bm{\mu}_{c}-\overline{\bm{\mu}}\right\|_
{2}}-\frac{\bm{\mu}_{c^{\prime}}-\overline{\bm{\mu}}}{\left\|\bm{\mu}_{c^{
\prime}}-\overline{\bm{\mu}}\right\|_{2}}\right\|^{-1}\quad(\forall c\neq c^{
\prime})$. A less $(\mathcal{G})\mathcal{NC}_{2}$ indicates more expanded and collapsed class means.
- •
$(\mathcal{U})\mathcal{NC}_{3}$ quantifies the alignment between classifiers and class means, termed as “uniform ($\mathcal{U}$) duality”. We first measure $\left\langle\frac{\bm{w}_{c}}{\left\|\bm{w}_{c}\right\|_{2}},\frac{\bm{\mu}_{c
}-\overline{\bm{\mu}}}{\left\|\bm{\mu}_{c}-\overline{\bm{\mu}}\right\|_{2}}\right\rangle$ for each class, and then calculate
their standard deviations $\text{std}(\cdot)$ across classes.
A less $(\mathcal{U})\mathcal{NC}_{3}$ indicates class means are more consistently correlated with the classifier, i.e., more collapsed.
- •
$\mathcal{NC}_{4}$ simplified the linear projection in Eq. 1 into nearest-class center (NCC) classifier $\underset{c\in\mathbb{V}}{\operatorname{argmax}}\left\langle\bm{w}_{c},\bm{h}
\right\rangle+\bm{b}_{c}\rightarrow\underset{c\in\mathbb{V}}{\operatorname{
argmin}}\left\|\bm{h}-\bm{\mu}_{c}\right\|_{2}$ and further measure the token prediction accuracy by nearest neighbors. A greater $\mathcal{NC}_{4}$ indicates a stronger (more collapsed) NCC classifier.

### 3.2 Debiased LMs are More Collapsed Than Biased LMs

In the context of fairness of language models, the main idea is to pursue debiased word embeddings.
For example, people measure the geometry of gender-related tokens in the embedding space [; ; ] and try to remove gender-related information of stereotypical words and mitigate biases in the word embedding space.
To counteract artifacts from training that leads to the encoding of stereotypes, people even adopt static embeddings [; ; ; ].

Inspired by these previous works, we are expecting to see more collapse between the word embedding layer and token representations in debiased language models.

#### 3.2.1 𝒩 ⁢ 𝒞 𝒩 𝒞 \mathcal{NC} caligraphic_N caligraphic_C Metrics in Debiased LMs

We first evaluate $\mathcal{NC}$ metrics in popular debiased language models and compare them with their biased baselines.

##### Settings.

Following previous works, we measure $\mathcal{NC}$ metrics based on different training datasets used in each work, concerning the subset of the whole vocabulary of only gender-related words, dubbed $\mathbb{V}_{\text{gender}}$
(see Appendix A for the full list of words).
The detailed datasets used by these works are listed as follows:
Mabel on SNLI [] and MNLI []; ASE on OntoNotes []; BEC on TinyStories [];
Among these, only BEC’s training dataset [] is relatively small, while the datasets in other works exceed 100K sentences. To ensure meaningful comparisons, we evaluated BEC on a larger dataset TinyStories [].

We show this result in Table 1. All methods start from the same pretrained BERT model. However, each work studied the BERT model on its own training data, leading to different $\mathcal{NC}$ measurements for the same BERT model.
From Table 1, we can see that debiased language models exhibit neural collapse in certain perspectives: $\mathcal{NC}3$ is consistently improved (minimized) in debiased models, whereas $\mathcal{NC}1/2/4$ are diverging.
This indicates that the alignments between token representations (“class means”) and debiased word embeddings (“classifier weights”) are more consistent, as illustrated in Figure 1.
For evaluations of additional models, please refer to Appendix C.

Our explanations are as follows.
The neural collapse behavior manifests under certain conditions [], including: 1) models are trained towards zero training loss; 2) clean labels with balanced classes; 3) the number of classes is not greater than the model’s hidden dimension.
However, these conditions are commonly violated in practice: 1) The training loss is difficult to be minimized to zero due to the complexity of language data; 2) The occurrence of tokens in $\mathbb{V}$ is highly unbalance due to the nature of languages; 3) Not all language models have greater hidden sizes than the vocabulary size(^1^11Eg. The hidden size of BERT is 768, which is smaller than the vocabulary size of 30,522.).
Therefore, token representations and word embeddings are typically not balanced and well-trained in practical language models, and thus not all perspectives of neural collapse ($\mathcal{NC}1/2/3/4$) can be consistently observed in debiased models.
Instead, in the next section, we study how to calibrate these metrics to be more consistent with fairness encoded in models.

**Table 1: $(\mathcal{U})\mathcal{NC}_{3}$ is consistently improved (minimized) on all debiasing methods. Metrics are measured on the subset of the whole vocabulary of only gender-related words $\mathbb{V}_{\text{gender}}$ (Appendix A). In these three groups, each pair of “(BERT, debiased model)” is tested on a customized dataset as detailed in Section 3.2.1, leading to different measurements for the same pretrained BERT model.**
| Model | $\mathcal{N}\mathcal{C}_{1}\downarrow$ | $(\mathcal{G})\mathcal{NC}_{2}\downarrow$ | $(\mathcal{U})\mathcal{NC}_{3}\downarrow$ | $\mathcal{NC}_{4}\uparrow$ | $\mathcal{N}\mathcal{C}_{1}^{(\bm{w})}\downarrow$ | $(\mathcal{G})\mathcal{NC}_{2}^{(\bm{w})}\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| BERT | 0.967 | 0.148 | 0.096 | 2.799 | 827.8 | 0.337 |
| Mabel | 0.786${}_{{\color[rgb]{0.0,0.5,0.0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.0,0.5,0.0}~{}0.181\uparrow}}$ | 0.145${}_{{\color[rgb]{0.0,0.5,0.0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.0,0.5,0.0}~{}0.003\uparrow}}$ | 0.070${}_{{\color[rgb]{0.0,0.5,0.0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.0,0.5,0.0}~{}0.026\uparrow}}$ | 2.235${}_{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}~{}0.564<br>\downarrow}}$ | 724.3${}_{{\color[rgb]{0.0,0.5,0.0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.0,0.5,0.0}~{}103.5\uparrow}}$ | 0.331${}_{{\color[rgb]{0.0,0.5,0.0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.0,0.5,0.0}~{}0.006\uparrow}}$ |
| BERT | 2.430 | 0.051 | 0.063 | 1.113 | 1134.7 | 0.364 |
| ASE | 3.008${}_{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}~{}0.578<br>\downarrow}}$ | 0.500${}_{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}~{}0.449<br>\downarrow}}$ | 0.056${}_{{\color[rgb]{0.0,0.5,0.0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.0,0.5,0.0}~{}0.007\uparrow}}$ | 0.023${}_{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}~{}1.090<br>\downarrow}}$ | 382.6${}_{{\color[rgb]{0.0,0.5,0.0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.0,0.5,0.0}~{}752.1\uparrow}}$ | 0.372${}_{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}~{}0.008<br>\downarrow}}$ |
| BERT | 2.358 | 0.152 | 0.062 | 10.44 | 1032.7 | 0.359 |
| BEC | 2.509${}_{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}~{}0.151<br>\downarrow}}$ | 0.185${}_{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}~{}0.033<br>\downarrow}}$ | 0.056${}_{{\color[rgb]{0.0,0.5,0.0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.0,0.5,0.0}~{}0.006\uparrow}}$ | 7.442${}_{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}~{}2.998<br>\downarrow}}$ | 1015.6${}_{{\color[rgb]{0.0,0.5,0.0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.0,0.5,0.0}~{}17.1\uparrow}}$ | 0.335${}_{{\color[rgb]{0.0,0.5,0.0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.0,0.5,0.0}~{}0.024\uparrow}}$ |

#### 3.2.2 Calibrations of 𝒩 ⁢ 𝒞 ⁢ 1 𝒩 𝒞 1 \mathcal{NC}1 caligraphic_N caligraphic_C 1 and 𝒩 ⁢ 𝒞 ⁢ 2 𝒩 𝒞 2 \mathcal{NC}2 caligraphic_N caligraphic_C 2

When we further analyze Table 1 to understand why only $\mathcal{NC}3$ is reduced in debiased LMs as we expected, we find that only $\mathcal{NC}3$ involves the classifier weights $\bm{w}_{c}$, but $\mathcal{NC}1/2/4$ all consider class means $\bm{\mu}_{c}$.
We hypothesize that noises in language data and the complexity of different fine-tuning methods make measurements of $\mathcal{NC}1/2/4$ unstable and inconsistent.

To calibrate these representation-based collapse metrics, we further study replacing class means with classifier weights in $\mathcal{NC}1/2$.

- •
$\mathcal{N}\mathcal{C}_{1}^{(\bm{w})}:=\mathbb{E}_{c,c^{\prime}}\frac{\sigma_{
c}^{2}+\sigma_{c^{\prime}}^{2}}{2\left\|\bm{w}_{c}-\bm{w}_{c^{\prime}}\right\|
_{2}^{2}}\quad(\forall c\neq c^{\prime})$.
- •
$(\mathcal{G})\mathcal{NC}_{2}^{(\bm{w})}:=\mathbb{E}_{c,c^{\prime}}\log\left\|
{\bm{w}_{c}}-{\bm{w}_{c^{\prime}}}\right\|^{-1}\quad(\forall c\neq c^{\prime})$.

Note that we cannot calibrate $\mathcal{NC}4$ in this way since that will trivially reduce $\mathcal{NC}4$ back to the standard token prediction (Eq. 1).

We show the results of these calibrated $\mathcal{NC}1/2$ in Table 1 (right two columns).
Debiased language models exhibit either comparable or more collapsed measurements on $\mathcal{N}\mathcal{C}_{1}^{(\bm{w})}$ and $(\mathcal{G})\mathcal{NC}_{2}^{(\bm{w})}$, showing greater consistency than the uncalibrated $\mathcal{N}\mathcal{C}_{1}$ and $(\mathcal{G})\mathcal{NC}_{2}$ metrics. This suggests that these calibrated metrics more reliably capture neural collapse in debiased language models.

### 3.3 Debiased LMs Are More Collapsed in Fairness-Sensitive Words

Beyond using different versions of metrics to quantify neural collapse in language models, we further study the impact of different choices of fairness-sensitive words on neural collapse. Our core questions are:

- 1.
Do debiased language models collapse more across the whole vocabulary, or only on fairness-sensitive words?
- 2.
Will the size of subsets of words affect the comparison of $\mathcal{NC}$ metrics?

To answer the above two questions, we calculate $\mathcal{NC}$ metrics on both the whole vocabulary set, and also a random vocabulary subset of the same size as that we used Table 1 (Appendix A). Our gender word list includes 210 female-related words and 215 male-related words.

From Table 2 and Table 3, it is evident that gaps of $\mathcal{NC}$ metrics between BERT and debiased BERT models are much smaller than gaps in Table 1.
This indicates that debiased BERT models exhibit more different token representations and word embeddings only on gender-related vocabulary, and they perform much more similarly with BERT on the whole vocabulary.
This also explains why neural collapse cannot determine if a language model is debiased or not across the whole vocabulary, as none of the original $\mathcal{NC}$1/2/3/4 metrics show consistent improvement in Table 2.
Moreover, as shown in Table 3, this observation also holds on a random subset of vocabulary with matched size with gender-related words in Table 1.
Meanwhile, in this context, $\mathcal{NC}$ measurements are generally smaller than those measured on only gender words, which implies that it is generally challenging to learn more separable representations of gender words compared with others insensitive to fairness.

**Table 2: $\mathcal{NC}$ metrics of different debiased language models on the whole vocabulary $\mathbb{V}$. Gaps of $\mathcal{NC}$ metrics between BERT and debiased BERT models are much smaller than gaps in Table 1.**
| Model | $\mathcal{N}\mathcal{C}_{1}\downarrow$ | $(\mathcal{G})\mathcal{NC}_{2}\downarrow$ | $(\mathcal{U})\mathcal{NC}_{3}\downarrow$ | $\mathcal{NC}_{4}\uparrow$ | $\mathcal{N}\mathcal{C}_{1}^{(\bm{w})}\downarrow$ | $(\mathcal{G})\mathcal{NC}_{2}^{(\bm{w})}\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| BERT | 0.340 | 0.248 | 0.056 | 0.712 | 446.8 | 0.407 |
| Mabel | 0.293${}_{{\color[rgb]{0.0,0.5,0.0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.0,0.5,0.0}~{}0.047\uparrow}}$ | 0.240${}_{{\color[rgb]{0.0,0.5,0.0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.0,0.5,0.0}~{}0.008\uparrow}}$ | 0.064${}_{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}~{}0.006<br>\downarrow}}$ | 0.865${}_{{\color[rgb]{0.0,0.5,0.0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.0,0.5,0.0}~{}0.153\uparrow}}$ | 383.7${}_{{\color[rgb]{0.0,0.5,0.0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.0,0.5,0.0}~{}63.1\uparrow}}$ | 0.397${}_{{\color[rgb]{0.0,0.5,0.0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.0,0.5,0.0}~{}0.010\uparrow}}$ |
| BERT | 0.922 | 0.212 | 0.062 | 0.125 | 806.9 | 0.413 |
| ASE | 1.321${}_{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}~{}0.329<br>\downarrow}}$ | 0.417${}_{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}~{}0.205<br>\downarrow}}$ | 0.049${}_{{\color[rgb]{0.0,0.5,0.0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.0,0.5,0.0}~{}0.013\uparrow}}$ | 0.002${}_{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}~{}0.123<br>\downarrow}}$ | 193.1${}_{{\color[rgb]{0.0,0.5,0.0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.0,0.5,0.0}~{}613.8\uparrow}}$ | 0.418${}_{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}~{}0.005<br>\downarrow}}$ |
| BERT | 1.308 | 0.009 | 0.050 | 0.734 | 815.9 | 0.415 |
| BEC | 1.334${}_{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}~{}0.026<br>\downarrow}}$ | 0.011${}_{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}~{}0.002<br>\downarrow}}$ | 0.047${}_{{\color[rgb]{0.0,0.5,0.0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.0,0.5,0.0}~{}0.003\uparrow}}$ | 0.516${}_{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}~{}0.218<br>\downarrow}}$ | 798.4${}_{{\color[rgb]{0.0,0.5,0.0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.0,0.5,0.0}~{}17.5\uparrow}}$ | 0.408${}_{{\color[rgb]{0.0,0.5,0.0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.0,0.5,0.0}~{}0.007\uparrow}}$ |

**Table 3: $\mathcal{NC}$ metrics of different debiased language models on the same number of words as used in Table 1 ($\mathbb{V}_{\text{gender}}$) but words are randomly selected. Gaps of $\mathcal{NC}$ metrics between BERT and debiased BERT models are much smaller than gaps in Table 1.**
| Model | $\mathcal{N}\mathcal{C}_{1}\downarrow$ | $(\mathcal{G})\mathcal{NC}_{2}\downarrow$ | $(\mathcal{U})\mathcal{NC}_{3}\downarrow$ | $\mathcal{NC}_{4}\uparrow$ | $\mathcal{N}\mathcal{C}_{1}^{(\bm{w})}\downarrow$ | $(\mathcal{G})\mathcal{NC}_{2}^{(\bm{w})}\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| BERT | 0.320 | 0.248 | 0.057 | 0.407 | 434.3 | 0.400 |
| Mabel | 0.314${}_{{\color[rgb]{0.0,0.5,0.0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.0,0.5,0.0}~{}0.006\uparrow}}$ | 0.237${}_{{\color[rgb]{0.0,0.5,0.0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.0,0.5,0.0}~{}0.011\uparrow}}$ | 0.060${}_{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}~{}0.003<br>\downarrow}}$ | 1.910${}_{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}~{}1.503<br>\downarrow}}$ | 406.4${}_{{\color[rgb]{0.0,0.5,0.0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.0,0.5,0.0}~{}27.9\uparrow}}$ | 0.392${}_{{\color[rgb]{0.0,0.5,0.0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.0,0.5,0.0}~{}0.008\uparrow}}$ |
| BERT | 1.086 | 0.197 | 0.059 | 0.235 | 859.7 | 0.406 |
| ASE | 1.158${}_{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}~{}0.072<br>\downarrow}}$ | 0.404${}_{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}~{}0.207<br>\downarrow}}$ | 0.046${}_{{\color[rgb]{0.0,0.5,0.0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.0,0.5,0.0}~{}0.013\uparrow}}$ | 0.000${}_{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}~{}0.235<br>\downarrow}}$ | 167.1${}_{{\color[rgb]{0.0,0.5,0.0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.0,0.5,0.0}~{}692.6\uparrow}}$ | 0.436${}_{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}~{}0.030<br>\downarrow}}$ |
| BERT | 1.555 | 0.021 | 0.047 | 0.145 | 866.7 | 0.411 |
| BEC | 1.501${}_{{\color[rgb]{0.0,0.5,0.0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.0,0.5,0.0}~{}0.054\uparrow}}$ | 0.031${}_{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}~{}0.010<br>\downarrow}}$ | 0.044${}_{{\color[rgb]{0.0,0.5,0.0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.0,0.5,0.0}~{}0.003\uparrow}}$ | 1.079${}_{{\color[rgb]{0.0,0.5,0.0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.0,0.5,0.0}~{}0.934\uparrow}}$ | 822.5${}_{{\color[rgb]{0.0,0.5,0.0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.0,0.5,0.0}~{}44.2\uparrow}}$ | 0.405${}_{{\color[rgb]{0.0,0.5,0.0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.0,0.5,0.0}~{}0.006\uparrow}}$ |

## 4 Bias Mitigation via Enforcing Explicit Collapse in LMs

Motivated by our observations in Section 3, we further ask: can we enforce explicit neural collapse in language models and thus improve their fairness?

We propose minimizing the regularization of language models using $(\mathcal{U})\mathcal{NC}_{3}$ as an auxiliary objective during fine-tuning:

$$ $\mathcal{L}_{\mathcal{NC}_{3}}=\text{std}\left(\left\langle\frac{\bm{w}_{c}}{ \left\|\bm{w}_{c}\right\|_{2}},\frac{\bm{\mu}_{c}-\overline{\bm{\mu}}}{\left\| \bm{\mu}_{c}-\overline{\bm{\mu}}\right\|_{2}}\right\rangle\right),\quad c\in \mathbb{V}_{\text{gender}}.$ (3) $$

Although sounds straightforward, this principled approach could potentially be very important in improving the fairness of language models from two perspectives:

- 1.
Our method is simple, principled, and is agnostic to any pretraining or fine-tuning methods for fairness. As we will show in our results, it can be adopted in a wide range of fairness algorithms in a plug-and-play fashion.
- 2.
Our method can avoid manual filtration or augmentation of the underlying language training data.

In the following experiments, we demonstrate that our regularization consistently de-biases language models across different fairness metrics (Sec. 4.1), while still preserving models’ language modeling performance (Sec. 4.2).
We also provide ablation studies on the strength of this regularization in Appendix B.

### 4.1 Enforcing 𝒩 ⁢ 𝒞 𝒩 𝒞 \mathcal{NC} caligraphic_N caligraphic_C Promotes Fairness, Both Intrinsically and Extrinsically

#### 4.1.1 Implementation Details

We conduct experiments following the original settings of each work.
Mabel+$(\mathcal{U})\mathcal{NC}_{3}$: We implement Mabel with a batch size of 24, a learning rate of $5\times 10^{-5}$, and use the Adam optimizer, training it for two epochs.
ASE+$(\mathcal{U})\mathcal{NC}_{3}$: ASE is trained for 50 epochs with the Adam optimizer. The learning rate is set to $2\times 10^{-5}$, a dropout probability of 0.1 is used, and a batch size of 6 is chosen.
BEC+$(\mathcal{U})\mathcal{NC}_{3}$: BEC is trained for three epochs using the Adam optimizer, with a learning rate of $2\times 10^{-5}$ and a batch size of 16.
Due to the short fine-tuning duration, we directly accumulate $\bm{\mu}_{c}$ for regularization when computing $(\mathcal{U})\mathcal{NC}_{3}$.

The evaluation of the fairness of language models can be categorized into addressing intrinsic and extrinsic biases [], which we detail in the following subsections.
We also select Sent-Debias [], Context-Debias  [], and FairFil [] as our primary baselines, all of which introduce general-purpose methods for generating debiased representations.

**Table 4: Results on StereoSet. ${\star}$: results are reported in he2022mabel []; We follow previous works to evaluate on their datasets. $\diamond$: the closer to 50, the better. LM: language modeling score, SS: Steoreotype score, ICAT: combined score, defined as $\text{LM}\cdot(\min(\text{SS},100-\text{SS}))/50$.**
|  | StereoSet |  |  |
| --- | --- | --- | --- |
| Model | LM $\uparrow$ | SS $\diamond$ | ICAT $\uparrow$ |
| BERT^⋆ | 84.17 | 60.28 | 66.86 |
| BERT+Dropout^⋆ | 83.04 | 60.66 | 65.34 |
| BERT+CDA^⋆ | 83.08 | 59.61 | 67.11 |
| Sent-Debias^⋆ | 84.20 | 59.37 | 68.42 |
| Context-Debias^⋆ | 85.42 | 59.35 | 69.45 |
| FAIRFIL^⋆ | 44.85 | 50.93 | 44.01 |
| Adept | 86.37 | 58.70 | 71.34 |
| Mabel^⋆ | 84.80 | 56.92 | 73.07 |
| Mabel+$(\mathcal{U})\mathcal{NC}_{3}$ | 83.55 | 55.38 | 74.55 |
| ASE | 83.83 | 57.33 | 71.54 |
| ASE+$(\mathcal{U})\mathcal{NC}_{3}$ | 84.06 | 56.36 | 73.37 |
| BEC | 86.02 | 58.30 | 71.73 |
| BEC+$(\mathcal{U})\mathcal{NC}_{3}$ | 85.95 | 57.89 | 72.38 |

#### 4.1.2 Intrinsic Metrics

The goal of intrinsic debiasing is to reduce bias within model representations before they are applied to downstream tasks, making it task-agnostic.
Following previous works [; ], we consider two popular datasets for intrinsic metrics.

##### StereoSet [ 46 ]

evaluates the language model by testing for stereotypical associations. Following [],
we concentrate on examples within sentences that pertain to the gender domain. When encountering a partial context sentence, this task presents a fill-in-the-blank challenge where the model must choose from an unrelated word, an anti-stereotypical word, or a stereotypical word. The Language Modeling Score (LM) quantifies the percentage of times the model correctly identifies a relevant word, whether it is stereotypical or anti-stereotypical, or an irrelevant one. Meanwhile, the Stereotype Score (SS) is a fairness-sensitive metric, revealing how often the model shows a preference for the stereotype compared to the anti-stereotype. Lastly, the Idealized Context Association Test (ICAT) score integrates both LM and SS into a single metric, quantifying the balance between language modeling and fairness.

As shown in Table 4, our method consistently improves both SS and ICAT on all fairness methods. With our assistance, Mabel’s SS score improved by 1.54 and ICAT increased by 1.48, reaching a value of 74.55. Similarly, ASE achieved a significant ICAT improvement of 1.83.

**Table 5: Results on BEC-Pro. We show the mean association scores between gender words and professions. “Diff” represents the score difference between female and male (smaller the better).**
| Model | Female | Male | Diff $\downarrow$ |
| --- | --- | --- | --- |
| BERT | -0.0931 | -0.3388 | 0.2457 |
| Adept | -0.0056 | 0.0476 | 0.0532 |
| Mable | -0.0641 | -0.0237 | 0.0404 |
| Mable+$(\mathcal{U})\mathcal{NC}_{3}$ | 0.0039 | -0.0038 | 0.0077 |
| ASE | -0.7534 | -0.5125 | 0.2409 |
| ASE+$(\mathcal{U})\mathcal{NC}_{3}$ | -1.0093 | -0.9613 | 0.0480 |
| BEC | 0.0841 | 0.1349 | 0.0508 |
| BEC+$(\mathcal{U})\mathcal{NC}_{3}$ | 0.1145 | 0.1435 | 0.0290 |

##### BEC-Pro [ 1 ]

create a template-based corpus in two languages, English and German, to measure bias in BERT. The sentence templates include a gender-denoting noun phrase, or <person word>, along with a <profession>. Table 5
presents the average association scores between person-related terms (targets) and professions (attributes) before and after applying $(\mathcal{U})\mathcal{NC}_{3}$ to the debiased model on the GAP corpus [], with Counterfactual Data Substitution (CDS) [] applied, highlighting the difference between female and male associations.

We observe a significant reduction in the mean association score difference between female and male after applying our method. This indicates that the model perceives less distinction between these two concepts, implying a reduction in bias. Specifically, Mabel and ASE show a significant order-of-magnitude decrease in the score difference with our method, and BEC also shows over 40% reduction.

#### 4.1.3 Extrinsic Metrics

Extrinsic metrics [; ; ] focus on enhancing fairness in downstream tasks, such as sentiment analysis [] and machine translation [], by ensuring that models produce consistent outputs across different demographic groups.

##### WinoBias [ 71 ]

is an intra-sentence coreference resolution task designed to assess a system’s ability to correctly associate a gendered pronoun with an occupation in both pro-stereotypical and anti-stereotypical contexts. Coreference can be inferred using syntactic cues in Type 1 sentences, or more challenging semantic cues in Type 2 sentences. We fine-tune models on the OntoNotes 5.0 dataset [] and then evaluate on the WinoBias benchmark. We report the average F1-scores for pro-stereotypical and anti-stereotypical examples,
along with two fairness metrics: TPR-1 (Type 1: pro-stereotypical minus anti-stereotypical) and TPR-2 (Type 2: pro-stereotypical minus anti-stereotypical), measured by average F1-scores.

As shown in Table 6, after adding $(\mathcal{U})\mathcal{NC}_{3}$ as the regularization, all the models exhibit significant improvements on both fairness metrics. Especially, Mabel equipped with our $(\mathcal{U})\mathcal{NC}_{3}$ improves 6.8 on TPR-1 and more than a 40$\%$ improvement on TPR-2 over the Mabel baseline, achieving the best results for our tested models.

**Table 6: Average F1-scores on WinoBias, and TPR scores across Winobias categories. 1 = Type 1; 2 = Type 2. A = anti-stereotypical; P = pro-stereotypical. TPR-1 = 1P - 1A; TPR-2 = 2P - 2A.**
| Model | 1A $\uparrow$ | 1P $\uparrow$ | 2A $\uparrow$ | 2P $\uparrow$ | TPR-1 $\downarrow$ | TPR-2 $\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| BERT | 53.96 | 86.57 | 82.20 | 94.67 | 32.79 | 12.48 |
| Sent-Debias | 54.11 | 85.09 | 83.29 | 94.73 | 30.98 | 11.44 |
| Context-Debias | 59.40 | 85.54 | 83.63 | 93.20 | 26.14 | 9.57 |
| FairFil | 53.24 | 85.77 | 77.37 | 91.40 | 32.43 | 14.03 |
| Adept | 62.50 | 84.04 | 87.66 | 91.51 | 21.54 | 3.85 |
| Mabel | 61.21 | 84.93 | 92.78 | 96.20 | 23.73 | 3.41 |
| Mabel+$(\mathcal{U})\mathcal{NC}_{3}$ | 64.15 | 81.08 | 93.55 | 95.51 | 16.93 | 1.97 |
| ASE | 56.00 | 87.02 | 76.44 | 91.06 | 31.02 | 14.62 |
| ASE+$(\mathcal{U})\mathcal{NC}_{3}$ | 58.57 | 85.71 | 84.38 | 92.54 | 27.14 | 8.16 |
| BEC | 60.32 | 84.04 | 86.86 | 93.98 | 23.72 | 7.12 |
| BEC+$(\mathcal{U})\mathcal{NC}_{3}$ | 62.98 | 84.88 | 87.50 | 94.40 | 21.91 | 6.90 |

##### Bias-in-Bios [ 10 ]

is a third-person biography dataset labeled by both occupation and gender.
We fine-tune the encoder with the linear classification layer to predict an individual’s occupation from their biography. During evaluation, we present the overall accuracy for the task, as well as the accuracy segmented by gender.
Additionally, we use two widely adopted fairness metrics [; ]: 1) $GAP_{TPR}$, which captures the disparity in true positive rates (TPR) between male- and female-labeled instances; and 2) $GAP_{RMS}$, which represents the root-mean-square (RMS) of TPR gaps across different occupation categories ($C$), the closer their score is to 0, the better.
Their formula is as follows:

$$ $GAP_{TPR}=|TPR_{M}-TPR_{F}|,\quad\quad GAP_{RMS}=\sqrt{\frac{1}{|C|}\sum_{y\in C }(GAP_{TPR,y})^{2}}.$ (4) $$

In Table 7, we observe that introducing $(\mathcal{U})\mathcal{NC}_{3}$ during the fine-tuning process improves the model’s performance on these two TPR metrics. Additionally, it enhances the model’s accuracy in predicting female-related occupations, helping BEC achieve an accuracy of 85.17 for female predictions. Furthermore, we also conducted validation on the Bias-NLI [], and the results can be found in the Appendix D.

**Table 7: Fine-tuning results on Bias-in-Bios.**
|  | Acc. | Acc. | Acc. | GAP | GAP |
| --- | --- | --- | --- | --- | --- |
| Model | (All) $\uparrow$ | (M) $\uparrow$ | (F) $\uparrow$ | TPR $\downarrow$ | RMS $\downarrow$ |
| BERT | 84.14 | 84.69 | 83.50 | 1.189 | 0.144 |
| Sent-Debias | 83.56 | 84.10 | 82.92 | 1.180 | 0.144 |
| Context-Debias | 83.67 | 84.08 | 83.18 | 0.931 | 0.137 |
| FairFil | 83.18 | 83.52 | 82.78 | 0.746 | 0.142 |
| Adept | 84.07 | 83.64 | 84.58 | 0.945 | 0.121 |
| Mable | 84.85 | 84.92 | 84.34 | 0.599 | 0.132 |
| Mable+$(\mathcal{U})\mathcal{NC}_{3}$ | 84.30 | 84.09 | 84.54 | 0.455 | 0.126 |
| ASE | 84.63 | 84.19 | 85.14 | 0.949 | 0.127 |
| ASE+$(\mathcal{U})\mathcal{NC}_{3}$ | 84.55 | 84.21 | 84.95 | 0.740 | 0.126 |
| BEC | 84.43 | 83.99 | 84.95 | 0.954 | 0.124 |
| BEC+$(\mathcal{U})\mathcal{NC}_{3}$ | 84.66 | 84.23 | 85.17 | 0.938 | 0.128 |

### 4.2 Enforcing 𝒩 ⁢ 𝒞 𝒩 𝒞 \mathcal{NC} caligraphic_N caligraphic_C Preserves LM Performance on NLU Tasks

**Table 8: Fine-tuning results on the GLUE benchmark: $(\mathcal{U})\mathcal{NC}_{3}$ preserves LM performance while encouraging fairness. BERT-NLI: BERT fine-tuned on NLI data then fine-tuned on GLUE. Bold numbers indicate the best for different metrics.**
|  | CoLA $\uparrow$ | SST-2 $\uparrow$ | MRPC $\uparrow$ | QQP $\uparrow$ | MNLI $\uparrow$ | QNLI $\uparrow$ | RTE $\uparrow$ | STS-B $\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model | (mcc.) | (acc.) | (f1/acc.) | (acc./f1) | (acc.) | (acc.) | (acc.) | (pears./spear.) |
| BERT | 56.5 | 92.3 | 89.5/85.3 | 90.7/87.5 | 84.3 | 92.2 | 65.0 | 88.4/88.2 |
| BERT-NLI | 58.6 | 93.6 | 89.4/85.1 | 90.4/86.8 | 83.3 | 89.0 | 69.0 | 88.3/87.9 |
| Sent-Debias | 50.5 | 89.1 | 87.5/81.6 | 87.5/90.7 | 83.9 | 91.4 | 63.2 | 88.1/87.9 |
| Context-Debias | 55.2 | 92.0 | 85.1/77.5 | 90.7/87.4 | 84.6 | 89.9 | 57.0 | 88.4/88.1 |
| FairFil | 55.5 | 92.4 | 87.5/80.6 | 91.2/88.1 | 84.8 | 91.3 | 63.2 | 88.4/88.1 |
| Adept | 57.3 | 92.4 | 88.0/82.4 | - - / - - | 84.1 | 91.0 | 61.4 | 85.8/85.7 |
| Mabel | 57.8 | 92.2 | 89.5/85.0 | 91.2/88.1 | 84.5 | 91.6 | 64.3 | 89.6/89.2 |
| Mabel+$(\mathcal{U})\mathcal{NC}_{3}$ | 56.3 | 92.1 | 90.5/86.5 | 91.0/88.0 | 84.5 | 91.1 | 60.7 | 89.1/88.7 |
| ASE | 54.9 | 92.9 | 87.3/80.9 | - - / - - | 84.4 | 91.2 | 59.3 | 88.4/88.1 |
| ASE+$(\mathcal{U})\mathcal{NC}_{3}$ | 55.3 | 92.8 | 87.5/81.4 | - - / - - | 84.6 | 91.4 | 61.7 | 88.2/87.9 |
| BEC | 55.5 | 92.2 | 88.8/83.8 | - - / - - | 84.9 | 91.5 | 65.7 | 89.2/88.9 |
| BEC+$(\mathcal{U})\mathcal{NC}_{3}$ | 55.7 | 92.5 | 88.7/83.6 | - - / - - | 84.9 | 91.6 | 66.4 | 89.2/88.9 |

We also evaluate language models fine-tuned with our method on general natural language understanding (NLU) tasks.
As shown in Table 8, fine-tuning with Eq. 3 preserves the performance of language models on general tasks with minimal difference from each baseline model.
This indicates that our method can debias language models without catastrophically forgetting and sacrificing their pretrained knowledge.

## 5 Ablation Studies and Visualizations

##### Fairness Can Be Improved Solely via 𝒩 ⁢ 𝒞 𝒩 𝒞 \mathcal{NC} caligraphic_N caligraphic_C .

As we have seen that debiased language models can implicitly become more collapsed, here we further provide an ablation study to verify that the effectiveness of our regularization is not because of any external debiasing algorithm, namely, by only adopting our NC-based regularization (Eq. 3) we can already encourage fairness in language models.
We perform this ablation study in the standard language model fine-tuning for masked language modeling (MLM), where pronouns in the input sentence are replaced with [MASK] tokens. The model is then trained to predict the correct gendered pronoun for each [MASK] token in the masked sentence. The training objective is to minimize the cross-entropy loss between the original pronouns and the predicted logits corresponding to the [MASK] tokens. The MLM loss is denoted as $\mathcal{L}_{\text{MLM}}$:

$$ $\mathcal{L}_{\text{MLM}}=\frac{1}{|M|}\sum_{m\in\text{masked}}^{M}CE(\bm{W}^{ \top}\bm{h}_{m},x_{m})$ (5) $$

where $CE$ denotes the cross entropy loss, and $\bm{h}_{m}\in\mathbb{R}^{d}$ is the last hidden state of the masked token $x_{m}$. $\bm{W}\in\mathbb{R}^{d,C}$ is a linear layer for the MLM task.

We fine-tune the language model using $\mathcal{L}_{\text{MLM}}$ alone, and compare against the fine-tuning with the addition of $\mathcal{L}_{\mathcal{NC}_{3}}$.
We fine-tune models for three epochs.
This allows us to more clearly observe the gains brought by $(\mathcal{U})\mathcal{NC}_{3}$. We evaluate the model on StereoSet using intrinsic metrics, and as shown in Table 9, it is evident that not only does it help to reduce bias in the model, but also enhances the language capabilities. Notably, we achieve a significant improvement of 4.29 on the ICAT metric.

**Table 9: Fine-tuning BERT with only mask-token predictions (Eq. 5) and $(\mathcal{U})\mathcal{NC}_{3}$ (Eq. 3). Metrics are consistent with Table 4: “LM” for language modeling score, “SS” for Steoreotype score, “ICAT” for combined score, defined as $\text{LM}\cdot(\min(\text{SS},100-\text{SS}))/50$.**
|  | StereoSet |  |  |
| --- | --- | --- | --- |
| Model | LM $\uparrow$ | SS $\diamond$ | ICAT $\uparrow$ |
| BERT+$\mathcal{L}_{\text{MLM}}$ | 72.85 | 58.65 | 60.24 |
| BERT+$\mathcal{L}_{\text{MLM}}$+$\mathcal{L}_{\mathcal{NC}_{3}}$ | 75.43 | 57.23 | 64.53 |

##### Visualizations of Debiased Token Representations.

In addition to comparing quantitative results, we also conducted t-SNE visualizations of models’ logits to analyze the two training methods. Specifically, we visualize two opposing word pairs: (“Herself”, “Himself”) and (“Mother”, “Father”). As shown in Figure 2, with the original training method, words of the same gender tend to cluster together (e.g., “Himself” and “Father”). However, after introducing $(\mathcal{U})\mathcal{NC}_{3}$, we can find that: 1) token representations from different classes (words) are more clearly separated to each other; 2) tokens from each word are more clustered. This indicates that intra-class variances have been significantly reduced, thus gender-related words are more collapsed to their class means.

Figure: Figure 2: t-SNE plots of logits of two models in Table 9. We collect 15 samples each of “Herself,” “Himself,” “Mother,” and “Father.”
Refer to caption: x2.png

## 6 Conclusion

Aiming to provide a principled understanding and improvement of debiasing language models, we try to find connections between language models and neural collapse for fairness purposes.
We demonstrated that debiased models exhibit more collapsed token representations, especially for fairness-sensitive words, leading to better alignment with word embeddings. Leveraging this insight, we introduced a principled regularization method based on neural collapse that can consistently improve fairness in language models across various tasks without sacrificing performance. Our method is simple, effective, and applicable to a wide range of debiasing techniques, providing a robust tool for enhancing fairness in language models.
We expect our understanding and fine-tuning to become a principled method for debiasing language models.

## Appendix A gender words

Male: countryman, fraternal, wizards, manservant, fathers, divo, actor, bachelor, papa, dukes, barman, countrymen, brideprice, hosts, airmen, andropause, penis, prince, governors, abbot, men, widower, gentlemen, sorcerers, sir, bridegrooms, baron, househusbands, gods, nephew, widowers, lord, brother, grooms, priest, adultors, andrology, bellboys, his, marquis, princes, emperors, stallion, chairman, monastery, priests, boyhood, fellas, king, dudes, daddies, manservant, semen, spokesman, tailor, cowboys, dude, bachelors, barbershop, emperor, daddy, masculism, guys, enchanter, guy, fatherhood, androgen, cameramen, godfather, strongman, god, patriarch, uncle, chairmen, sir, brotherhood, host, testosterone, husband, dad, steward, males, cialis, spokesmen, pa, beau, stud, bachelor, wizard, sir, nephews, fathered, bull, beaus, councilmen, landlords, grandson, fiances, stepfathers, horsemen, grandfathers, adultor, schoolboy, rooster, grandsons, bachelor, cameraman, dads, him, master, lad, policeman, monk, actors, salesmen, boyfriend, councilman, fella, statesman, paternal, chap, landlord, brethren, lords, blokes, fraternity, bellboy, duke, balletdancer, dudes, fiance, colts, husbands, suitor, paternity, he, businessman, masseurs, hero, deer, busboys, boyfriends, kings, brothers, masters, stepfather, grooms, son, studs, cowboy, mentleman, sons, baritone, salesman, paramour, malehost, monks, menservants, mr., headmasters, lads, congressman, airman, househusband, priest, barmen, barons, abbot, handyman, beard, fraternities, stewards, colt, czar, stepsons, himself, boys, lions, gentleman, penis, his, masseur, bulls, uncles, bloke, beards, hubby, lion, sorcerer, macho, father, gays, male, waiters, sperm, prostate, stepson, prostaticutricle, businessmen, heir, waiter, headmaster, man, governor, god, bridegroom, grandpa, groom, dude, gay, gents, boy, grandfather, gelding, paternity, roosters, prostaticutricle, priests, manservants, stailor, busboy, heros.

Female: countrywoman, sororal, witches, maidservant, mothers, diva, actress, spinster, mama, duchesses, barwoman, countrywomen, dowry, hostesses, airwomen, menopause, clitoris, princess, governesses, abbess, women, widow, ladies, sorceresses, madam, brides, baroness, housewives, goddesses, niece, widows, lady, sister, nun, adultresses, obstetrics, bellgirls, marchioness, princesses, empresses, mare, chairwoman, convent, priestesses, girlhood, gals, mommies, maid, female ejaculation, spokeswoman, seamstress, cowgirls, chick, hairsalon, empress, mommy, feminism, enchantress, gal, motherhood, estrogen, camerawomen, godmother, strongwoman, goddess, matriarch, aunt, chairwomen, ma’am, sisterhood, hostess, estradiol, wife, mom, stewardess, females, viagra, spokeswomen, ma, belle, minx, maiden, witch, miss, nieces, mothered, cow, belles, councilwomen, landladies, granddaughter, fiancees, stepmothers, horsewomen, grandmothers, adultress, schoolgirl, hen, granddaughters, bachelorette, camerawoman, moms, mistress, lass, policewoman, saleswomen, girlfriend, councilwoman, stateswoman, maternal, wenches, sorority, ballerina, chicks, fiancee, fillies, suitress, maternity, she, businesswoman, masseuses, heroine, doe, busgirls, girlfriends, queens, sisters, mistresses, stepmother, daughter, minxes, cowgirl, mezzo, saleswoman, nuns, maids, mrs., headmistresses, lasses, congresswoman, airwoman, housewife, priestess, barwomen, baronesses, abbesses, handywoman, toque, sororities, stewardesses, filly, czarina, stepdaughters, herself, girls, lionesses, vagina, hers, masseuse, aunts, wench, toques, heiress, waitress, headmistress, bride, grandma, lesbian, girl, grandmother, hens, uterus, maidservants, seamstress’, busgirl, heroines.

## Appendix B Ablation Study on 𝒩 ⁢ 𝒞 3 𝒩 subscript 𝒞 3 \mathcal{NC}_{3} caligraphic_N caligraphic_C start_POSTSUBSCRIPT 3 end_POSTSUBSCRIPT Regularization

Our method is simple and principled. It adds our $\mathcal{NC}_{3}$ constraint to the original training loss used in each work, as shown in the following formula:

$$ $\mathcal{L}_{total}:=\mathcal{L}_{original}+\alpha\cdot\mathcal{L}_{\mathcal{ NC}_{3}},$ (6) $$

where $\mathcal{L}_{original}$ represents the original loss from the respective study, and $\mathcal{L}_{\mathcal{NC}_{3}}$ denotes our regularization term.

**Table 10: StereoSet results on different hyper-parameter settings of $\alpha$. Metrics are consistent with Table 4: “LM” for language modeling score, “SS” for Steoreotype score, “ICAT” for combined score, defined as $\text{LM}\cdot(\min(\text{SS},100-\text{SS}))/50$.**
|  | ASE+$(\mathcal{U})\mathcal{NC}_{3}$ |  | Mabel+$(\mathcal{U})\mathcal{NC}_{3}$ | BEC+$(\mathcal{U})\mathcal{NC}_{3}$ |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | LM $\uparrow$ | SS $\diamond$ | ICAT $\uparrow$ |  | LM $\uparrow$ | SS $\diamond$ | ICAT $\uparrow$ | LM $\uparrow$ | SS $\diamond$ | ICAT $\uparrow$ |
| $\alpha=1$ | 83.78 | 57.38 | 71.41 | $\alpha=10$ | 83.57 | 55.53 | 74.33 | 86.02 | 58.07 | 72.12 |
| $\alpha=3$ | 84.06 | 56.36 | 73.37 | $\alpha=30$ | 83.54 | 55.63 | 73.88 | 85.95 | 57.89 | 72.38 |
| $\alpha=5$ | 82.96 | 55.84 | 73.27 | $\alpha=50$ | 83.55 | 55.38 | 74.55 | 85.70 | 58.92 | 71.44 |

From Table 10, it’s clear that as $\alpha$ increases, the SS metric generally improves, indicating that the model becomes more biased. However, a more appropriate value of $\alpha$ can strike a better balance between language quality and bias, resulting in a higher ICAT score.

## Appendix C 𝒩 ⁢ 𝒞 𝒩 𝒞 \mathcal{NC} caligraphic_N caligraphic_C evaluations for more language models

In addition to the three debiasing methods tested in Section 3, we supplement our analysis here with experiments from two popular models: BERT base and RoBERTa base.
The results from these models (Table 11, 12, and 13) are consistent with the observations made in Section 3.
The three additional model setups are as follows: Adept on News-Commentary v15 []; Roberta and FairBERTa on PANDA []; Mabel under Roberta as the backbone.

**Table 11: The performance of more debiasing methods on $\mathcal{NC}$ metrics varies across datasets on the gender words ($\mathbb{V}_{\text{gender}}$).**
| Model | $\mathcal{N}\mathcal{C}_{1}\downarrow$ | $(\mathcal{G})\mathcal{NC}_{2}\downarrow$ | $(\mathcal{U})\mathcal{NC}_{3}\downarrow$ | $\mathcal{NC}_{4}\uparrow$ | $\mathcal{N}\mathcal{C}_{1}^{(\bm{w})}\downarrow$ | $(\mathcal{G})\mathcal{NC}_{2}^{(\bm{w})}\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| BERT | 2.090 | 0.088 | 1.317 | 0.169 | 1197.6 | 0.359 |
| Adept [] | 1.941 | 0.072 | 0.074 | 0.118 | 1143.5 | 0.364 |
| Roberta | 0.868 | 0.010 | 0.066 | 0.513 | 11.53 | 1.614 |
| FairBERTa [] | 1.287 | 0.661 | 0.038 | 0.000 | 5.883 | 1.698 |
| Roberta | 0.443 | 0.059 | 0.062 | 0.250 | 10.22 | 1.488 |
| Mabel-Roberta [] | 0.513 | 0.013 | 0.058 | 0.250 | 9.878 | 1.487 |

**Table 12: $\mathcal{NC}$ metrics of more debiased language models on the whole vocabulary ($\mathbb{V}$).**
| Model | $\mathcal{N}\mathcal{C}_{1}\downarrow$ | $(\mathcal{G})\mathcal{NC}_{2}\downarrow$ | $(\mathcal{U})\mathcal{NC}_{3}\downarrow$ | $\mathcal{NC}_{4}\uparrow$ | $\mathcal{N}\mathcal{C}_{1}^{(\bm{w})}\downarrow$ | $(\mathcal{G})\mathcal{NC}_{2}^{(\bm{w})}\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| BERT | 1.235 | 0.165 | 0.064 | 0.013 | 946.2 | 0.413 |
| Adept [] | 1.133 | 0.149 | 0.066 | 0.012 | 889.2 | 0.420 |
| Roberta | 0.405 | 0.116 | 0.065 | 0.647 | 8.334 | 1.617 |
| FairBERTa [] | 0.766 | 0.520 | 0.036 | 0.000 | 4.545 | 1.732 |
| Roberta | 0.246 | 0.135 | 0.060 | 0.369 | 6.219 | 1.567 |
| Mabel-Roberta [] | 0.242 | 0.061 | 0.061 | 0.568 | 5.363 | 1.567 |

**Table 13: $\mathcal{NC}$ metrics of more debiased language models on the same number of words as used in Table 11 ($\mathbb{V}_{\text{gender}}$) but words are randomly selected.**
| Model | $\mathcal{N}\mathcal{C}_{1}\downarrow$ | $(\mathcal{G})\mathcal{NC}_{2}\downarrow$ | $(\mathcal{U})\mathcal{NC}_{3}\downarrow$ | $\mathcal{NC}_{4}\uparrow$ | $\mathcal{N}\mathcal{C}_{1}^{(\bm{w})}\downarrow$ | $(\mathcal{G})\mathcal{NC}_{2}^{(\bm{w})}\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| BERT | 1.292 | 0.158 | 0.065 | 0.002 | 974.7 | 0.409 |
| Adept [] | 1.111 | 0.150 | 0.064 | 0.002 | 888.6 | 0.423 |
| Roberta | 0.407 | 0.116 | 0.067 | 0.811 | 8.416 | 1.618 |
| FairBERTa [] | 0.728 | 0.511 | 0.036 | 0.000 | 4.414 | 1.733 |
| Roberta | 0.217 | 0.146 | 0.058 | 0.162 | 5.653 | 1.561 |
| Mabel-Roberta [] | 0.237 | 0.063 | 0.064 | 0.280 | 5.267 | 1.572 |

## Appendix D Extra Extrinsic Metrics

##### Bias-NLI [ 11 ]

is an NLI dataset composed of neutral sentence pairs, systematically generated by populating sentence templates with a gendered word and an occupation that has a strong gender association (e.g., “The woman ate a bagel” “The nurse ate a bagel”). Bias is measured as a deviation from neutrality and is evaluated using three metrics: Net Neutral (NN), Fraction Neutral (FN), and Threshold:$\tau$ (T:$\tau$). Specifically, 1) NN represents the mean probability assigned to the neutral label across all entailment pairs; 2) FN calculates the proportion of sentence pairs classified as neutral; and 3) Threshold:$\tau$ (T:$\tau$) is a hyperparameter that determines the proportion of entailment pairs for which the probability of being neutral exceeds a given threshold. In this paper, $\tau$ is set to $0.5$ and $0.7$. And a bias-free model would achieve a score as 1 across all three metrics. NN and FN are defined in the following manner:

$$ $NN=\frac{1}{M}\sum_{i=1}^{M}n_{i},\quad\quad FN=\frac{1}{M}\sum_{i=1}^{M}n_{i} 1[n_{i}=max\{e_{i},n_{i},c_{i}\}],$ (7) $$

We fine-tune the model on SNLI and assess its performance on Bias-NLI during inference. After incorporating $(\mathcal{U})\mathcal{NC}_{3}$ during fine-tuning, we find that in such a rare-class classification scenario, the improvement is relatively limited. However, it generally helps the model maintain its performance, leading to some gains in accuracy across various metrics for ASE and BEC. Specific details can be found in Table 14.

**Table 14: Results on Bias-NLI. We fine-tune the models on SNLI and then evaluate on Bias-NLI.**
| Model | TN $\uparrow$ | FN $\uparrow$ | T:0.5 $\uparrow$ | T:0.7 $\uparrow$ |
| --- | --- | --- | --- | --- |
| BERT | 0.799 | 0.879 | 0.874 | 0.798 |
| Sent-Debias | 0.793 | 0.911 | 0.897 | 0.788 |
| Context-Debias | 0.858 | 0.906 | 0.902 | 0.857 |
| FairFil | 0.829 | 0.883 | 0.846 | 0.845 |
| Adept | 0.841 | 0.937 | 0.934 | 0.866 |
| Mable | 0.900 | 0.977 | 0.974 | 0.935 |
| Mable+$(\mathcal{U})\mathcal{NC}_{3}$ | 0.882 | 0.977 | 0.974 | 0.919 |
| ASE | 0.886 | 0.974 | 0.971 | 0.932 |
| ASE+$(\mathcal{U})\mathcal{NC}_{3}$ | 0.910 | 0.974 | 0.973 | 0.948 |
| BEC | 0.933 | 0.955 | 0.955 | 0.934 |
| BEC+$(\mathcal{U})\mathcal{NC}_{3}$ | 0.883 | 0.980 | 0.978 | 0.943 |

## Appendix E 𝒩 ⁢ 𝒞 𝒩 𝒞 \mathcal{NC} caligraphic_N caligraphic_C evaluations for Neutral words

In addition to gender-related words such as "female" and "male," there are also neutral words. To assess the impact of these neutral words, we conducted ablation experiments. The experimental setup is as follows:

- •
Gender words + Neutral words: including 425 gender-related words from Appendix A and 208 neutral words mentioned below.
- •
Gender words + Random words: including 425 gender-related words from Appendix A and 208 Random words.

Neutral: aerobics, adventurer, apparel, aggressive, assistant, tycoon, baker, warrior, bathing, ambitious, beautiful, trucker, beauty, welder, blonde, strong, bookkeeper, terrorist, ca, soldier, cashier, astronomer, chatty, sniper, cheerleader, skipper, cheerleading, banker, clerk, baseball, cocktail, sergeant, cooking, bodyguard, counselor, boss, crafting, boxer, cute, cabbie, dancer, captain, educator, cardiologist, emotional, carpenter, flirt, ceo, flirtatious, chairperson, flower, chancellor, gossip, chef, graceful, colonel, hairdresser, commander, hairdryer, conductor, homemaker, police, hooker, custodian, housekeeper, dentist, housekeepers, detective, housework, diplomat, hula, doctor, indoor, driving, jealousy, drummer, jewelry, economist, kawaii, electrician, laundering, engineer, librarian, engineering, librarians, entrepreneur, lotion, lovely, firefighter, marvelous, footballer, mirror, gambler, moisturizer, gamer, nanny, gangster, neat, geek, nurse, geeks, nursery, gentle, nurses, guitarist, nurturing, industrialist, parenting, inventor, passive, investigator, pink, laborer, pretty, lawyer, receptionist, leader, ribbon, lieutenant, romance, lifeguard, romantic, magistrate, secretary, manager, selfie, marshal, server, mathematician, sew, mechanic, sewing, muscle, shopping, muscular, smoothie, owner, soft, philosopher, softball, physicist, stylist, pilot, submissive, plumber, sweet, politician, tailor, president, tall, professor, teacher, programmer, thin, rugby, violinist, sailor, waiter, science, weak, scientist, yoga, sculptor, hysterical, blue, makeup, football, executive, management, professional, corporation, salary, office, business, career, home, parents, children, family, cousins, marriage, wedding, relatives, math, algebra, geometry, calculus, equations, computation, numbers, addition, poetry, art, dance, literature, novel, symphony, drama, sculpture, science, technology, physics, chemistry, Einstein, NASA, experiment, astronomy, Shakespeare.

As shown in  Table 15 and  Table 16, results of the two different word combinations on the $\mathcal{NC}$ metrics are very similar, which further supports the rationale for considering only gender-related words in our method. However, upon closer inspection, we can observe that when using neutral words, the metric values tend to be closer to those obtained with only gender words (Table 1 and Table 11), whereas the values with random words are more aligned with those generated by random words (Table 3 and Table 13). This further supports the validity of the $\mathcal{NC}$ metric.

**Table 15: $\mathcal{NC}$ metrics of different debiased language models on Gender words + Neutral words.**
| Model | $\mathcal{N}\mathcal{C}_{1}\downarrow$ | $(\mathcal{G})\mathcal{NC}_{2}\downarrow$ | $(\mathcal{U})\mathcal{NC}_{3}\downarrow$ | $\mathcal{NC}_{4}\uparrow$ | $\mathcal{N}\mathcal{C}_{1}^{(\bm{w})}\downarrow$ | $(\mathcal{G})\mathcal{NC}_{2}^{(\bm{w})}\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| BERT | 0.894 | 0.139 | 0.066 | 1.634 | 790.3 | 0.337 |
| Mabel [] | 0.698 | 0.141 | 0.069 | 1.421 | 687.6 | 0.330 |
| BERT | 2.182 | 0.073 | 0.064 | 0.684 | 1155.1 | 0.354 |
| ASE [] | 2.975 | 0.566 | 0.054 | 0.013 | 324.3 | 0.363 |
| BERT | 1.988 | 0.145 | 0.060 | 6.260 | 1033.0 | 0.351 |
| BEC [] | 2.105 | 0.183 | 0.055 | 4.469 | 1014.7 | 0.346 |
| BERT | 2.275 | 0.071 | 0.070 | 0.099 | 1267.4 | 0.351 |
| Adept [] | 2.104 | 0.053 | 0.070 | 0.070 | 1206.4 | 0.357 |
| Roberta | 0.737 | 0.018 | 0.065 | 0.693 | 10.48 | 1.621 |
| FairBERTa [] | 1.138 | 0.632 | 0.035 | 0.000 | 5.459 | 1.706 |
| Roberta | 0.317 | 0.075 | 0.065 | 0.155 | 6.986 | 1.553 |
| Mabel-Roberta [] | 0.333 | 0.069 | 0.065 | 0.134 | 5.573 | 1.553 |

**Table 16: $\mathcal{NC}$ metrics of different debiased language models on Gender words + Random words.**
| Model | $\mathcal{N}\mathcal{C}_{1}\downarrow$ | $(\mathcal{G})\mathcal{NC}_{2}\downarrow$ | $(\mathcal{U})\mathcal{NC}_{3}\downarrow$ | $\mathcal{NC}_{4}\uparrow$ | $\mathcal{N}\mathcal{C}_{1}^{(\bm{w})}\downarrow$ | $(\mathcal{G})\mathcal{NC}_{2}^{(\bm{w})}\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| BERT | 0.466 | 0.225 | 0.061 | 1.137 | 533.5 | 0.390 |
| Mabel [] | 0.413 | 0.219 | 0.065 | 1.132 | 475.6 | 0.384 |
| BERT | 1.534 | 0.142 | 0.064 | 0.644 | 943.3 | 0.391 |
| ASE [] | 2.076 | 0.445 | 0.055 | 0.012 | 288.17 | 0.397 |
| BERT | 1.742 | 0.052 | 0.083 | 5.721 | 891.8 | 0.399 |
| BEC [] | 1.903 | 0.093 | 0.051 | 4.115 | 892.6 | 0.389 |
| BERT | 1.555 | 0.140 | 0.068 | 0.093 | 1043.0 | 0.394 |
| Adept [] | 1.473 | 0.117 | 0.069 | 0.067 | 1010.9 | 0.397 |
| Roberta | 0.515 | 0.093 | 0.068 | 0.256 | 9.151 | 1.631 |
| FairBERTa [] | 0.909 | 0.552 | 0.037 | 0.000 | 5.032 | 1.731 |
| Roberta | 0.255 | 0.135 | 0.063 | 0.591 | 6.508 | 1.564 |
| Mabel-Roberta [] | 0.260 | 0.040 | 0.062 | 1.617 | 5.595 | 1.567 |

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. In section 3.2.2, the authors present a calibrated version of $\mathcal{NC}1$ and  $\mathcal{NC}2$. While I can understand that the main reason to replace class means with classifier weights is because the authors find that $\mathcal{NC}3$ (solely derived from classifier weights) is significant on the debiased models, could this modification compromise the original physical meaning of these metrics Specifically, 
    - $\mathcal{NC}1$, as proposed in [1], measures the variability via an inverse signal-to-noise ratio. However, in the calibrated $\mathcal{N}\mathcal{C}_{1}^{(w)}$, the numerator is the variance of the _class means_, while the denominator measures the distance of the _classifier weight_, which seems to lack physical coherence.
    - Why is $w_c$ not normalized in $\mathcal{(G)NC_2^{(w)}}$ while $\mu_c$ is normalized in $\mathcal{(G)NC_2}$ to measure uniformity?
    - I hope the authors can elaborate more on the reason they choose these calibrated metrics.
2. In section 3.3, the authors argue that _"debiased BERT models exhibit more different token representations and word embeddings only on gender-related vocabulary"_ based on __numerical gaps__ in $\mathcal{NC}$ metrics. However, the baseline level of the base BERT model differ across gender-related words, full vocabulary, and a randomly selected subset. By looking at the __ratios__ of the gap w.r.t. base BERT model's performance under $\mathcal{NC_1^{(w)}}$, I cannot tell there are significant differences among Table 1-3. Similarly, with $\mathcal{(U)NC_3}$, for the gap of ASE and BEC, it's still hard to tell the difference among these three tables.
3. In section 4.1.2, could the authors clarify why the difference in association scores between female and male groups is measured? If the association score is defined as $\log \frac{P_T}{P_\text{prior}}$, I feel it's more reasonable to measure the absolute value of the association score since if the absolute value is close to 0, it should suggest that <person word> predictions are less likely influenced by <profession>, which indicates a less biased LM.
4. All experiments in this paper are conducted on masked language models (e.g, BERT and RoBERTa). Could the proposed method also generalize to casual language models, as explored in the Linguistic Collapse[1] paper?

[1] Wu, Robert, and Vardan Papyan. "Linguistic Collapse: Neural Collapse in (Large) Language Models." arXiv preprint arXiv:2405.17767 (2024).
soundness: 2
strengths: 1. Analyzing the debiased language models from the perspective of neural collapse is novel. This offers valuable insights into the structure of the model's embedding space across different debiasing approaches.
2. The proposed method is simple but effective and can be easily integrated with other approaches. It's interesting to see that by only adding a NC-based regularization term in the loss function, all the compared debiasing methods can be enhanced.
3. The experiments are comprehensive and well-motivated, with the proposed method grounded in observations of $\mathcal{NC}$ perspectives on both biased and debiased LMs, effectively establishing a strong rationale for the proposed method, making it clear and convincing.
summary: This work explores fairness in language models from the perspective of Neural Collapse (NC), a phenomenon observed during the final stages of training neural networks on classification tasks (thus can also be applied to language modeling), where the variability and alignment of the penultimate layer and the final classifier reach a structured state. This work further applies NC to debiased LMs and demonstrates that  $\mathcal{NC}3$, which measures the alignment between token representations (contextual class means) and word embeddings (classifiers), consistently improved with debiasing. The author argues that other NC perspectives are less evident in debiased LMs due to common training configurations that violate NC’s typical conditions—such as vocabulary sizes exceeding embedding dimensions and imbalanced token distributions. To address this, they propose calibrated versions of  $\mathcal{NC}1$ and  $\mathcal{NC}2$ by replacing class means with classifier weights. Debiased LM shows significant collapse on these two metrics, and fairness-sensitive words demonstrate more collapse.

Based on the empirical analysis, the authors suggest using  $\mathcal{NC}3$ as a regularization term during fine-tuning to enhance fairness in LMs. Experiments on four fairness datasets demonstrate consistent gains on fairness metrics over three prior debiasing approaches and standard masked language modeling and also doesn't hurt the model's NLU ability.
weaknesses: 1. **Clarify of Notations**: Some notations are not clearly defined, which could impact readability.
    - The class embedding variances mentioned on line 161 and line 240 could benefit from a more explicit definition, similar to how it is presented in Eq. 3 of the Linguistic Collapse[1] paper. 
    - In Eq. 1, to align with the token representation defined at line 137, the notation should ideally be ${h}(E(x_{1:t}))$ rather than ${h}(x_{1:t})$ for clarity and consistency. 
2. **Evaluation Metrics**: I have concerns regarding the evaluation metrics used in section 3.3 and section 4.2.1. Please see question 2 and 3 in the following section.
3. **Definition of Unfairness**: Some definition of unfairness were not explicitly defined, which could make the paper less accessible to readers unfamiliar with the fairness literature. For example, in the *BEC-Pro* subsection of section 4.1.2, the average association score is used to measure model unfairness. While this is defined in [2] as $\log \frac{P_T}{P_\text{prior}}$, it's not explicitly explained here, which may cause confusion.

[1] Wu, Robert, and Vardan Papyan. "Linguistic Collapse: Neural Collapse in (Large) Language Models." arXiv preprint arXiv:2405.17767 (2024).    
[2] Bartl, Marion, Malvina Nissim, and Albert Gatt. "Unmasking contextual stereotypes: Measuring and mitigating BERT's gender bias." arXiv preprint arXiv:2010.14534 (2020).

---

confidence: 3
contribution: 2
initial_rating: 8
presentation: 3
questions: No questions, see weakness section above.
soundness: 3
strengths: ### Strengths

- The paper studies a missing link between neural collapse and fairness and proposes a bias mitigation strategy based on neural collapse objective that is agnostic to pre-training or fine-tuning methods. This also avoids manual data balancing or filtering.
- Both intrinsic and extrinsic fairness metrics see improvements with this approach.
summary: ### Summary

- The paper studies the relationship between neural collapse and fairness and tries to answer two research questions
	- "Do debiased LMs exhibit greater neural collapse"
	- "Can this inductive bias be leveraged to improve fairness"
- Background on neural collapse:
	- This is a phenomenon that happens in the last layer representations of classifiers.
	- Since the last layer of LM predicts the next word, it can be viewed as a classifier
	- The phenomenon of neural collapse causes the representations of certain gender sensitive words to collapse closer to their class means in debiased LMs
- The paper presents 4 ways to quantify neural collapse borrowing it from prior work. These metrics measure separability between classes.
	- NC1: intra-class variance over inter class variance
	- NC2: Measures separability from a geometric perspective
	- NC3: Uses dot products
	- NC4: Uses a linear projection

In section 3.2.1 the paper provides a worry about diverging values for various NC metrics but convincingly provides a resolution. This is achieved by calibrating the NC metrics by replacing class means with classifier weights in the expressions.
- The experimental results justify the claim of the presence of neural collapse for gender words.
- The second part of the paper deals with proposing a bias mitigation strategy by explicitly enforcing Neural collapse. To this end, they use the NC3 formulation as an auxiliary objective during fine-tuning.
- Both intrinsic and extrinsic metrics show improvements with this modified objective.

Please consider citing the following additional work in the field of LM fairness and evaluation of bias in LMs
- https://arxiv.org/pdf/2208.01448#page=14.61
- https://aclanthology.org/2022.findings-acl.55.pdf
- https://aclanthology.org/2022.acl-long.401.pdf
 
nits
- 3.2.2 CALIBRATIONS -> CALIBRATION
- Section 4.1.3
	- TPR-1 (Type 1: pro-stereotypical minus anti-stereotypical) and TPR-2 (Type 2: pro-stereotypical minus anti-stereotypical) -> TPR-1 (Type 1: pro-stereotypical minus anti-stereotypical) and TPR-2 (Type 2: pro-stereotypical minus anti-stereotypical) reads a bit odd. I think Type 1 are ambiguous sentences and Type 2 are non-ambiguous sentences. Explicitly call this out with a simple example.
weaknesses: ### Weakness

- Table 9 does not have the best results bold which makes it reading a little harder. 
- While the paper is well-written, simple to follow and proposes a simple technique which works, all the experiments conducted are on BERT based MLM based models. I understand this is intentional to ensure fair comparison with prior work but this also limits understanding the degradation on non MLM tasks. For eg. in Table 9, the results reported are on the GLUE benchmark which are all multiple choice questions. I am curious if this technique can really prove to be a generic technique even for generation tasks.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: Clarifications Needed: 
1. The σ symbol on line 161. 
2. On line 235, the authors hypothesize that "noises in language data and the complexity of different fine-tuning methods make measurements of NC1/2/4 unstable and inconsistent." Could the authors elaborate on why they consider these factors to be influential? 
3. Could the authors justify the rationale behind replacing class means with classifier weights in NC1/2?
4. Besides gender-related words, have the authors explored other words that might cause the model to collapse more? This experiment would complement Section 3.3.
5. I am curious about the effectiveness of the proposed method in fairness scenarios beyond gender. Firstly, what if it is challenging to obtain relevant word lists? Secondly, will the model’s general performance being affected by other word lists? I think the authors could counduct experiments by adding random words/high frequency words to the word list.
6. Can the authors provide further insights into the results of the visualization in Section 5? Why do they consider this to be a better feature distribution?
soundness: 2
strengths: The paper is logically structured and easy to follow, with a clear progression from problem statement to method proposal and experimental validation.
summary: This paper proposes a method to enhance model fairness based on the phenomenon of Neural Collapse (NC). The authors first analyze NC in the context of fairness-related words and find that debiased language models exhibit more aligned token representations with word embeddings. Inspired by this observation, they introduce a regularization method based on NC to reduce bias in gender-sensitive words during fine-tuning. The experiments demonstrate that this method can improve fairness across various debiasing algorithms while maintaining the language model's performance on standard natural language understanding tasks.
weaknesses: 1. In Section 4, the details of the datasets and metrics can be more concisely presented, with some details moved to the appendix. Conversely, the discussion of results is somewhat inadequate. It would be beneficial to highlight the advantages of the proposed method in comparison to the baselines.
2. Additional analysis should be included in section 4.2. From the results alone, it is observed that many outcomes with the addition of (U)NC3 have significantly decreased.
3. It is unclear whether the proposed method is also effective for LLMs.
4. It is uncertain whether the method remains effective for aspects of fairness beyond gender

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. Can this approach generalize to non-binary protected attributes? If so, how?

2. How does the method work for decoder-only models, like gpt and llama?
soundness: 4
strengths: 1.the paper draws the connection between neural collapse theory and fairness in language models in a principal way. It provides novel analytical tools for understanding debiasing mechanisms and offers new insights into how representation geometry relates to model fairness. 

2.  Comprehensive empirical experiments across multiple models and datasets, demonstrating the practical utility of the proposed approach.

3. The proposed regularization method seems to be simple yet effective. It can be easily integrated with existing debiasing approaches without significant computational overhead, making it highly practical for real-world applications. 

3. The empirical evaluation is thorough, covering both intrinsic and extrinsic fairness metrics. The results clearly demonstrate improvements across various benchmarks while preserving model performance on downstream tasks. 

5. The authors also provide helpful visualizations and analyses that illuminate how their method affects model representations.
summary: This paper investigates the relationship between the neural collapse phenomena and fairness in language models. The paper demonstrates that debiased language models naturally exhibit more "collapsed" representations for fairness-sensitive words, and leverage this insight to develop a neural collapse-based regularization method. The proposed approach enhances fairness across different debiasing methods while maintaining model performance on general language tasks.
weaknesses: 1. The primary weakness of the paper lies in its theoretical foundation. While the empirical results are promising, the authors don't provide sufficient theoretical justification for why NC3 and calibrated metrics work better than others. The paper would benefit from a formal analysis or proof connecting neural collapse to fairness improvements. 

2. The scope of the evaluation is somewhat limited, focusing primarily on **binary** gender bias. While this allows for deep analysis in one area, it leaves questions about the method's generalizability to other (possibly non-binary) demographic groups.

Meta-review: (a) **summary** This paper investigates the connection between Neural Collapse (NC) -- a learning phenomenon that happens in last-layer representations and classifiers in deep networks -- and fairness in language models (LMs). The authors hypothesize that similar effects may manifest in masked language models (MLMs), particularly for fairness-sensitive (e.g., gender-related) words, with two findings: (1) Debiased LMs show stronger Neural Collapse for sensitive words, (2) It is possible to leverage NC to improve fairness.

(b) **strengths**: The paper proposes a simple yet effective approach for improving fairness in LMs based on the observations of NC, and the evaluation is comprehensive

(c) **weakness**: (1) some reviewers request a formal analysis on why NC3 specifically underpins fairness improvements better than other NC metrics or calibrated versions (NC1/2/4); (2) lack of clarity in notations and definitions (3) narrow scope, which only focuses on gender bias and masked LM. It seems unclear whether it can be applied to other types of bias as well as decoder-only models

(d) **decision** Overall, the reviewers find the paper sound in its empirical demonstration and novel in its perspective connecting Neural Collapse to fairness. The method is straightforward to incorporate into existing debiasing setups and consistently yields fairness improvements. While the paper would benefit from more thorough theoretical grounding and clarifications in notations, given the paper’s novelty, promising empirical results, and relatively simple yet effective approach, I recommend acceptance.

Decision: Accept (Poster)