Venue: NAACL 2024

# Leveraging Prototypical Representations for Mitigating Social Bias without Demographic Information

Shadi Iskander  Kira Radinsky  Yonatan Belinkov

shadi.isk@campus.technion.ac.il kirar@cs.technion.ac.il belinkov@technion.ac.il Technion - Israel Institute of Technology

###### Abstract

Mitigating social biases typically requires identifying the social groups associated with each data sample. In this paper, we present DaFair, a novel approach to address social bias in language models. Unlike traditional methods that rely on explicit demographic labels, our approach does not require any such information. Instead, we leverage predefined prototypical demographic texts and incorporate a regularization term during the fine-tuning process to mitigate bias in the model's representations. Our empirical results across two tasks and two models demonstrate the effectiveness of our method compared to previous approaches that do not rely on labeled data. Moreover, with limited demographic-annotated data, our approach outperforms common debiasing approaches.1

Footnote 1: Our code is available at [https://github.com/technion-cs-nlp/DaFair](https://github.com/technion-cs-nlp/DaFair)

+
Footnote †: Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), pages 379–390

June 16-21, 2024 02024 Association for Computational Linguistics

## 1 Introduction and Background

The presence of social bias in training data presents a significant challenge in the development of language models for real-world applications. While these models possess remarkable capabilities, biases within the data can lead to unfair outcomes. Mitigating these biases is crucial, but it becomes particularly challenging when acquiring or accessing sensitive attribute labels is costly or unfeasible.

Studies showed that language models have the ability to capture demographic information about the writer, including race or gender, within their representations Caliskan et al. (2017); Zhao et al. (2018). However, this capability can introduce unintended biases, leading to discriminatory outputs De-Arteaga et al. (2019).

Common approaches for social bias mitigation require explicit annotation of biases for each sample in the data Beutel et al. (2017); Zhang et al. (2018). Recent concept removal methods Rayfogel et al. (2020, 2022a,b); Iskander et al. (2023) have shown promise in addressing social bias by removing sensitive attributes. These approaches rely on training classifiers for predicting the sensitive attribute, and training such classifiers typically requires a significant amount of annotated data.

A promising line of research has emerged that aims to mitigate bias without relying on explicit information about the biases present in the data. For instance, Just Train Twice (JTT) Liu et al. (2021) employs a two-step training process. In the second step, a second model is trained on up-weighed training examples that were misclassified by the first model. Another method is BLIND Orgad and Belinkov (2023), which introduces a success detector and down-weighs examples for which the detector accurately predicts the outcome.

In this paper, we propose **DaFair**: **D**emographics-**A**gnostic **F**airness, a novel approach for mitigating social bias during the fine-tuning process of language models, without

Figure 1: Our debiasing method consists of defining task-specific representations for each social attribute, measuring similarity in the representation space for each example, and utilizing the KL loss to encourage uniform probabilities across social groups.



relying on demographic information. Our approach aims to ensure equal similarity between the representation of a text and prototypical representations of different demographic groups. For instance, when classifying a biographical text of a person into their profession, our method aims to make the representation of the text equally similar to the representations of both males and females. More concretely, DaFair first defines prototypical texts, such as "This is a biography about a male" and "This is a biography about a female". It then adds a regularization term that makes the representation of a training example equally similar to the representations of each of the prototypical texts (Figure 1).

Furthermore, we extend our approach to scenarios where limited demographic-annotated data is available. In such cases, we obtain the prototypical representation by averaging the sample representations corresponding to each social attribute.

We evaluate the effectiveness of DaFair and its extension on two tasks: occupation prediction and sentiment analysis of twitter posts. In these tasks, we investigate the performance of our approach under the settings of limited demographic labels or no labels at all, reflecting real-world scenarios where labeled data is challenging to obtain. The experimental results with two base models demonstrate that our approach outperforms previous approaches that do not rely on demographic information, as well as common approaches with limited data.

## 2 Methodology

Assume a dataset \(D=\{t_{i},y_{i},z_{i}\}_{i=1}^{n}\) of input texts \(t_{i}\in\mathcal{T}\), main task labels \(y_{i}\in\mathcal{Y}\), and sensitive attributes \(z_{i}\in\mathcal{Z}\) that correspond to discrete demographic attributes, such as race. This sensitive attribute can either be unobserved during training or available in a small subset of the data. Our aim is to learn a model \(F:\mathcal{T}\rightarrow\mathbb{R}^{|\mathcal{Y}|}\) that does not rely on the sensitive attribute \(z_{i}\) in its prediction.

### Demographic-Agnostic Fairness Approach

Our method, depicted in Fig. 1, involves several key steps to mitigate social bias. First, we establish multiple representations for each group of sensitive attributes (Section 2.1.1). During fine-tuning, we measure similarity between the representation of an example and each attribute representation. These similarities are then transformed into a probability distribution. Subsequently, we use the Kullback-Leibler (KL) divergence loss [35] to compare the predicted probability distribution with a uniform distribution (Section 2.1.3). This loss term encourages the model to mitigate bias by penalizing deviations from a uniform distribution, promoting fair and unbiased predictions.

#### 2.1.1 Social Attribute Representations

We employ two approaches to define representations for social attribute groups, depending on the availability of labels: no labels, or few labels.

Pre-defined Representations (No Labels).In the absence of labeled data, we leverage semantic similarity and define pairs of texts that capture the models' understanding of text describing different social attribute groups. For example, to represent gender in an occupation prediction task we can use the encoder's representations of "This biography is about a man" and "This biography is about a woman". To generate these pre-defined representations, we employ a generative model. We provided ChatGPT [60] with a description of the approach, DaFair, along with a description of each dataset and task, and instructed the model to produce 10 pairs of prototypical texts for each task. The prototypical texts (Tables 7 and 8) and the full prompt (Figure 4) are provided in the appendix.

Data-driven Representations (Few Labels).When a limited number of labels are available, we leverage the representations generated by the text encoder to derive data-driven representations for each labeled group. Specifically, we calculate the mean representation of each labeled group using the available labeled samples. We call this method Semi-DaFair.

We will assume a binary case for simplicity and denote the pair of representations as \([X_{A},X_{B}]\).2

Footnote 2: Our approach can be extended to handle multiple social attribute groups, denoted as \([X_{A},X_{B},X_{C},...]\).

#### 2.1.2 Ensemble of Representations

Inspired by Stacey et al. (2020), we adopt an ensemble approach by leveraging multiple pairs of representations instead of using a single pair. We denote the ensemble of representations as \(\{[X_{A}^{j},X_{B}^{j}]\}_{j=1}^{K}\), where \(K\) represents the number of pairs.

In the case of pre-defined representations, we use multiple pre-defined pairs that capture different perspectives. For data-driven representations, we divide the labeled data into K partitions and calculate the mean representation for each partition, resulting in K pairs of representations.

 

By incorporating an ensemble of representations, we aim to capture a diverse range of information and perspectives related to biases.

#### 2.1.3 Calculating KL Loss

During fine-tuning, we calculate the similarity between the representation of example \(X_{i}\) and each pair of attribute representations using dot product:

\[[sim^{j}_{A},sim^{j}_{B}]=X_{i}\cdot[X^{j}_{A},X^{j}_{B}] \tag{1}\]

Then we apply the softmax function \(\sigma(a,b)=\frac{e^{n}}{c^{a}+e^{k}}\) to obtain the similarity distribution:

\[d^{j}_{sim}=\sigma(sim^{j}_{A},sim^{j}_{B}) \tag{2}\]

To calculate the overall KL loss, we compute KL divergence between each of the similarity distributions \(d^{j}_{sim}\) and a uniform distribution \(d_{uni}\):

\[L_{kl}=\sum_{j=1}^{K}D_{KL}(d^{j}_{sim},d_{uni}) \tag{3}\]

Finally, we compute the total loss:

\[L_{total}=L_{ce}+\lambda L_{kl}, \tag{4}\]

where \(L_{ce}\) is the usual cross-entropy loss. The hyper-parameter \(\lambda\) adjusts the balance between task performance and fairness, providing flexibility to prioritize either aspect.

## 3 Experimental Setup

### Tasks

We conduct experiments on two classification tasks: occupation prediction and sentiment analysis, focusing on social bias related to gender and race.

Occupation Prediction.We use the Bias in Bios Dataset DeArteaga et al. (2019). The task involves predicting the occupation of individuals based on their biographical information. The dataset consists of 394K biographies of 28 professions, with gender annotations.

Twitter Sentiment Analysis.We follow the setup of elazar2018bertweet, who leveraged a Twitter dataset originally gathered by blodgett2016bert. elazar2018bertweet used emojs in the tweets to derive sentiment labels for the classification task. Tweets are labeled with sociolects--African American English (AAE) or Standard American English (SAE)--based on the author's geo-location, serving as a proxy for their racial identity. We work with a subset of 100K samples, consistent with orgad2023good.

### Models

We use two pre-trained text encoders: BERT Devlin et al. (2019) and DeBERTa-V3 He et al. (2022). By considering two diverse tasks and different models, we can evaluate the effectiveness of our approach in mitigating social bias in various contexts and with different model architectures.

### Metrics

Performance Evaluation.We evaluate the model's accuracy (**Acc**) on the downstream task to ensure that it has not been significantly affected.

Fairness Assessment.To evaluate extrinsic bias, we align with previous work DeArteaga et al. (2019); Ravfogel et al. (2020) and use the True Positive Rate Gap (**TPR-GAP**) as the main fairness metric to assess performance disparities across different protected attribute groups. Following the guidelines in orgad2023good for a comprehensive evaluation, we also incorporate statistical fairness metrics: **Independence**, **Separation** and **Sufficiency**. The metrics details and calculation procedures are provided in Appendix B.

### Compared Methods

We compare our approach with several methods for bias mitigation and with a baseline (**Original**) without any debiasing procedure.

We compare with two existing methods that do not rely on demographic information:

**JTT**Liu et al. (2021), which trains in a second phase on up-wighed hard examples.

BLINDOrgad and belinkov (2023), which uses a success detector to down-weigh biased examples.

When only limited demographic labeled samples are available, we evaluate three methods:

**INLP**Ravfogel et al. (2020) removes linear information from the neural representation by iteratively training a linear classifier to predict the demographic attribute from the representation, then projecting the representations to the null-space of the linear classifier.

RLACERavfogel et al. (2022) is similar to INLP with the goal of linear information removal from the neural representations. However, it uses a different approach of a linear minimax game.

IGBPIskander et al. (2023) overcome the drawbacks of INLP and RLACE which only remove linearly encoded information, and removes non-linear information from representations by gradient-based projections 



### Settings

No Demographic Labels.In this setting, we explore scenarios where demographic labels are not available. We evaluate the performance of demographic-agnostic methods: JTT, BLIND and DAFAIR.

Limited Demographic Labels.Additionally, we investigate a scenario where we have limited access to demographic labels. In this setting, we apply information removal methods along with Semi-DaFairwhile varying the size of the available demographic-labeled data to analyze their effectiveness.

We run each method using 5 random seeds and report the mean and standard deviation of the test results. More details on training setup and evaluation procedures are described in Appendix A.

### DaFair Hyperparameters

Under the setting of no demographic labels, there is no validation set to optimize the selection of prototypical texts or the number of pairs. To avoid dependency on the choice of prototypical representations, we first generate \(N>K\) pairs, and within each iteration, we randomly sample \(K\) pairs. For all experiments, we set \(N=10\) , \(K=4\) to capture diverse associations of the training samples with demographic attributes, without relying on an extensive set of pairs. In Section 4.3, we analyze the impact of \(K\) on the model's performance and assess its implications on fairness and bias mitigation.

\(\lambda\) Tuning.To perform \(\lambda\) tuning without the need for a validation set with demographic annotations, we adopt Orgad and Belinkov (2023)'s strategy that prioritizes selecting the most radical parameter, while ensuring that the downstream task accuracy remains above 0.97 of the original accuracy. More details are described in Appendix C.

## 4 Results and Analysis

### No Demographic Labels

Table 1 presents the evaluation results on the occupation prediction and twitter sentiemnt tasks using BERT as encoder. In both tasks, Our proposed method, DaFair, achieves a slightly lower accuracy compared to the finetuned model. However, it significantly reduces the TPR-GAP, outperforming BLIND and JTT in mitigating bias related to gender or race. This could be attributed to the fact that JTT and BLIND do not directly address social bias, but assign different weights to examples based on their difficulty. In contrast, DaFair uses a regularization term designed to lower the association of text representation with specific social groups, which might explain the superior reduction of social bias measures. Evaluation with other statistical fairness metrics reveals similar patterns to TPR-GAP (Appendix C). Results with the DeBERTA-V3 model exhibit same trend, as presented in Appendix C.

### Limited Demographic Labels

Figure 2 presents TPR-GAP for both tasks under different levels of labeled data for social attributes, showcasing the performance of various debiasing methods (Section 3.4), including our proposed methods, DaFAir and Semi-DaFair.3 While using no labels (horizontal solid lines), DaFair outperforms other methods even when they are provided a limited number of labels (up to 100 in twitter sentiment and 1000 in occupation prediction). DaFAir further benefits from labels (Semi-DaFair lines), even outperforming prior methods with limited labeled data. With an abundance of labeled data (1000 in sentiment and 100K in occupation prediction), other methods perform better.

Footnote 3: Accuracy figures are available in Appendix C.1.

\begin{table}
\begin{tabular}{l c l l l} \hline \hline  & \multicolumn{2}{c}{Occupation Prediction} & \multicolumn{2}{c}{ Sentiment Analysis} \\ \cline{2-6} Method & Accuracy \(\uparrow\) & TPR-GAP \(\downarrow\) & Accuracy \(\uparrow\) & TPR-GAP \(\downarrow\) \\ \hline Original & \(83.43\pm 0.08\) & \(14.66\pm 0.51\) & \(79.18\pm 0.22\) & \(25.34\pm 0.82\) \\ \hline JTT & \(81.34\pm 0.81\) & \(14.19\pm 1.08\) & \(78.08\pm 1.21\) & \(23.67\pm 1.87\) \\ BLIND & \(82.52\pm 0.23\) & \(13.76\pm 1.18\) & \(76.45\pm 0.67\) & \(22.68\pm 2.40\) \\ DaFair & \(82.32\pm 0.13\) & \(12.29\pm 0.32\) & \(77.20\pm 1.17\) & \(21.72\pm 0.82\) \\ \hline \hline \end{tabular}
\end{table}
Table 1: Evaluation results for occupation prediction and sentiment analysis tasks with BERT as the text encoder.



The aim of information removal methods is to learn and neutralize decision boundaries between different social attributes using supervised learning. However, limited labeled examples may hinder classifiers from modeling social attribute subspace in high-dimensional spaces. In Semi-DaFair, we aim to mitigate associations with specific social groups. Surprisingly, a small set of labeled data seems sufficient for this purpose, as more labeled data does not offer additional benefits.

### Effect of Number of Prototypical Texts

To investigate the effect of the number of prototypical text pairs (K) on model performance, we conducted experiments with varying K values of (1, 2, 4, 8). The results presented in Table 2 reveal that all K values contributes to the reduction of the TPR-GAP without affecting accuracy. While larger values of K result in more substantial reductions, the incremental improvements become less significant for \(K>2\). These findings suggest that a small K may be sufficient for DaFair.

## 5 Conclusion

We introduced DaFair, a novel approach for mitigating social bias in language models without explicit demographic information. Our method leverages semantic similarity to manipulate the model's text representations during finetuning to promote fairness. Experimental results on two tasks and under different settings demonstrated the effectiveness of DaFair in reducing bias and improving fairness while maintaining competitive downstream task performance, even with limited or no labeled demographic data. With its focus on social bias, DaFair offers a flexible framework adaptable to address other forms of bias through the modification of prototypical texts.

In conclusion, our approach offers a practical and flexible solution for bias mitigation in real-world applications, contributing to the development of fairer language models.

\begin{table}
\begin{tabular}{l l c c c} \hline \hline  & \multicolumn{3}{c}{Occupation Prediction} & \multicolumn{2}{c}{Sentiment Analysis} \\ \cline{2-6} K & Accuracy \(\uparrow\) & TPR-GAP \(\downarrow\) & Accuracy \(\uparrow\) & TPR-GAP \(\downarrow\) \\ \hline Original & \(83.43\pm 0.08\) & \(14.66\pm 0.51\) & \(79.18\pm 0.22\) & \(25.34\pm 0.82\) \\
1 & \(82.85\pm 0.18\) & \(13.11\pm 0.62\) & \(77.52\pm 1.23\) & \(22.69\pm 1.81\) \\
2 & \(82.55\pm 0.23\) & \(12.84\pm 0.59\) & \(77.16\pm 1.12\) & \(22.21\pm 1.08\) \\
4 & \(82.32\pm 0.13\) & \(12.29\pm 0.32\) & \(77.20\pm 1.17\) & \(21.72\pm 0.82\) \\
8 & \(82.29\pm 0.30\) & \(12.20\pm 0.25\) & \(77.47\pm 0.94\) & \(22.17\pm 1.17\) \\ \hline \hline \end{tabular}
\end{table}
Table 2: Effect of varying K on accuracy and TPR-GAP for occupation prediction and sentiment analysis tasks.

Figure 2: Effect of bias mitigation methods on TPR-GAP with varying labeled data sizes. In scenarios with limited demographic-annotated data, our approach outperforms common debiasing approaches.

 



### Limitations

While our approach shows promise in mitigating social bias in language models without relying on demographic labels, it is important to recognize its limitations. First, our method relies on predefined texts that represent different social attribute groups, which may not fully capture the complexity and diversity of these attributes. Language models are complex systems, and they may still exhibit bias or unintended associations despite our efforts.

Moreover, it is important to acknowledge that gender is non-binary, and the expirements we conducted were focused on addressing binary gender biases. Additionally, our analysis of racial biases is centered around the African-American race, using sociolect as a proxy which might be inaccurate. We believe there is a need for more comprehensive research to address biases related to African American case and other racial and ethnic groups, in a more precise manner.

## Ethics Statement

The development and implementation of our method for mitigating bias in language models require careful ethical considerations. By employing the KL loss regularization term with non-uniform probabilities, there is a possibility of inadvertently amplifying biases or introducing unintended consequences. Additionally, while our aim is to mitigate bias without relying on demographic labels, we acknowledge the need for evaluation and validation to minimize any unforeseen biases that may persist. To mitigate these risks, we strongly recommend the collection of a small validation set to assess the performance of the system and ensure its alignment with ethical considerations.

## Acknowledgements

This research was supported by the Israel Science Foundation (grant 448/20), an Azrieli Foundation Early Career Faculty Fellowship, and an AI Alignment grant from Open Philanthropy.

## References

* Beutel et al. [2017] Alex Beutel, Jilin Chen, Zhe Zhao, and Ed H. Chi. 2017. Data decisions and theoretical implications when adversarially learning fair representations. _CoRR_, abs/1707.00075.
* Blodgett et al. [2016] Su Lin Blodgett, Lisa Green, and Brendan O'Connor. 2016. Demographic dialectal variation in social media: A case study of African-American English. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 1119-1130, Austin, Texas. Association for Computational Linguistics.
* Caliskan et al. [2017] Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. _Science_, 356(6334):183-186.
* De-Arteaga et al. [2019] Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Choluldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. 2019. Bias in bios: A case study of semantic representation bias in a high-stakes setting. In _Proceedings of the Conference on Fairness, Accountability, and Transparency_, FAT* 19, page 120-128, New York, NY, USA. Association for Computing Machinery.
* Devlin et al. [2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_, pages 4171-4186. Association for Computational Linguistics.
* Elazar and Goldberg [2018] Yanai Elazar and Yoav Goldberg. 2018. Adversarial removal of demographic attributes from text data. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 11-21, Brussels, Belgium. Association for Computational Linguistics.
* Ganin et al. [2016] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Francois Laviolette, Mario Marchand, and Victor Lempitsky. 2016. Domain-adversarial training of neural networks. _The journal of machine learning research_, 17(1):2096-2030.
* He et al. [2022] Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2022. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. In _The Eleventh International Conference on Learning Representations_.
* Iskander et al. [2023] Shadi Iskander, Kira Radinsky, and Yonatan Belinkov. 2023. Shielded representations: Protecting sensitive attributes through iterative gradient-based projection. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 5961-5977, Toronto, Canada. Association for Computational Linguistics.
* Kullback and Leibler [1951] Solomon Kullback and Richard A Leibler. 1951. On information and sufficiency. _The annals of mathematical statistics_, 22(1):79-86.
* Liu et al. [2016] Evan Z Liu, Behzad Haghoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy

Liang, and Chelsea Finn. 2021. Just train twice: Improving group robustness without training group information. In _International Conference on Machine Learning_, pages 6781-6792. PMLR.
* OpenAI (2022)OpenAI. 2022. _OpenAI. OpenAI: Introducing ChatGPT, 2022. URL [https://openai.com/blog/chatgpt_](https://openai.com/blog/chatgpt_).
* Orpad and Belinkov (2022) Hadas Orpad and Yonatan Belinkov. 2022. Choose your lenses: Flaws in gender bias evaluation. In _Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP)_, pages 151-167.
* Orpad and Belinkov (2023) Hadas Orpad and Yonatan Belinkov. 2023. Debiasing NLP models without demographic information. In _Proceedings of the 61th Annual Meeting of the Association for Computational Linguistics, ACL 2023, July 9-14, 2023_. Association for Computational Linguistics.
* Ravfogel et al. (2020) Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. 2020. Null it out: Guarding protected attributes by iterative nullspace projection. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_, pages 7237-7256. Association for Computational Linguistics.
* Ravfogel et al. (2022a)Shauli Ravfogel, Michael Twiton, Yoav Goldberg, and Ryan Cotterell. 2022a. Linear adversarial concept erasure. In _International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA_, volume 162 of _Proceedings of Machine Learning Research_, pages 18400-18421. PMLR.
* Ravggel et al. (2022b) Shauli Ravfogel, Francisco Vargas, Yoav Goldberg, and Ryan Cotterell. 2022b. Adversarial concept erasure in kernel space. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 6034-6055, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Stacey et al. (2020) Joe Stacey, Pasquale Minervini, Haim Dubossarsky, Sebastian Riedel, and Tim Rocktaschel. 2020. Avoiding the Hypothesis-Only Bias in Natural Language Inference via Ensemble Adversarial Training. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 8281-8291, Online. Association for Computational Linguistics.
* Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 38-45, Online. Association for Computational Linguistics.
* Zhang et al. (2018) Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. 2018. Mitigating unwanted biases with adversarial learning. In _Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society_, pages 335-340.
* Zhao et al. (2018) Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 2 (Short Papers)_, pages 15-20. Association for Computational Linguistics.

 

Setup

### Training Setup

We conduct experiments using two models: BERT (Devlin et al., 2019) and DeBERTa-V3 (He et al., 2022). For BERT we used the bert-base-uncased and for DeBERTa we used the microsoft/deberta-v3-base, both from the Huggingface library (Wolf et al., 2020). We utilize the transformer models as a text encoder, where the input text is transformed into a contextualized representation. The [CLS] token of the encoder is then passed through a linear classifier for the downstream task. We used a \(65/10/25\) training-validation-test split ratio for all tasks. Training was done with a learning rate of \(5e-5\) and a stochastic gradient descent optimizer for 1 epoch.

#### a.1.1 DaFair

To maintain suitable representations in the embedding space, we processed the pre-defined text (Sec 2.1.1) through the text encoder every 200 batches during fine-tuning. This fixed frequency was found to be effective for stable training and did not require further tuning.

\(\lambda\) TuningTo determine the appropriate value for the parameter \(\lambda\), we adopted the approach outlined by Ganin et al. (2016). The parameter \(\lambda\) was initially set to 0 and gradually adjusted towards a predefined threshold value, denoted as \(\lambda_{threshold}\), using a specific schedule. The schedule for updating \(\lambda\) is determined by the following formula:

\[\lambda=(\frac{2}{1+\exp(-\gamma\cdot p)}-1)\cdot\lambda_{threshold}, \tag{5}\]

where \(p\) represents a measure of progress, and \(\gamma\) controls the rate of change. The parameter \(\gamma\) allows us to control the speed at which \(\lambda\) approaches \(\lambda_{threshold}\).

In our experiments, we set \(\gamma\) to a fixed value of 5. This schedule allows for stable training in the early stages of training. In order to optimize the performance of DaFair and SemiDaFair, we conducted a grid search to find the optimal value for the parameter \(\lambda_{threshold}\). The search was performed over the following values \(\lambda_{threshold}\in(0.1,0.2,0.5,1,2,5,10,20,50,100)\).

#### a.1.2 Compared Methods

For **BLIND**, we used their implementation and ran a grid search for \(\gamma\in(1,2,4,8,16)\) and \(T\in(2,4)\).

For **JTT**, we provide our own implementation and tune for \(\lambda_{up}\in(1,2,4,6,8,10)\).

For the **INLP**, **RLACE**, and **IGBP** methods, we utilized the implementations provided by the respective authors. To ensure optimal performance, we conducted hyperparameter tuning specifically for the number of iterations. We applied these post-hoc methods on the model's representations extracted from a fine-tuned model (**Original**). In experiments with a limited number of labeled data for social attributes, we modify the concept removal methods by training the debiasing classifiers on the available labeled data.

### Evaluation Setup

As discussed in Section 3.6, to perform parameter tuning without the need for a validation set with demographic annotations, we adopt Orgad and Bellinkov (2023)'s strategy that prioritizes selecting the most radical hyperparameters, while ensuring that the downstream task accuracy remains above 0.97 of the original accuracy. This is done for all methods.

Intuitively, increasing the regularization term weight (in our case: \(\lambda_{threshold}\)) promotes fairness by encouraging the model to distribute its predictions more evenly among different social groups. However, it can also lead to a decrease in task accuracy if applied excessively. By setting a threshold of 0.97 for the accuracy, we strike a balance between bias mitigation and maintaining competitive performance.

## Appendix B Fairness Metrics

TPR-GAPWe calculate the True Positive Rate (TPR) by:

\[\mathrm{TPR}_{\text{z,y}}=\mathrm{P}(\hat{Y}=\mathrm{y}|Z=\mathrm{z},Y= \mathrm{y}) \tag{6}\]

where \(\hat{Y}\) represents the predicted label, \(Z\) denotes the protected attribute, and \(Y\) represents the true label.

The TPR Gap is computed as:

\[\mathrm{GAP}^{\mathrm{z,y}}_{\mathrm{TPR}}=\mathrm{TPR}_{\text{ z,y}}-\mathrm{TPR}_{\text{z^{\prime},y}} \tag{7}\]

where \(z\) and \(z^{\prime}\) correspond to different values of the protected attribute.

To assign a single bias measure across all values of \(y\), we calculate the root mean square \(\mathrm{GAP}^{\mathrm{z}}_{\mathrm{TPR}}\) 

\[{\rm GAP}^{z}_{\rm TPR}=\sqrt{\frac{1}{|C|}\sum_{y=1}^{N}\left({\rm GAP }^{z,y}_{\rm TPR}\right)^{2}} \tag{8}\]

where \(C\) represents the total number of label categories.

Statistical MeasuresAnother family of fairness metrics involves statistical measures based on probability distributions. We utilize three key metrics:

**Independence**: This metric measures the statistical dependence between the model's prediction and protected attributes. It employs the Kullback-Leibler divergence between two distributions, namely \(KL(P(y),P(\hat{y}|z=z))\), for \(z\in Z\). The sum over \(z\) yields a single value describing the model's independence. It assesses how the model's behavior varies across different demographics.
**Separation**: This metric assesses the statistical dependence between the model's prediction given the target label and the protected attributes. It utilizes \(KL(P(\hat{y}|y=y),P(\hat{y}|y=y,z=z))\) for all \(y\in Y\), \(z\in Z\). This metric is similar to True Positive Rate (TPR) and False Positive Rate (FPR) gaps. It evaluates if the model behaves differently across classes and demographics.
**Sufficiency**: This metric measures the statistical dependence between the target label given the model's prediction and the protected attributes. It employs \(KL(P(y|\hat{y}=\hat{y}),P(y|\hat{y}=\hat{y},z=z))\) for \(\hat{y}\in Y\), \(z\in Z\). The sum over \(\hat{y}\) and \(z\) results in a single value. It intuitively assesses whether a model disproportionately promotes or penalizes specific demographic groups.

To measure these statistical fairness metrics, we used the AllenNLP fairness library. ([https://github.com/allenai/allennlp](https://github.com/allenai/allennlp)).

## Appendix C Full Results

Tables 3 and 4 present comprehensive results for the occupation prediction and sentiment analysis tasks, respectively, employing BERT as the text encoder. Each method's performance is evaluated across multiple metrics, including Accuracy, TPRGAP, Independence, Separation, and Sufficiency (Section B). Here we also see that our proposed method reduces Independence, Separation, and Sufficiency values, in both tasks.

DeBERTA-V3 ResultsThe evaluation results for the occupation prediction and sentiment analysis tasks using the DeBERTa-V3 model are presented in Tables 5, and 6. In the two tasks, Both JTT and BLIND methods demonstrate some success in reducing bias, although not substantial. However, DaFAir , outperforms both JTT and BLIND in terms of mitigating bias related to social attributes. It achieves a lower TPR-GAP, Independence, Separation and Sufficiency in most cases, while maintaining a comparable level of accuracy. This indicates that our approach is more effective in reducing bias without sacrificing the overall performance of the model also with DeBERTa-V3 model.

### Accuracy Figures

In Figure 3 we provide the accuracy of different methods across varying dataset sizes.

Figure 3: Effect of bias mitigation methods on accuracy with varying labeled data sizes.

 



\begin{table}
\begin{tabular}{l c c c c c} \hline \hline  & \multicolumn{5}{c}{Sentiment Analysis} \\ \cline{2-6} Method & Accuracy \(\uparrow\) & TPR-GAP \(\downarrow\) & Indep \(\downarrow\) & Sep \(\downarrow\) & Suff \(\downarrow\) \\ \hline Original & \(78.36\pm 0.93\) & \(29.62\pm 1.58\) & \(0.19\pm 0.003\) & \(0.12\pm 0.004\) & \(0.09\pm 0.005\) \\ \hline JTT & \(77.41\pm 1.76\) & \(28.06\pm 2.03\) & \(0.18\pm 0.013\) & \(0.13\pm 0.009\) & \(0.08\pm 0.005\) \\ BLIND & \(77.20\pm 1.22\) & \(27.32\pm 2.84\) & \(0.15\pm 0.001\) & \(0.07\pm 0.001\) & \(0.08\pm 0.001\) \\ DaFair & \(77.92\pm 1.21\) & \(26.24\pm 1.61\) & \(0.16\pm 0.002\) & \(0.08\pm 0.003\) & \(0.07\pm 0.003\) \\ \hline \hline \end{tabular}
\end{table}
Table 6: Full results for the sentiment analysis task with DeBERTa as the text encoder.

\begin{table}
\begin{tabular}{l l l l l l} \hline \hline  & \multicolumn{5}{c}{Sentiment Analysis} \\ \cline{2-6} Method & Accuracy \(\uparrow\) & TPR-GAP \(\downarrow\) & Indep \(\downarrow\) & Sep \(\downarrow\) & Suff \(\downarrow\) \\ \hline Original & \(79.18\pm 0.22\) & \(25.34\pm 0.82\) & \(0.17\pm 0.003\) & \(0.11\pm 0.003\) & \(0.08\pm 0.004\) \\ \hline JTT & \(78.08\pm 1.21\) & \(23.67\pm 1.87\) & \(0.18\pm 0.01\) & \(0.11\pm 0.008\) & \(0.08\pm 0.006\) \\ BLIND & \(76.45\pm 0.67\) & \(22.68\pm 2.40\) & \(0.14\pm 0.001\) & \(0.05\pm 0.001\) & \(0.06\pm 0.001\) \\ DaFAir & \(77.20\pm 1.17\) & \(21.72\pm 0.82\) & \(0.15\pm 0.002\) & \(0.09\pm 0.002\) & \(0.08\pm 0.004\) \\ \hline \hline \end{tabular}
\end{table}
Table 4: Full results for the sentiment analysis task with BERT as the text encoder.

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline  & \multicolumn{5}{c}{Occupation Prediction} \\ \cline{2-6} Method & Accuracy \(\uparrow\) & TPR-GAP \(\downarrow\) & Indep \(\downarrow\) & Sep \(\downarrow\) & Suff \(\downarrow\) \\ \hline Original & \(83.43\pm 0.08\) & \(14.66\pm 0.51\) & \(0.16\pm 0.002\) & \(2.59\pm 0.07\) & \(2.46\pm 0.08\) \\ \hline JTT & \(81.34\pm 0.81\) & \(14.19\pm 1.08\) & \(0.16\pm 0.002\) & \(2.77\pm 0.01\) & \(2.22\pm 0.14\) \\ BLIND & \(82.52\pm 0.23\) & \(13.76\pm 1.18\) & \(0.16\pm 0.002\) & \(2.35\pm 0.10\) & \(2.21\pm 0.09\) \\ DaFAir & \(82.32\pm 0.13\) & \(12.29\pm 0.32\) & \(0.14\pm 0.001\) & \(1.90\pm 0.09\) & \(2.20\pm 0.10\) \\ \hline \hline \end{tabular}
\end{table}
Table 3: Full results for the occupation prediction task with BERT as the text encoder.

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline  & \multicolumn{5}{c}{Sentiment Analysis} \\ \cline{2-6} Method & Accuracy \(\uparrow\) & TPR-GAP \(\downarrow\) & Indep \(\downarrow\) & Sep \(\downarrow\) & Suff \(\downarrow\) \\ \hline Original & \(78.36\pm 0.93\) & \(29.62\pm 1.58\) & \(0.19\pm 0.003\) & \(0.12\pm 0.004\) & \(0.09\pm 0.005\) \\ \hline JTT & \(77.41\pm 1.76\) & \(28.06\pm 2.03\) & \(0.18\pm 0.013\) & \(0.13\pm 0.009\) & \(0.08\pm 0.005\) \\ BLIND & \(77.20\pm 1.22\) & \(27.32\pm 2.84\) & \(0.15\pm 0.001\) & \(0.07\pm 0.001\) & \(0.08\pm 0.001\) \\ DaFAir & \(77.92\pm 1.21\) & \(26.24\pm 1.61\) & \(0.16\pm 0.002\) & \(0.08\pm 0.003\) & \(0.07\pm 0.003\) \\ \hline \hline \end{tabular}
\end{table}
Table 6: Full results for the sentiment analysis task with DeBERTa as the text encoder.





\begin{table}
\begin{tabular}{|p{142.3pt}|p{170.7pt}|} \hline
**Male Prototypical Texts** & **Female Prototypical Texts** \\ \hline This is a biography about a male. & This is a biography about a female. \\ \hline A man who excelled in his field. & A woman who excelled in her field. \\ \hline He is known for his achievements in various industries. & She is known for her achievements in various industries. \\ \hline A prominent male figure in history. & A prominent female figure in history. \\ \hline His career and accomplishments are well-regarded. & Her career and accomplishments are well-regarded. \\ \hline This biography focuses on the life of a distinguished man. & This biography focuses on the life of a distinguished woman. \\ \hline An influential male individual. & An influential female individual. \\ \hline He made significant contributions to his profession. & She made significant contributions to her profession. \\ \hline This is a story about a man who shaped his industry. & This is a story about a woman who shaped her industry. \\ \hline His impact on his field is noteworthy. & Her impact on her field is noteworthy. \\ \hline \end{tabular}
\end{table}
Table 7: Pre-defined Representations for Male and Female Biographical Texts

\begin{table}
\begin{tabular}{|p{142.3pt}|p{195.1pt}|} \hline
**AAE Prototypical Texts** & **SAE Prototypical Texts** \\ \hline This tweet reflects a [sentiment] from a white writer. & This tweet reflects a [sentiment] sentiment from a black writer. \\ \hline A tweet expressing a [sentiment] moment by a white individual. & A tweet expressing a [sentiment] moment by a black individual. \\ \hline A [sentiment] viewpoint shared by a writer using Standard American English. & A [sentiment] viewpoint shared by a writer using African American English. \\ \hline This post, written in standard English, conveys [sentiment] from a white perspective. & This post, written in AAE, conveys [sentiment] from a black perspective. \\ \hline A message filled with [sentiment] from a white communicator. & A message filled with [sentiment] from a black communicator. \\ \hline A white person shares their [sentiment] thoughts in this tweet. & A black person shares their [sentiment] thoughts in this tweet. \\ \hline This is an example of a tweet with [sentiment] in white sociolect. & This is an example of a tweet with [sentiment] sentiment in AAE. \\ \hline A tweet written by a white speaker that conveys [sentiment]. & A tweet written by a black speaker that conveys [sentiment]. \\ \hline This post by a white individual radiates [sentiment] and [sentiment]. & This post by a black individual radiates [sentiment] and [sentiment]. \\ \hline A [sentiment] perspective presented by a writer using white sociolect. & A [sentiment] perspective presented by a writer using African American English. \\ \hline  & \multicolumn{1}{c}{} \\ \end{tabular}
\end{table}
Table 8: Pre-defined Representations for AAE and SAE Tweet Texts

* [185] M. C. Bender, A. D. Krizhevsky, and R. G. Hinton. The classification of deep neural networks with applications to image recognition. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pages 239-246, 2007.
* [186] J. L. Friedman, S. F. Friedman, and P. W. Friedman. Deep learning for large scale data sets. _Journal of Machine Learning Research_**26**(1):1-10, 1994.

[MISSING_PAGE_POST]