Sociodemographic Bias in Language Models: A Survey and Forward Path (2024)

Abstract: Sociodemographic bias in language models (LMs) has the potential for harm when deployed in real-world settings. This paper presents a comprehensive survey of the past decade of research on sociodemographic bias in LMs, organized into a typology that facilitates examining the different aims: types of bias, quantifying bias, and debiasing techniques. We track the evolution of the latter two questions, then identify current trends and their limitations, as well as emerging techniques. To guide future research towards more effective and reliable solutions, and to help authors situate their work within this broad landscape, we conclude with a checklist of open questions.

heading: 1 Introduction
text: Language models (LMs) have demonstrated impressive performance across diverse tasks (Raffel et al., 2020; Zhong et al., 2020; Yang et al., 2019). However, much work reveals that LMs adopt biases present in training data (Wen et al., 2022; España-Bonet and Barrón-Cedeño, 2022; Gupta et al., 2022b; Hutchinson and Mitchell, 2019). Sociodemographic bias has been defined to occur when a model performs differently across social groups (Czarnowska et al., 2021; Chouldechova and Roth, 2020). When LMs are used in real-world applications, sociodemographic bias has the potential for negative societal impacts (Field et al., 2023; Rudin, 2019; Blodgett et al., 2020). With increasingly widespread usage, the urgency to understand and mitigate bias has grown. Fig. 1 shows an increasing rate of publications on LM bias over the past decade, sourced from SCOPUS. Our survey synthesizes results from this rapidly growing area into a roadmap for future investigations.
Other surveys on bias in NLP have thoroughly examined a particular aspect of bias, such as methods for measuring bias (Czarnowska et al., 2021; Bansal, 2022), or identification of gender bias
(Stanczak and Augenstein, 2021; Devinney et al., 2022). Unlike previous surveys, we provide a typology of works on bias over the past decade. Further, we build upon foundational issues identified by Blodgett et al. (2020) by delving more deeply into methodological limitations, such as reliability issues. We also follow the recommendations of Blodgett et al. (2020) in consulting interdisciplinary approaches to improve the understanding of social bias. Thus we begin the survey with a discussion of psychosocial perspectives on benefits versus harms of bias. Our survey offers an up-todate understanding of a topic that has been garnering increasing interest. Early in this literature, the development of bias mitigation or debiasing methods had questionable success; we argue that recent work using expert models during training shows particular promise. We conclude with a checklist of key questions that have continued to be challenging, to help steer future studies toward more effective and reliable methods, well-situated within the landscape of work on bias.
We present a synthesis of works on bias through three perspectives: 1) a taxonomic categorization, 2) an evolutionary timeline, and 3) a roadmap for future work. We categorized the surveyed works into three major strands of investigation, as shown in Fig. 2: types of bias, quantifying bias, and debiasing techniques. We then organize the findings
295
within each category and subcategory of our taxonomy. In addition, we examined the evolution over the past decade of techniques for bias measurement and bias mitigation, as shown in Fig. 3. This perspective separates trends that had a brief life from those that continue to have promise.
While LM bias measurement and mitigation are critical for progress towards equitable LMs, understanding the potential for harm is deeply intertwined with social factors outside the scope of NLP proper. Thus we precede the presentation of the major types of bias research with a discussion of psychosocial perspectives (cf. Omrani et al., 2023; Mei et al., 2023). This is followed by a section describing our process for identifying candidate works, and our resulting typology where we place most of the surveyed works. Sections 4-6 present limitations, the checklist and future directions.

heading: 2 Understanding Bias
text: Interdisciplinary approaches to understanding bias as a psychosocial phenomenon have been argued to be important for clarifying how social harms might arise. Research into human cognition and social behavior can provide valuable insights on sociodemographic bias in LMs, as well as assessment of their potential for harm. For instance, research in psychology has long addressed the origins and expressions of social bias (Osborne et al., 2023) Recent studies have begun to integrate ideas from psychology with NLP to better understand bias (Spliethöver et al., 2022; Omrani et al., 2023; Mei et al., 2023; Omrani Sabbaghi et al., 2023), showcasing the usefulness of interdisciplinary approaches. For example, research in psychology proposes that reduction of social bias can be achieved by engaging with individuals from diverse groups (Pettigrew and Tropp, 2006; Reimer and Sengupta, 2023). A similar idea is reflected in Blodgett et al. (2020), which advocates for LM engineers to reduce bias through engagement with people who might be affected by applications that use LMs. One of the early works on quantifying bias - WEAT (Caliskan et al., 2017) used the Implicit Association Test from psychology (Greenwald et al., 1998) to develop a foundation bias metric for LMs.
The Stereotype Content Model (SCM), a framework from social psychology, categorizes stereotypes into interpersonal and intergroup interactions, providing insights into bias dynamics (Cuddy et al., 2008). It proposes that human stereotypes are cap-
tured by two social dimensions: warmth (e.g., trustworthiness, friendliness) and competence (e.g., capability, assertiveness). A recent work by Omrani et al. (2023) used the SCM framework to develop a bias mitigation method that generalizes across multiple social attributes, rather than one at a time.
The Nobel Prize-winning psychologist and behavioral economist, Daniel Kahneman, argues that mental shortcuts (biases) are advantageous in situations requiring quick judgments (Kahneman, 2011). For example, due to bias based on strong knowledge priors, the sentence “a large mouse climbed over a small elephant” will immediately call to mind the appropriate relative sizes; to counter this assumption would require extra information (Grice, 1975). Extrapolating Kahneman’s argument to NLP, bias based on common-sense knowledge could be advantageous in enhancing an LM’s understanding of relations among real-world entities. This argues for the potential benefit of certain kinds of bias.
Kahneman (2011) defines disadvantageous bias as “the tendency to make systematic errors in judgment or decisions based on factors that are irrelevant or immaterial to the task at hand” and cautions that human judgment is susceptible to bias from irrelevant factors. Turning to LM bias, we find previous NLP work aligned with Kahneman’s perspective in definitions of representational harm (Crawford, 2017) and alloted harm (Barocas et al., 2017). Representational harm arises when an NLP system represents some social groups in a less favorable light than others. Allotted harm arises when a system allocates resources or opportunities unfairly to a social group (Shahbazi et al., 2023).
In conclusion, ideas from psychology and behavioral economics provide a more informed understanding of bias. While some biases might contribute positively to model performance, others can have detrimental societal effects. An interdisciplinary approach would not only enrich our theoretical understanding of bias but could also guide the development of more effective methods to identify undesirable LM bias and lessen social harm.

heading: 3 Categories of Work on Bias in LMs
text: We used two strategies to identify candidate papers for our survey: 1) using the keywords “bias”, “fair” and “fairness”, we searched for papers in the ACL Anthology, NeurIPS proceedings, FAccT, and AIES conferences; 2) we included papers from citation graphs for retrieved papers. We examined pa-
pers released before January 1, 2024, and included them only if they addressed language modeling, thus omitting papers on speech, where different issues arise. These criteria narrowed down an initial large set of 308 papers to 273.
We categorized the literature into three key areas. Fig. 2 illustrates our taxonomy. We came up with this organization while iteratively reviewing all papers, and we believe it effectively captures the main trends in the field. Due to the rapidly evolving field of LMs, some upcoming studies may not fit neatly into these categories. To address this, we plan to release our literature repository publicly and update it regularly with the latest research. Our work summarizes all 273 surveyed papers to provide a comprehensive understanding. Due to space constraints, we couldn’t cite all 273 works in the main body. To maximize coverage within the page limits, we selected at least two papers from each line of research depicted in Figure 2 to be part of the main paper. In some cases, we wanted to cite more works but had to remove them due to space limitations. We apologize for any relevant works missed in the main body and have included a comprehensive list of all 273 papers in the Appendix.

heading: 3.1 Types of Bias - T1
text: In the realm of NLP, sociodemographic bias is particularly concerning as it can lead to differential model performance across various social groups (Deas et al., 2023; Smith et al., 2022). Sociodemographic bias includes gender bias, when models are biased against a particular gender (Hada et al., 2023; De-Arteaga et al., 2019; Park et al., 2018; Du et al., 2021; Bartl et al., 2020; Webster et al., 2021); racial bias, when models are biased against certain races (Nadeem et al., 2021; Garimella et al., 2021;
Nangia et al., 2020; Tan and Celis, 2019); ethnic bias, when models are partial towards certain ethnicity (Ahn and Oh, 2021; Garg et al., 2018; Li et al., 2020; Abid et al., 2021; Manzini et al., 2019; Narayanan Venkit et al., 2023); age bias (Nangia et al., 2020; Diaz et al., 2018), sexual-orientation bias (Nangia et al., 2020; Cao and Daumé III, 2020) and many others as shown in Table 1.
Sociodemographic bias can emerge from language patterns that imply assumptions about demographic differences (Lauscher et al., 2020). These biases are often ingrained in the cultural or societal nuances of training data. For example, LMs can perpetuate biases by associating certain lexical items more strongly with particular social groups. Beyond the influence of training data, Zhou et al. (2023b) found that the size of the model, its training objectives, and tokenization strategies are important factors that affect the social bias in LMs.
Our review indicates a disproportionate focus on gender bias: it is the subject of nearly half of the surveyed papers, as Table 1 illustrates. Additionally, we observed that bias evaluation and mitigation efforts are often specific to certain biases and may not generalize well. Furthermore, over 90% of the papers we reviewed focus on English, with lim-
ited coverage of other languages such as German, Spanish, Korean, Turkish, Chinese, and Hindi.

heading: 3.2 Quantifying Bias
text: Measurement of bias is challenging because it is often hidden within complex LMs. However, quantifying bias is a precondition to addressing or mitigating bias that might be harmful. Here we review different methods of measuring bias in LMs and how they differ from each other. We present an overview of evaluation datasets in the appendix.
Methods in Q1 and Q2 are often known as intrinsic methods as they focus on a model’s internal representations to quantify bias.

heading: 3.2.1 Distance-based metrics - Q1
text: Distance in vector space. Early efforts to quantify bias in NLP (from 2013-2019, as seen in Fig. 3) primarily utilized distance metrics within embedding spaces. These approaches define certain words as ’target words’ (like professions ’engineer’ and ’nurse’), along with certain words as ’attributes’ (often related to social categories like ’male’ and ’female’) (Bolukbasi et al., 2016; Brunet et al., 2019a; Dev et al., 2021). The aim was to measure the conceptual distance between these targets and attributes. The pioneering work is the Word Embedding Association Test (WEAT) score (Caliskan et al., 2017). They calculate bias as the differential association of target words with attribute sets based on cosine similarity. Subsequent to WEAT, Dev and Phillips (2019) proposed ECT score, which simplifies an attribute category, like ‘female’, into a single vector by averaging the embeddings of related attribute words such as ‘she’, ‘women’, and ‘girl’. Ethayarajh et al. (2019) introduced RIPA, for which they used the inner product instead of cosine similarity to account for vector magnitude and directionality in measuring bias.
Some works expanded WEAT to contextual embeddings (Guo and Caliskan, 2021; Tan and Celis, 2019) and sentence-level embeddings (May et al., 2019). Other metrics used the clustering of word embeddings (Chaloner and Maldonado,
2019). Some work quantified bias based on cooccurrence of words (Valentini et al., 2023; Bordia and Bowman, 2019). Bordia and Bowman (2019) hypothesized that words occurring in close proximity to a particular gender in the train data are prone to be more biased towards that gender during testing.

heading: 3.2.2 Probing metrics - Q2
text: This category evaluates bias by examining how LMs process information, often by adding a classification layer or employing probes to test the inner workings of LMs (Chen et al., 2021; White et al., 2021). Mendelson and Belinkov (2021) used a classifier trained on latent spaces to detect biases and found that debiasing against a particular bias may increase the extent to which that bias is encoded in the inner representations of models. Orgad et al. (2022) trained a classifier to predict gender from the model’s representations and shows it correlates with extrinsic bias measures better than metrics in Q1. Immer et al. (2022) proposed a Bayesian framework for quantifying inductive bias with probes.
In recent years, there has been less use of intrinsic methods, as they require accessing a model’s internal layers to quantify bias. The increasing size of modern LMs complicates identifying the right layer for bias assessment, and the limited opensource availability of LMs raises further obstacles.
Methods in the next two subsections, Q3 and Q4 are often known as extrinsic methods as they focus on bias that shows up in a downstream task.

heading: 3.2.3 Performance-based metrics - Q3
text: These approaches examine how models perform across different social groups. They typically divide the test dataset into different groups to assess disparities. These works aim to quantify group differences in performance - to document whether models perform the same for all groups. De-Arteaga et al. (2019) measured gender bias by comparing the true positive rates for classification involving male versus female names and pronouns. Dixon et al. (2018) and Zhao et al. (2018a) took similar ap-
proaches, using area under the curve and false positive rate (Dixon et al., 2018), and relative accuracy (Zhao et al., 2018a). Zhang et al. (2022) and Huang et al. (2020) generated augmented datasets to measure bias as the difference in accuracy between the original and augmented datasets. Stanovsky et al. (2019) proposed a metric based on differences in accuracy across genders for machine translation.

heading: 3.2.4 Prompt-based metrics - Q4
text: Here we review methods that use various promptgeneration techniques. The first two methods in this subsection are specific to autoregressive models, while the latter focuses on Masked LMs.
Template-based methods. In these approaches, models are prompted through a set of pre-defined templates, or patterns, that capture specific types of bias or stereotypes. The templates contain slots that are filled through selection from a set of predefined demographic target terms during evaluation. For instance, a template could be "A <PERSON> is walking" where <PERSON> is systematically substituted with names associated with different demographic groups. By analyzing the differences in the model’s responses to these substitutions, the presence and degree of bias can be measured.
Prabhakaran et al. (2019) generated templates for toxicity detection, and proposed metrics based on performance differences for target groups. Smith et al. (2022) introduces a holistic dataset, measuring bias across a dozen social demographic axes. Webster et al. (2021) defined fourteen templates to determine gender identity bias. Felkner et al. (2023) created a dataset of 45,540 sentences using 11 templates for measuring anti-LGBTQ+ bias in LMs. Gupta et al. (2023) focused on creating a robust dataset and generated 224 templates from diverse domains across 3 tasks. An et al. (2023); Parrish et al. (2022a); Li et al. (2020) proposed questionanswering datasets to measure demographic bias. In contrast to performance-based metrics (Q3), these approaches are primarily concerned with representational harms, which occur when certain groups are depicted stereotypically or inaccurately.
Counterfactual-based methods. Several works aim to make template-based approaches more rigorous by examining how changing irrelevant attributes, known as protected attributes, affects model predictions. Specifically, “a decision is fair towards an individual if it is the same in (a) the
actual world and (b) a counterfactual world where the individual belongs to a different social group.”
Counterfactual methods alter these protected attributes in test examples to identify attributes that significantly affect model decisions (Garg et al., 2019; Kusner et al., 2017). Huang et al. (2020) created counterfactuals for a test dataset and found that generative LLMs like GPT-2 (Radford et al., 2019) tend to generate continuations with more positive sentiment for “baker”, and more negative sentiment for “accountant” as the occupation. Gardner et al. (2020) created contrast sets by generating counterfactuals for ten NLP datasets and showed that model performance drops significantly on counterfactuals. Liang et al. (2022) substituted terms linked to specific demographic groups in the test set, examining the impact on model accuracy.
Masking Sentences. Another approach to bias measurement is to mask certain words in sentences, and then analyze the model’s predictions for these blanks to assess bias. Kurita et al. (2019) used this technique with occupation-related sentences, like “[MASK] is a programmer,” comparing the probabilities given to male and female pronouns to identify gender biases in job associations. Similarly, Ahn and Oh (2021) quantified bias as the variance of normalized probabilities across various demographic groups. Other works using this approach include (Ousidhoum et al., 2021; Bartl et al., 2020).
Extrinsic approaches, particularly templatebased ones, have gained traction in recent years (Nagireddy et al., 2023; Touileb et al., 2023; Akyürek et al., 2022), as seen in Fig. 3. The advantage of Q4 metrics is their ability to reflect potential real-world impacts of bias by focusing on model outputs rather than solely analyzing internal parameters as in Q1. Extrinsic methods apply broadly to open-source or proprietary models of any size.

heading: 3.3 Debiasing
text: Debiasing methods aim to make models more fair and accurate in their predictions and recommendations (Subramanian et al., 2021). Turning to Daniel Kahneman again, he argues that reducing social stereotyping and bias has costs, but that the costs are worthwhile to achieve a better society (Kahneman, 2011). Extending the same principle to NLP, the effort and cost required for reducing biases are essential for creating fair NLP systems.

heading: 3.3.1 Debiasing during Finetuning - D1
text: These debiasing methods are applied during the finetuning phase of pre-trained LMs.
Data augmentation. Zmigrod et al. (2019) and Lu et al. (2020) introduced Counterfactual Data Augmentation (CDA), to reduce gender bias by generating counterfactual instances to balance gender representation. This involves substituting genderspecific words, such as he and she to construct novel sentences. Maudslay et al. (2019) enhanced this approach with Counterfactual Data Substitution (CDS), which assigns probabilities to these changes, aiming for more realistic modifications. Building upon these insights, various swapping mechanisms have been proposed to re-balance data distributions (Zhou et al., 2023a; Panda et al., 2022; Liang et al., 2020; Lauscher et al., 2021; Wen et al., 2022). Some of these augmentation approaches are also being adapted for use during model training.
Modifying vector space. Limisiewicz and Mareček (2022); Dev et al. (2020, 2021) proposed a subspace correction method for modifying embedding space. They aimed to disentangle associations between concepts that are bias-prone. Yifei et al. (2023); Manzini et al. (2019) used principal component analysis to identify and address the bias in embedding spaces. Gaci et al. (2022) redistributed attention scores to assign an equal weight for words related to bias. Ravfogel et al. (2020) learned a linear projection over representations after training, to remove the bias components in embeddings.
Fine-tuning with large corpora. Park et al. (2018) demonstrated that debiasing models benefit from fine-tuning with extensive datasets, avoiding the pitfalls of small, biased datasets. Ahn and Oh (2021) proposed that training BERT (Devlin et al., 2019) on multiple languages helps to reduce ethnic biases in each language.
Human-in-the-loop. These methods involve humans to detect and mitigate biases. Yao et al. (2021) used human-provided explanations to identify and reduce bias. Felkner et al. (2023) showed bias against marginalized communities can be mitigated using data written by that community. Chopra et al. (2020) used humans to find words linking a social group to a positive or negative trait.
Model Unlearning Recently, there has been more focus on model unlearning methods (Fig. 3).
Here, the main idea is to identify and alter specific model weights that are responsible for bias. Meissner et al. (2022) identified a subset of model weights responsible for bias and masked them during testing. The advantage of their approach is it does not require finetuning. Lauscher et al. (2021); Kumar et al. (2023) captured bias mitigation functionalities using “adapters” attached to transformer blocks. Adapters offer a unique advantage in that they can be added to the model for bias correction in a plug-and-play fashion. Agarwal et al. (2023) improved on adapters by adjusting weights with data augmentation, then finetuning for specific tasks with fixed weights to prevent relearning.
Works in D1 offer greater ease of implementation, with customizable solutions for each model. However, as the prevalence of large language models grows, they are being trained on enormous amounts of data. In such cases, bias becomes more difficult to mitigate after models have been trained.

heading: 3.3.2 Debiasing during Training - D2
text: These works apply debiasing at the pre-training time or to word embeddings used at initialization.
Debiased word embeddings Bolukbasi et al. (2016) proposed a hard debiasing technique aimed at reducing gender bias in embeddings by adjusting the vector deviations between gendered and genderneutral terms, offering these adjusted embeddings as an alternative to standard Word2Vec embeddings. Park et al. (2018); Zhao et al. (2018b) further illustrate the effectiveness of debiased embeddings in reducing gender bias in LMs.
Loss function Several methods employ specialized loss functions to minimize bias during model pre-training. Garimella et al. (2021) used declustering loss to reduce bias. Bordia and Bowman (2019) proposed a loss regularization method. Huang et al. (2020) proposed a three-step curriculum training using the distance between the embeddings as a fairness loss to reduce sentiment bias. Liu et al. (2021) and He et al. (2022a) used adversarial training and contrastive loss respectively to reduce bias in LMs. Li et al. (2023) shows that using contrastive learning during training helps in debiasing.
Expert Models for Bias Reduction Recently methods using an auxiliary model, or so-called expert model, to reduce bias have gained prominence (cf. Fig. 3). Orgad and Belinkov (2023) predicted biased samples using an auxiliary model and per-
formed sample reweighting to downweight these samples during pre-training. Jeon et al. (2023) used binary classifiers, referred to as bias experts, to identify biased examples within a specific class. Zhang et al. (2023) used gradient-based explanations to focus on sensitive attributes and downstream tasks, adjusting the training process to balance fairness and performance effectively.

heading: 3.3.3 Debiasing at Inference Time- D3
text: These methods apply debiasing methods at test time. In general, these methods are quite diverse. Venkit et al. (2023b) and Abid et al. (2021) applied adversarial machine learning to trigger positive associations in text generative models to reduce anti-Muslim bias and nationality bias, respectively, through prompt modifications. Majumder et al. (2023) used humans to provide feedback to balance between task performance and bias mitigation. Qian et al. (2021) performed keywordbased distillation to remove bias during inference, and to block bias acquired during training. Zhao et al. (2019) addressed gender bias through averaging representations for different gender vocabulary. Schick et al. (2021a) also presents the concept of self-debiasing, in which a model can identify and eliminate biases after generating text.
Work on debiasing during inference time faces the same issues as those in D1. They are easy to implement but give a false impression of debiasing and do not completely remove the model bias.

heading: 4 Limitations of Current Approaches
text: The works surveyed here offer valuable insights towards understanding bias in LMs, and demonstrating many innovative approaches and methodologies that have advanced the field. Alongside the commendable progress, however, a thorough analysis of the body of work on bias reveals limitations.
Reliability issues with bias metrics. The robustness of existing bias metrics is questionable. Metrics introduced in works within Q1 and Q3 change significantly, given minor changes in datasets or evaluation settings (Antoniak and Mimno, 2021; Spliethöver et al., 2022; Du et al., 2021; Valentini et al., 2022). Similarly, template-based methods are highly sensitive to small modifications to words used in the templates (Selvam et al., 2023; Seshadri et al., 2022; Alnegheimish et al., 2022).
Use of identical templates across bias categories. Many of the works using template-based approaches (An et al., 2023; Smith et al., 2022) use
the same templates to assess diverse social biases, without considering whether certain template features should be specific to distinct types of bias. This approach risks conflating bias scores across categories, suggesting a need for more tailored templates to measure specific social biases accurately. Alternatively, investigation of ways to generalize across templates to a more abstract approach, as in Omrani Sabbaghi et al. (2023), holds promise.
Limited scope of template-based bias measurement. Template-based methods often use a restricted range of templates and target words, for example, focusing on US-based names for targets. Additionally, these approaches suffer from author bias, as templates are manually designed by the authors (Seshadri et al., 2022; Pikuliak et al., 2023).
Gap in translating bias metrics to real-world effects. There is a notable disconnect between bias metrics and their implications for real-world applications. Bias metrics in Q1 have been claimed to correlate poorly with real-world biases (GoldfarbTarrant et al., 2021; Cao et al., 2022). Orgad et al. (2022) argued that intrinsic and extrinsic metrics do not correlate with each other. Such observations underscore the need for improvements in metric robustness and interpretability.
Weaknesses in finetuning approaches for debiasing. The majority of recent works on LM debiasing focus on finetuning, valued mainly for its simplicity and adaptability. However, its effectiveness is often questionable (DiCiccio et al., 2023). The complexity and size of modern LLMs, which require extensive data, time, and resources to train, make it particularly challenging to eliminate bias through finetuning. Further, these methods treat symptoms rather than root causes of bias, adjusting model outputs to appear less biased without actually removing bias from models (Gonen and Goldberg, 2019; Tokpo et al., 2023). Remarkably, some debiasing techniques can potentially increase bias (Mendelson and Belinkov, 2021). The absence of reliable bias metrics complicates the evaluation of the effectiveness of debiasing methods. We recommend that future works utilize a variety of metrics to thoroughly assess debiasing results.
Overemphasis on gender bias. Table 1 shows that about half of the literature focuses solely on gender bias. Although gender bias is a significant concern, other types of sociodemographic bias also deserve attention. Expanding research to cover a wider range of bias categories could provide a more
comprehensive understanding of bias. Lack of sociotechnical understanding of bias.
The NLP literature exhibits little attention to the sociotechnical impacts of bias (Venkit et al., 2023a). Similarly, there can be incomplete consideration of the complexity of sociodemographic bias (Blodgett et al., 2020). Interdisciplinary collaborations could offer more nuanced insights and improved methodologies to measure, mitigate, prevent, and assess harms from bias.
Difficulty of comparing approaches. A better understanding is needed of strengths and weaknesses across approaches, given that works often focus on different domains and tasks. Kaneko et al. (2023) compared different bias evaluation approaches without requiring the expense of human labels. We need more work in the direction of reliable and cost-effective comparison among different measurement and mitigation methods.

heading: 5 Checklist
text: A checklist can assist future work to avoid the pitfalls of previous work and build more effective and reliable measurement and debiasing methods across more types of sociodemographic bias. We present 13 questions divided into three categories. Questions 1-6 focus on bias measurement (QB); questions 7-8 focus on bias mitigation (BM); questions 9-13 are applicable to all works on LM bias. We hope that future work guided by these questions can help authors situate their results within the broader literature on sociodemographic bias.
[Q1:QB] Robustness: Is your bias measurement stable against small modifications to templates/descriptors?
[Q2:QB] Country-focused data: Does your method rely on country-specific data, such as the U.S.? If so, how can it be adapted to others?
[Q3:QB] Real-World Relevance: How do your bias measurements reflect real-world biases, and affect end-users?
[Q4:QB] Future Usability: Have you taken measures to make sure your approach is easily extendable to ensure that it is useable after 5 years?
[Q5:QB] Data Diversity: Have you used diverse data sources to diminish biases present in the data?
[Q6:QB] Verification of Bias Type: What measures have you taken to ensure your bias measurement on a given type of bias doesn’t overlap or confuse with other biases?
[Q7: BM] Scalability and Efficiency: Can your
debiasing method efficiently scale to large models and datasets while maintaining effectiveness?
[Q8: BM] Monitoring and Evaluation: Is there a way for you to continuously assess and adjust the effectiveness of your approach?
[Q9: GQ] Extensibility to other Social Groups: Can your method be extended to additional sociodemographic groups?
[Q10: GQ] Risk of Misinterpretation: Can there be a situation when your approach falsely indicates reduced bias in models?
[Q11: GQ] Cultural Sensitivity: Does your approach take into account the contextual and cultural variations in language use?
[Q12: GQ] Interdisciplinary Insights: Does your method integrate knowledge from multiple disciplines to understand bias?
[Q13: GQ] Transparency and Reproducibility: Can others reproduce your method and results?

heading: 6 Future Directions
text: Looking ahead, we anticipate greater emphasis on bias mitigation at training time. Post-training bias mitigation adds to the costliness of very large LMs, and serves as a filter rather than a corrective. Subsequent to the first drafts of this survey, we have already seen progress in this direction (Jeon et al., 2023). The recent progress in the usage of contrastive learning during training (Li et al., 2023) and using expert models during training (Orgad and Belinkov, 2023), has shown to generate less biased models and we expect more research in these directions.
Despite their growing popularity, template-based methods for measuring bias face challenges (Selvam et al., 2023; Seshadri et al., 2022). We believe that these challenges can be tackled with careful consideration of the limitations, such as lack of robustness, leading to more effective and reliable bias measurement. We anticipate that prompt-based methods will gain prominence. Additionally, integrating interdisciplinary insights with algorithmic analysis will likely gain traction for quantifying and mitigating sociodemographic bias.
Finally, as robust methodologies emerge, we anticipate increased hope for and emphasis on intersectional bias, the overlap of multiple types of bias.

heading: 7 Conclusion
text: We have presented a comprehensive literature survey based on the iterative consideration of 273
works on sociodemographic bias in LMs. Our proposed typology provides an overview of the current research landscape. We identified promising directions for future research and introduced a 13- question checklist designed to guide future research towards more effective and reliable approaches and to avoid pitfalls of previous works. We encourage increased reliance on interdisciplinary methods to better capture and address the nuances of sociodemographic bias in LMs.

heading: 8 Limitations
text: In our survey, we focused on works from ACL Anthology, NeurIPS proceedings, FAccT, and AIES. We might have missed some relevant works in our survey, that appeared in other venues. While we have systematically organized the bias literature into categories as shown in Fig. 2, which came from an extensive survey of current literature, our framework might not encompass all existing or future research. We would like to emphasize that most of the works covered in this survey focus on the English language and some approaches discussed might not transfer to other languages. Additionally, our emphasis on sociodemographic bias means that valuable insights from works addressing other forms of bias in language models were not covered in our analysis.

heading: 9 Bias Statement
text: In this work, we provide a comprehensive survey of works on sociodemographic bias in language models. We defined sociodemographic bias as the difference in model performance across social groups. Such bias has the potential for harm in a realworld setting. Our definition applies to prominent demographic distinctions such as gender identity (male, female, non-binary), or income-based groupings (e.g., low, middle, and high income), or other broad-coverage distinctions that are learnable by LMs. For example, associating “Caucasian man” with “handsome”, and "African-American man" with “angry” is a clear indication of bias in models (Garimella et al., 2021). In occupation-related tasks, associating “receptionist” with “she”, and “philosopher” with “he” can have harmful effects in real-world settings (Bolukbasi et al., 2016).

heading: 10 Ethics Statement
text: Our work addresses the ethical impact of sociodemographic bias in NLP, offering a comprehensive
survey of 273 peer-reviewed articles to highlight the presence and implications of bias within language models. By systematically organizing research findings and tracking bias approaches over the past decade, our work promotes transparency, awareness, and accountability within and beyond the NLP community. The survey provides a meticulously designed checklist, based on the weaknesses and limitations of the field, to guide future research toward more effective solutions for mitigating bias.
We also emphasize the social and ethical implications of bias underscoring the significance of addressing these issues to prevent potential negative consequences. We hope that our analysis aids in shaping more inclusive and equitable NLP technologies by fostering dialogue, awareness, and proactive measures to address sociodemographic bias, incorporating ideas beyond the field of NLP.

heading: A Appendix
text: A.1 Evaluation Datasets Bias benchmark datasets provide valuable resources for NLP fairness research. These datasets commonly contain illustrative examples of biased language, often templated sentences filled with contrastive social group terms. Datasets allow standardized bias evaluation on diverse tasks using controlled examples. Many of them focus on a particular type of language context, such as coreference, sentiment, or question answering, while others probe for stereotype bias through word associations. Table present in the Appendix summarizes these datasets.
In the case of coreference resolution, Zhao et al. (2018a) proposed a method for identifying gender bias using Winograd-schema sentences for occupation terms. Webster et al. (2018) introduced GAP, a gender-balanced, labeled corpus of 8,908 ambiguous pronoun–name pairs designed to detect gender bias in coreference resolution. In the word association domain, Nangia et al. (2020) presented CrowS-Pairs, a sentence pair corpus that measures a model’s bias by assessing if it favors sentences with stereotypes. Nadeem et al. (2021) released StereoSet, a large-scale natural dataset in English designed to measure stereotypical bias using inter- and intra-sentence association of words to stereotypical contexts. Li et al. (2020) proposed UNQOVER, a general framework for probing bias in question answering models using questions to probe whether a model associates a sociodemographic group to a stereotype. Smith et al. (2022) published HolisticBias, consisting of 450,000 unique sentence prompts for measuring 13 types of sociodemographic bias in generative LMs.
In the domain of sentiment evaluation, Kiritchenko and Mohammad (2018) released EEC, an 8,640 English sentence collection curated to test bias toward certain races and genders in sentiment analysis models. BITS (Venkit and Wilson, 2021; Venkit et al., 2023c) is a similar corpus of 1,126 sentences curated to measure disability, race, and gender bias in sentiment and toxicity analysis models.
Table 2 provides list of datasets for quantifying bias in NLP models.
A.2 List of papers surveyed
Below is the list of papers surveyed in this work, sorted based on our taxonomy.
Explicit Bias(T1) : (Mei et al., 2023; Deas et al., 2023; Liu et al.,
2021; De-Arteaga et al., 2019; Bell and Sagun, 2023; Silva et al., 2021; Park et al., 2018; Sap et al., 2020; B et al., 2021; Lauscher and Glavaš, 2019; Rozado, 2020; Rudinger et al., 2017; Shah et al., 2020; Du et al., 2022; Nozza et al., 2022; Honnavalli et al., 2022; Lucy and Bamman, 2021; Mendelson and Belinkov, 2021; Matthews et al., 2021; Cao et al., 2022; Papakyriakopoulos et al., 2020; Kementchedjhieva et al., 2021; GarridoMuñoz et al., 2021; Strengers et al., 2020; Delobelle et al., 2022; Fisher et al., 2020; Sheng et al., 2020; Zhang et al., 2020a; Hendricks et al.,
2018; Mehrabi et al., 2021; Mayfield et al., 2019; Schwartz et al., 2021; Nozza et al., 2019; Vaidya et al., 2020; He et al., 2019; Hovy and Søgaard, 2015; Wolfe and Caliskan, 2021; Sakaguchi et al., 2021; Agarwal et al., 2019; White and Cotterell, 2021; Luo and Glass, 2023)
Gender Bias : (Sharma et al., 2022; Kaneko et al., 2022a; Stahl et al., 2022; Kaneko et al., 2023; Toro Isaza et al., 2023; Hada et al., 2023; Attanasio et al., 2023; Goldfarb-Tarrant et al., 2023; Lee et al., 2023; Gaut et al., 2020; Sun et al., 2019; Hamidi et al., 2018; Zhou et al., 2019; Savoldi et al., 2021; Sahlgren and Olsson, 2019; Ahn et al., 2022; Tal et al., 2022; Kaneko et al., 2022b; Field and Tsvetkov, 2020; Garimella et al., 2019; Escudé Font and Costa-jussà, 2019; Bhaskaran and Bhallamudi, 2019; McCurdy and Serbetci, 2020; Kaneko and Bollegala, 2019; Larson, 2017; Du et al., 2021; Bartl et al., 2020; Webster et al., 2021; Tan and Celis, 2019; Bolukbasi et al., 2016; Maudslay et al., 2019; Zhao et al., 2019; Rudinger et al., 2018; Lu et al., 2020)
Racial Bias : (Goldfarb-Tarrant et al., 2023; Levy et al., 2023; Field et al., 2023; Cheng et al., 2023; Sap et al., 2019; Hanna et al., 2020; Blodgett et al., 2016; Davidson et al., 2019; Friedman et al., 2019; Shen et al., 2018; Karve et al., 2019; Nadeem et al.,
2021; Garimella et al., 2021; Nangia et al., 2020; Tan and Celis, 2019; Guo and Caliskan, 2021; Brown et al., 2020)
Disability bias : (Venkit and Wilson, 2021; Venkit et al., 2022; Hutchinson et al., 2020; Bennett and Keyes, 2020; Mills and Whittaker, 2019; Hassan et al., 2021; Narayanan Venkit, 2023)
Ethnicity bias : (Bauer et al., 2023; Levy et al., 2023; Malik et al., 2022; Li et al., 2022; Ahn and Oh, 2021; Garg et al., 2018; Li et al., 2020; Abid et al., 2021; Manzini et al., 2019; Venkit et al., 2023b; Bhatt et al., 2022), Nationality bias - (Ladhak et al., 2023; Levy et al., 2023; Narayanan Venkit et al., 2023), Political bias - (Thebault-Spieker et al., 2023; Shen et al., 2018; Rozado, 2020), Age bias (Nangia et al., 2020; Diaz et al., 2018) and sexual-orientation bias (Ovalle et al., 2023; Nangia et al., 2020; Cao and Daumé III, 2020)
Distance based metrics(Q1) : (Caliskan et al., 2017; Dev and Phillips, 2019; Zhao et al., 2017; Basta et al., 2019; Shen et al., 2018; Brunet et al., 2019b; May et al., 2019; Dev et al., 2021; Zhou et al., 2019; Pujari et al., 2020; Sutton et al., 2018; Lauscher et al., 2020; Guo and Caliskan, 2021; Bolukbasi et al., 2016; Ross et al., 2021; Tan
and Celis, 2019; Ethayarajh et al., 2019; Chaloner and Maldonado, 2019; Bordia and Bowman, 2019; Valentini et al., 2023)
Probing based metrics(Q2) : (Orgad et al., 2022; Immer et al., 2022; Chen et al., 2021; Limisiewicz and Mareček, 2021; Kennedy et al., 2020; Sweeney and Najafian, 2019; Tan et al., 2020; Mendelson and Belinkov, 2021; White et al., 2021)
Performance metrics(Q3) : (De-Arteaga et al., 2019; Kwon and Mihindukulasooriya, 2022; Zhang et al., 2022; Huang et al., 2020; Dixon et al., 2018; Zhao et al., 2018a; Cho et al., 2019; Stanovsky et al., 2019; Gonen and Webster, 2020; Borkan et al., 2019; Dev et al., 2020)
Prompt based metrics(Q4) : (Nagireddy et al., 2023; Webster et al., 2021; Smith et al., 2022; Kurita et al., 2019; Krishna et al., 2022; Bhaskaran and Bhallamudi, 2019; Gupta et al., 2022b; Prabhakaran et al., 2019; Ahn and Oh, 2021; Bartl et al., 2020; Li et al., 2020; Venkit and Wilson, 2021; Salazar et al., 2020; Dev et al., 2020; Diaz et al., 2018; Zhang et al., 2020b; Garg et al., 2019; Liang et al., 2022; Kusner et al., 2017; Huang et al., 2020; Akyürek et al., 2022; Gardner et al., 2020; Ousidhoum et al., 2021; Parrish et al., 2022a; Kiritchenko and Mohammad, 2018; Touileb et al., 2023; Gupta et al., 2023; Pikuliak et al., 2023; Touileb et al., 2023; An et al., 2023; Felkner et al., 2023; Ousidhoum et al., 2021)
Debiasing during Finetuning(D1) : (Ungless et al., 2022; Du et al., 2023; Omrani et al., 2023; Zhou et al., 2023a; Thakur et al., 2023; Jin et al., 2021; He et al., 2022b; Zmigrod et al., 2019; Jin et al., 2021; Gaci et al., 2022; Gupta et al., 2022a; Ghaddar et al., 2021; Kumar et al., 2020; Han et al., 2021; Attanasio et al., 2022; Joniak and Aizawa, 2022; Chopra et al., 2020; Maudslay et al., 2019; Park et al., 2018; Yao et al., 2021; Liang et al., 2020; Sen et al., 2022; Ma et al., 2020; Limisiewicz and Mareček, 2022; Yang et al., 2021; Wang et al., 2021; Pujari et al., 2020; Sedoc and Ungar, 2019; Tan et al., 2020; Sutton et al., 2018; Ravfogel et al., 2020; Kaneko and Bollegala, 2019; Karve et al., 2019; Gyamfi et al., 2020; Shin et al., 2020; Zhang et al., 2020a; Wen et al., 2022; Chopra et al., 2020; Yang and Feng, 2020; Lu et al., 2020; Lauscher et al., 2021; Garg et al., 2019; Dev et al., 2020, 2021; Manzini et al., 2019; Bolukbasi et al., 2016; Ahn and Oh, 2021; Orgad et al., 2022; Felkner
et al., 2023; de Vassimon Manela et al., 2021)
Debiasing during Training (D2) : (An et al., 2022; Bolukbasi et al., 2016; He et al., 2019; Han et al., 2022; Liu et al., 2020b; Escudé Font and Costa-jussà, 2019; Prost et al., 2019; James and Alvarez-Melis, 2019; Park et al., 2018; Zhao et al., 2018b; Gao et al., 2022; Sweeney and Najafian, 2020; Hube et al., 2020; Sen and Ganguly, 2020; Saunders and Byrne, 2020; Dixon et al., 2018; Karimi Mahabadi et al., 2020; He et al., 2022a; Richardson et al., 2023) Loss functions for bias mitigation : (Hashimoto et al., 2018; Qian et al., 2019; Berg et al., 2022; Romanov et al., 2019; Garimella et al., 2021; Bordia and Bowman, 2019; Huang et al., 2020; Provilkov and Malinin, 2021; Liu et al., 2021; Orgad and Belinkov, 2023; Li et al., 2023)
Debiasing during Inference (D3) : (Majumder et al., 2023; Qian et al., 2021; Zhao et al., 2019; Abid et al., 2021; Guo et al., 2022; Schick et al., 2021b; Venkit et al., 2023b)
Works on Bias : These are works that are difficult to categorize in one of the above categories. (Chouldechova and Roth, 2020; Green, 2019; Zhang and Bareinboim, 2018; Mayfield et al., 2019; Katell et al., 2020; Dwork et al., 2012; Jacobs et al., 2020; Anoop et al., 2022; Czarnowska et al., 2021; Blodgett et al., 2021; Zhuo et al., 2023; Mulligan et al., 2019; Jacobs and Wallach, 2021; Schoch et al., 2020; Franklin et al., 2022; Bender, 2019; España-Bonet and Barrón-Cedeño, 2022; Hutchinson and Mitchell, 2019; Bender et al., 2021; Goldfarb-Tarrant et al., 2021; Brown et al., 2020; Li et al., 2020; Bagdasaryan et al., 2019; Liu et al., 2020a; Zhiltsova et al., 2019; Chopra et al., 2020; Luo et al., 2023; Shah et al., 2020; Garrido-Muñoz et al., 2021; Delobelle et al., 2022; Czarnowska et al., 2021)

heading: Future Work
text: The survey provides a meticulously designed checklist, based on the weaknesses and limitations of the field, to guide future research toward more effective solutions for mitigating bias. We also emphasize the social and ethical implications of bias underscoring the significance of addressing these issues to prevent potential negative consequences. We hope that our analysis aids in shaping more inclusive and equitable NLP technologies by fostering dialogue, awareness, and proactive measures to address sociodemographic bias, incorporating ideas beyond the field of NLP