cair-nepal/ai-bias-research-landscape huggingface dataset:
Title: WEIRD Audits? Research Trends, Linguistic and Geographical Disparities in the Algorithm Audits of Online Platforms - A Systematic Literature Review
Authors: Aleksandra Urman; Mykola Makhortykh; Anikó Hannák
Affiliations: University of Zurich; University of Bern; University of Zurich
Year: 2025
Domain: General Fairness & Bias Mitigation

Abstract: The increasing reliance on complex algorithmic systems by online platforms has sparked a growing need for algorithm auditing, a methodology evaluating these systems' functionality and impact.In this paper, we systematically review 176 peer-reviewed online platform-focused algorithm auditing studies and identify trends in their methodological approaches, the geographic distribution of authors, and the selection of platforms, languages, geographies, and group-based attributes in the focus of the reviewed research.We find a significant skew of research focus towards few online platforms, Western contexts, particularly the US, and English language data.Additionally, our analysis indicates a tendency to focus on a narrow set of group-based attributes, often operationalized in simplified ways, which might obscure more nuanced aspects of algorithmic bias and discrimination.We provide a clearer understanding of the current state of the online platform-focused algorithm auditing and identify gaps to be addressed for a more inclusive and representative research landscape.
Title: Epistemic Injustice in Algorithmic Systems and the Limits of Civil Rights Law
Authors: Alexandra Wood; Elizabeth Edenberg
Affiliations: Harvard University; Baruch College
Year: 2025
Domain: General Fairness & Bias Mitigation

Abstract: Mathematical definitions of algorithmic fairness have become central to efforts to address potential discriminatory effects from machine learning and artificial intelligence applications.These quantitative measures carry significant weight, derived in large part from their purported grounding in legal conceptions of nondiscrimination, and they are increasingly deployed in a wide range of highly consequential settings.This paper critically examines how these mathematical constructs diverge from their legal foundations, with a focus on the anticlassification and antisubordination principles underpinning US civil rights law.It argues that applying an epistemic lens on algorithmic fairness previously introduced in the literature provides an opportunity for better aligning technical definitions of fairness with legal notions of nondiscrimination.In particular, it offers a theoretical foundation for affirming the antisubordination principle and thereby extending the protections of antidiscrimination law to classes of algorithmic harm that have yet to be formally recognized.
Title: It's only fair when I think it's fair: How Gender Bias Alignment Undermines Distributive Fairness in Human-AI Collaboration
Authors: Domenique Zipperling; Luca Deck; Julia Lanzl; Niklas Kühl
Affiliations: Fraunhofer Institute for Applied Information Technology; University of Bayreuth; University of Hohenheim; Fraunhofer Institute for Applied Information Technology
Year: 2025
Domain: LLMs & NLP

Abstract: Human-AI collaboration is increasingly relevant in consequential areas where AI recommendations support human discretion. However, human-AI teams’ effectiveness, capability, and fairness highly depend on human perceptions of AI. Positive fairness perceptions have been shown to foster trust and acceptance of AI recommendations. Yet, work on confirmation bias highlights that humans selectively adhere to AI recommendations that align with their expectations and beliefs—despite not being necessarily correct or fair. This raises the question whether confirmation bias also transfers to the alignment of gender bias between human and AI decisions. In our study, we examine how gender bias alignment influences fairness perceptions and reliance. The results of a 2x2 between-subject study highlight the connection between gender bias alignment, fairness perceptions, and reliance, demonstrating that merely constructing a “formally fair” AI system is insufficient for optimal human-AI collaboration; ultimately, AI recommendations will likely be overridden if biases do not align.1261
Title: Expressing stigma and inappropriate responses prevents LLMs from safely replacing mental health providers.
Authors: Jared Moore; Declan Grabb; William Agnew; Kevin Klyman; Stevie Chancellor; Desmond C. Ong; Nick Haber
Affiliations: Palo Alto University; Palo Alto University; Palo Alto University; University of Minnesota; The University of Texas at Austin; Palo Alto University
Year: 2025
Domain: General Fairness & Bias Mitigation

Abstract: Should a large language model (LLM) be used as a therapist? In this paper, we investigate the use of LLMs to *replace* mental health providers, a use case promoted in the tech startup and research space. We conduct a mapping review of therapy guides used by major medical institutions to identify crucial aspects of therapeutic relationships, such as the importance of a therapeutic alliance between therapist and client. We then assess the ability of LLMs to reproduce and adhere to these aspects of therapeutic relationships by conducting several experiments investigating the responses of current LLMs, such as `gpt-4o`. Contrary to best practices in the medical community, LLMs 1) express stigma toward those with mental health conditions and 2) respond inappropriately to certain common (and critical) conditions in naturalistic therapy settings -- e.g., LLMs encourage clients' delusional thinking, likely due to their sycophancy. This occurs even with larger and newer LLMs, indicating that current safety practices may not address these gaps. Furthermore, we note foundational and practical barriers to the adoption of LLMs as therapists, such as that a therapeutic alliance requires human characteristics (e.g., identity and stakes). For these reasons, we conclude that LLMs should not replace therapists, and we discuss alternative roles for LLMs in clinical therapy.
Title: Disambiguating Algorithmic Bias: From Neutrality to Justice
Authors: Elizabeth Edenberg; Alexandra Wood
Affiliations: Baruch College
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: As algorithms have become ubiquitous in consequential domains, societal concerns about the potential for discriminatory outcomes have prompted urgent calls to address algorithmic bias. In response, a rich literature across computer science, law, and ethics is rapidly proliferating to advance approaches to designing fair algorithms. Yet computer scientists, legal scholars, and ethicists are often not speaking the same language when using the term ‘bias.’ Debates concerning whether society can or should tackle the problem of algorithmic bias are hampered by conflations of various understandings of bias, ranging from neutral deviations from a standard to morally problematic instances of injustice due to prejudice, discrimination, and disparate treatment. This terminological confusion impedes efforts to address clear cases of discrimination.
Title: Evaluating the Fairness of Discriminative Foundation Models in Computer Vision
Authors: Junaid Ali; Matthäus Kleindeßner; Florian Wenzel; Kailash Budhathoki; Volkan Cevher; Chris Russell
Affiliations: Max Planck Institute for Software Systems; Amazon (Germany); Amazon (Germany); Amazon (Germany); Amazon (Germany); Amazon (Germany)
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: We propose a novel taxonomy for bias evaluation of discriminative foundation models, such as Contrastive Language-Pretraining (CLIP), that are used for labeling tasks. We then systematically evaluate existing methods for mitigating bias in these models with respect to our taxonomy. Specifically, we evaluate OpenAI’s CLIP and OpenCLIP models for key applications, such as zero-shot classification, image retrieval and image captioning. We categorize desired behaviors based around three axes: (i) if the task concerns humans; (ii) how subjective the task is (i.e., how likely it is that people from a diverse range of backgrounds would agree on a labeling); and (iii) the intended purpose of the task and if fairness is better served by impartiality (i.e., making decisions independent of the protected attributes) or representation (i.e., making decisions to maximize diversity). Finally, we provide quantitative fairness evaluations for both binary-valued and multi-valued protected attributes over ten diverse datasets. We find that fair PCA, a post-processing method for fair representations, works very well for debiasing in most of the aforementioned tasks while incurring only minor loss of performance. However, different debiasing approaches vary in their effectiveness depending on the task. Hence, one should choose the debiasing approach depending on the specific use case.
Title: Stress-Testing Bias Mitigation Algorithms to Understand Fairness Vulnerabilities
Authors: Karan Bhanot; Ioana Baldini; Dennis Wei; Jiaming Zeng; Kristin P. Bennett
Affiliations: Rensselaer Polytechnic Institute; Akamai (United States); Rensselaer Polytechnic Institute
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: To address the growing concern of unfairness in Artificial Intelligence (AI), several bias mitigation algorithms have been introduced in prior research. Their capabilities are often evaluated on certain overly-used datasets without rigorously stress-testing them under simultaneous train and test distribution shifts. To address this, we investigate the fairness vulnerabilities of these algorithms across several distribution shift scenarios using synthetic data, to highlight scenarios where these algorithms do and don’t work to encourage their trustworthy use. The paper makes three important contributions. Firstly, we propose a flexible pipeline called the Fairness Auditor to systematically stress-test bias mitigation algorithms using multiple synthetic datasets with shifts. Secondly, we introduce the Deviation Metric for measuring the fairness and utility performance of these algorithms under such shifts. Thirdly, we propose an interactive reporting tool for comparing algorithmic performance across various synthetic datasets, mitigation algorithms and metrics called the Fairness Report.
Title: Unmasking Nationality Bias: A Study of Human Perception of Nationalities in AI-Generated Articles
Authors: Pranav Narayanan Venkit; Sanjana Gautam; Ruchi Panchanadikar; Ting-Hao Huang; Shomir Wilson
Affiliations: Pennsylvania State University; Pennsylvania State University; Pennsylvania State University; Pennsylvania State University; Pennsylvania State University
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: We investigate the potential for nationality biases in natural language processing (NLP) models using human evaluation methods. Biased NLP models can perpetuate stereotypes and lead to algorithmic discrimination, posing a significant challenge to the fairness and justice of AI systems. Our study employs a two-step mixed-methods approach that includes both quantitative and qualitative analysis to identify and understand the impact of nationality bias in a text generation model. Through our human-centered quantitative analysis, we measure the extent of nationality bias in articles generated by AI sources. We then conduct open-ended interviews with participants, performing qualitative coding and thematic analysis to understand the implications of these biases on human readers. Our findings reveal that biased NLP models tend to replicate and amplify existing societal biases, which can translate to harm if used in a sociotechnical setting. The qualitative analysis from our interviews offers insights into the experience readers have when encountering such articles, highlighting the potential to shift a reader’s perception of a country. These findings emphasize the critical role of public perception in shaping AI’s impact on society and the need to correct biases in AI systems.
Title: Not So Fair: The Impact of Presumably Fair Machine Learning Models
Authors: Mackenzie Jorgensen; Hannah Richert; Elizabeth Black; Natalia Criado; José M. Such
Affiliations: King's College London; Osnabrück University; King's College London; Universitat Politècnica de València; King's College London
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: When bias mitigation methods are applied to make fairer machine learning models in fairness-related classification settings, there is an assumption that the disadvantaged group should be better off than if no mitigation method was applied. However, this is a potentially dangerous assumption because a “fair” model outcome does not automatically imply a positive impact for a disadvantaged individual—they could still be negatively impacted. Modeling and accounting for those impacts is key to ensure that mitigated models are not unintentionally harming individuals; we investigate if mitigated models can still negatively impact disadvantaged individuals and what conditions affect those impacts in a loan repayment example. Our results show that most mitigated models negatively impact disadvantaged group members in comparison to the unmitigated models. The domain-dependent impacts of model outcomes should help drive future bias mitigation method development.
Title: Target specification bias, counterfactual prediction, and algorithmic fairness in healthcare
Authors: Eran Tal
Affiliations: McGill University
Year: 2023
Domain: Health & Clinical AI

Abstract: Bias in applications of machine learning (ML) to healthcare is usually attributed to unrepresentative or incomplete data, or to underlying health disparities. This article identifies a more pervasive source of bias that affects the clinical utility of ML-enabled prediction tools: target specification bias. Target specification bias arises when the operationalization of the target variable does not match its definition by decision makers. The mismatch is often subtle, and stems from the fact that decision makers are typically interested in predicting the outcomes of counterfactual, rather than actual, healthcare scenarios. Target specification bias persists independently of data limitations and health disparities. When left uncorrected, it gives rise to an overestimation of predictive accuracy, to inefficient utilization of medical resources, and to suboptimal decisions that can harm patients. Recent work in metrology – the science of measurement – suggests ways of counteracting target specification bias and avoiding its harmful consequences.
Title: Perceived Algorithmic Fairness using Organizational Justice Theory: An Empirical Case Study on Algorithmic Hiring
Authors: Guusje Juijn; Niya Stoimenova; João Reis; Dong Nguyen
Affiliations: Utrecht University; Utrecht University
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: Growing concerns about the fairness of algorithmic decision-making systems have prompted a proliferation of mathematical formulations aimed at remedying algorithmic bias. Yet, integrating mathematical fairness alone into algorithms is insufficient to ensure their acceptance, trust, and support by humans. It is also essential to understand what humans perceive as fair. In this study, we, therefore, conduct an empirical user study into crowdworkers’ algorithmic fairness perceptions, focusing on algorithmic hiring. We build on perspectives from organizational justice theory, which categorizes fairness into distributive, procedural, and interactional components. By doing so, we find that algorithmic fairness perceptions are higher when crowdworkers are provided not only with information about the algorithmic outcome but also about the decision-making process. Remarkably, we observe this effect even when the decision-making process can be considered unfair, when gender, a sensitive attribute, is used as a main feature. By showing realistic trade-offs between fairness criteria, we moreover find a preference for equalizing false negatives over equalizing selection rates amongst groups. Our findings highlight the importance of considering all components of algorithmic fairness, rather than solely treating it as an outcome distribution problem. Importantly, our study contributes to the literature on the connection between mathematical– and perceived algorithmic fairness, and highlights the potential benefits of leveraging organizational justice theory to enhance the evaluation of perceived algorithmic fairness.
Title: Policy advice and best practices on bias and fairness in AI
Authors: José M. Álvarez; Alejandra Bringas Colmenarejo; Alaa Elobaid; Simone Fabbrizzi; Miriam Fahimi; Antonio Ferrara; Siamak Ghodsi; Carlos Mougan; Ioanna Papageorgiou; Paula Reyero Lobo; Mayra Russo; Kristen M. Scott; Laura State; Xuan Zhao; Salvatore Ruggieri
Affiliations: University of Pisa; University of Southampton; Centre for Research and Technology Hellas; Leibniz University Hannover; University of Klagenfurt; GESIS - Leibniz-Institute for the Social Sciences; Leibniz University Hannover; University of Southampton; Leibniz University Hannover; The Open University; Leibniz University Hannover; KU Leuven; University of Pisa; University of Pisa
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: Abstract The literature addressing bias and fairness in AI models ( fair-AI ) is growing at a fast pace, making it difficult for novel researchers and practitioners to have a bird’s-eye view picture of the field. In particular, many policy initiatives, standards, and best practices in fair-AI have been proposed for setting principles, procedures, and knowledge bases to guide and operationalize the management of bias and fairness. The first objective of this paper is to concisely survey the state-of-the-art of fair-AI methods and resources, and the main policies on bias in AI, with the aim of providing such a bird’s-eye guidance for both researchers and practitioners. The second objective of the paper is to contribute to the policy advice and best practices state-of-the-art by leveraging from the results of the NoBIAS research project. We present and discuss a few relevant topics organized around the NoBIAS architecture, which is made up of a Legal Layer, focusing on the European Union context, and a Bias Management Layer, focusing on understanding, mitigating, and accounting for bias.
Title: The Illusion of Artificial Inclusion
Authors: William S. Agnew; A. S. Bergman; Jennifer Chien; Mark Díaz; Seliem El-Sayed; Jaylen Pittman; Shakir Mohamed; Kevin R. McKee
Affiliations: Carnegie Mellon University; Google DeepMind (United Kingdom); University of California San Diego; Google (United States); Google DeepMind (United Kingdom); Stanford University; Google DeepMind (United Kingdom); Google DeepMind (United Kingdom)
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: Human participants play a central role in the development of modern artificial intelligence (AI) technology, in psychological science, and in user research. Recent advances in generative AI have attracted growing interest to the possibility of replacing human participants in these domains with AI surrogates. We survey several such “substitution proposals” to better understand the arguments for and against substituting human participants with modern generative AI. Our scoping review indicates that the recent wave of these proposals is motivated by goals such as reducing the costs of research and development work and increasing the diversity of collected data. However, these proposals ignore and ultimately conflict with foundational values of work with human participants: representation, inclusion, and understanding. This paper critically examines the principles and goals underlying human participation to help chart out paths for future work that truly centers and empowers participants.
Title: Navigating the Audit Landscape: A Framework for Developing Transparent and Auditable XR
Authors: Chris Norval; Richard Cloete; Jatinder Singh
Affiliations: University of Cambridge; University of Cambridge
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: “Extended reality” (XR) systems work to blend the physical and digital worlds. This means that XR is highly contextual: its functionality, operation and therefore consequences are driven by a tight, run-time coupling of the technology, the user, and their physical environment. It follows that XR brings particular challenges regarding transparency and accountability, given that it can be difficult to foresee and mitigate all potential issues that might arise from using such systems, given their many potential contexts of use. Further, the physicality of XR can directly result in injury, property damage, or worse, in addition to the more traditionally discussed harms arising from algorithmic systems. Therefore the ability to audit the operation of XR systems is paramount – where information revealing and enabling some reconstruction of an XR system’s use, run-time behaviour, and surrounding context is important for understanding and scrutinising what happens/happened, and why.
Title: Datafication Genealogies beyond Algorithmic Fairness: Making Up Racialised Subjects
Authors: Ana Valdivia; Martina Tazzioli
Affiliations: University of Oxford; Goldsmiths University of London
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: A growing scholarship has discussed how datafication is grounded on algorithmic discrimination. However, these debates only marginally address how racialised classification or race categories are enforced through quantification and neglect its political and historical conceptualisation. In this work, we argue that literature partially fails to show that datafication reinforces racial profiling beyond the creation of racial categories as features. This article casts a new light on datafication by retracing its genealogy focusing on identification procedures in the colony and at the border. Such a genealogy foregrounds how datafication enforces racialised profiles by showing that it is part of a longer historical trajectory of modes of racialising individuals beyond algorithms and racial categories. Building on archival material, it develops this argument through two case studies. First, it focuses on the study of datafication of colonised bodies through biometrics by Francis Galton during the 19th-century. Second, it takes into account police identification procedures about unauthorised migrants, enforced by the French police at the Italian border in the 20th-century. These two cases show that although race categories as variables have been historically used to translate individuals into data, datafication processes as such also produce racialised profiles. A genealogical approach highlights continuities as well as quantitative and qualitative shifts between analogue and digital datafication. The article concludes arguing that datafication mechanisms have historically enforced legal and political measures by states in the name of science and objectivity and debates around algorithmic fairness should bring this key aspect back to the core of their critiques.