cair-nepal/ai-bias-research-landscape huggingface dataset:
Title: Implications of algorithmic bias in AI-driven emergency response systems
Authors: Katsiaryna Bahamazava
Affiliations: Department of Mathematical Sciences G.L. Lagrange, Politecnico di Torino, Corso Duca degli Abruzzi, 24, Torino 10129, Italy
Year: 2025
Domain: General Fairness & Bias Mitigation

Abstract: In this paper, we introduce a framework to evaluate the economic implications of algorithmic bias specifically for the emergency response systems that incorporate AI. Unlike existing research, which mostly addresses technical or ethical aspects in isolation, our approach integrates economic theory with algorithmic fairness to quantify and systematically analyze how biases in data quality and algorithm design impact resource allocation efficiency, response time equity, healthcare outcomes, and social welfare. Using explicit modeling of emergency-specific variables, which includes time sensitivity and urgency, we demonstrate that biases substantially exacerbate demographic disparities. This could lead to delayed emergency responses, inefficient resource utilization, worsened health outcomes, and significant welfare losses. Our numerical simulations further illustrate the economic viability and effectiveness of bias mitigation strategies, such as fairness-constrained optimization and improved data representativeness, in simultaneously enhancing equity and economic efficiency. The framework presented provides policymakers, healthcare providers, and AI developers with actionable insights and a robust economic rationale for deploying equitable AI-driven solutions in emergency management contexts.
Title: The Butterfly Effect in artificial intelligence systems: Implications for AI bias and fairness
Authors: Emilio Ferrara
Affiliations: University College Dublin; University College Dublin; University College Dublin; University College Dublin; University College Dublin
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: The concept of the Butterfly Effect, derived from chaos theory, highlights how seemingly minor changes can lead to significant, unpredictable outcomes in complex systems. This phenomenon is particularly pertinent in the realm of AI fairness and bias. Factors such as subtle biases in initial data, deviations during algorithm training, or shifts in data distribution from training to testing can inadvertently lead to pronounced unfair results. These results often disproportionately impact marginalized groups, reinforcing existing societal inequities. Furthermore, the Butterfly Effect can magnify biases in data or algorithms, intensify feedback loops, and heighten susceptibility to adversarial attacks. Recognizing the complex interplay within AI systems and their societal ramifications, it is imperative to rigorously scrutinize any modifications in algorithms or data inputs for possible unintended effects. This paper proposes a combination of algorithmic and empirical methods to identify, measure, and counteract the Butterfly Effect in AI systems. Our approach underscores the necessity of confronting these challenges to foster equitable outcomes and ensure responsible AI evolution.
Title: Algorithmic bias: Social science research integration through the 3-D Dependable AI Framework
Authors: Kalinda Ukanwa
Affiliations: Hong Kong Baptist University; Hong Kong Baptist University; Hong Kong Baptist University
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: Algorithmic bias has emerged as a critical challenge in the age of responsible production of artificial intelligence (AI). This paper reviews recent research on algorithmic bias and proposes increased engagement of psychological and social science research to understand antecedents and consequences of algorithmic bias. Through the lens of the 3-D Dependable AI Framework, this article explores how social science disciplines, such as psychology, can contribute to identifying and mitigating bias at the Design, Develop, and Deploy stages of the AI life cycle. Finally, we propose future research directions to further address the complexities of algorithmic bias and its societal implications.
Title: EXplainable Artificial Intelligence (XAI) for facilitating recognition of algorithmic bias: An experiment from imposed users' perspectives
Authors: Ching-Hua Chuan; Ruoyu Sun; Shiyun Tian; Wan-Hsiu Sunny Tsai
Affiliations: Carnegie Mellon University; LinkedIn (United States); LinkedIn (United States); LinkedIn (United States)
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: An increasing number of decisions regarding the daily lives of human beings are being controlled by artificial intelligence (AI) algorithms in spheres ranging from healthcare, transportation, and education to college admissions, recruitment, provision of loans and many more realms. Since they now touch on many aspects of our lives, it is crucial to develop AI algorithms that are not only accurate but also objective and fair. Recent studies have shown that algorithmic decision-making may be inherently prone to unfairness, even when there is no intention for it. This paper presents an overview of the main concepts of identifying, measuring and improving algorithmic fairness when using AI algorithms. The paper begins by discussing the causes of algorithmic bias and unfairness and the common definitions and measures for fairness. Fairness-enhancing mechanisms are then reviewed and divided into pre-process, in-process and post-process mechanisms. A comprehensive comparison of the mechanisms is then conducted, towards a better understanding of which mechanisms should be used in different scenarios. The paper then describes the most commonly used fairnessrelated datasets in this field. Finally, the paper ends by reviewing several emerging research sub-fields of algorithmic fairness.
Title: Towards early detection of algorithmic bias from dataset's bias symptoms: An empirical study
Authors: Giordano d'Aloisio; Claudio Di Sipio; Antinisca Di Marco; Davide Di Ruscio
Affiliations: Middle Tennessee State University; Middle Tennessee State University; Middle Tennessee State University
Year: 2025
Domain: General Fairness & Bias Mitigation

Abstract: The rise of AI software has made fairness auditing essential, particularly where biased decisions have serious impacts. This entails identifying sensitive variables and calculating fairness metrics based on predictions from a baseline model. Since model training is computationally intensive, recent research focuses on early bias assessment to detect bias before extensive training starts. This paper presents an empirical study to evaluate how dataset statistics, named bias symptoms , can assist in the early identification of variables that may lead to bias in the system. The aim of this study is to avoid training a machine learning model before assessing - and, in case, mitigating - its bias, thus increasing the sustainability of the development process. We first identify a bias symptoms dataset, employing 24 datasets from diverse application domains commonly used in fairness auditing. Through extensive empirical analysis, we investigate the ability of these bias symptoms to predict variables associated with bias under three fairness definitions. Our results demonstrate that bias symptoms are effective in supporting early predictions of bias-inducing variables under specific fairness definitions. These findings offer valuable insights for practitioners and researchers, encouraging further exploration in developing methods for proactive bias mitigation involving bias symptoms.
Title: Human-Centric AI to Mitigate AI Biases
Authors: Antoine Harfouche; Bernard Quinio; Francesca Bugiotti
Affiliations: Swinburne University of Technology
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: The global health crisis represents an unprecedented opportunity for the development of artificial intelligence (AI) solutions. This article aims to tackle part of the biases in artificial intelligence by implementing a human-centric AI to help decision-makers in organizations. It relies on the results of two design science research (DSR) projects: SCHOPPER and VRAILEXIA. These two design projects operationalize the human-centric AI approach with two complementary stages: 1) the first installs a human-in-loop informed design process, and 2) the second implements a usage architecture that aggregates AI and humans. The proposed framework offers many advantages such as permitting to integrate of human knowledge into the design and training of the AI, providing humans with an understandable explanation of their predictions, and driving the advent of augmented intelligence that can turn algorithms into a powerful counterweight to human decision-making errors and humans as a counterweight to AI biases.
Title: Teaching AI awareness in middle school classrooms: Design, implementation and evaluation of two education modules on algorithmic bias and filter bubbles
Authors: Mehtab Iqbal et al.
Affiliations: Ben-Gurion University of the Negev; University of Sussex; Ben-Gurion University of the Negev; Beth Israel Deaconess Medical Center; Harvard–MIT Division of Health Sciences and Technology; Ben-Gurion University of the Negev; BI Norwegian Business School
Year: 2025
Domain: General Fairness & Bias Mitigation

Abstract: The nascent digital footprints of adolescents make them particularly susceptible to being misrepresented by AI systems. We design, implement, and evaluate two educational modules on the causes and consequences of two particularly salient types of AI misrepresentation: algorithmic biases and filter bubbles. We implement these modules in middle school math and computer science classrooms by integrating our materials with the existing curricula and curricular requirements of the partner school. This integration, as well as the many confounding factors of the classroom, are nontrivial problems that have heretofore not been considered in privacy education research and required us to adopt methodologies from learning sciences, particularly design-based research and backward design. Quantitative results of the pre-/post-test and qualitative assessments of open-ended questions find that students were engaged with the materials and exhibited substantial learning gains. The findings emphasize the need for tailored educational interventions and avenues for future research to address AI-related challenges.
Title: Fairness in precision medicine: Evaluating algorithmic bias in length of stay and readmission for asthma in the All of Us Research Program
Authors: Tesfaye B. Mersha; Esteban Correa-Agudelo; Yadu Gautam
Affiliations: La Trobe University; La Trobe University; La Trobe University
Year: 2025
Domain: Health & Clinical AI

Abstract: Neural Machine Translation (NMT) continues to improve in quality and adoption, yet the inadvertent perpetuation of gender bias remains a significant concern. Despite numerous studies on gender bias in translations into English from weakly gendered-languages, there are no benchmarks for evaluating this phenomenon or for assessing mitigation strategies. To address this gap, we introduce GATE X-E, an extension to the GATE (Rarrick et al., 2023) corpus, that consists of human translations from Turkish, Hungarian, Finnish, and Persian into English. Each translation is accompanied by feminine, masculine, and neutral variants. The dataset, which contains between 1250 and 1850 instances for each of the four language pairs, features natural sentences with a wide range of sentence lengths and domains, challenging translation rewriters on various linguistic phenomena. Additionally, we present a translation gender rewriting solution built with GPT-4 and use GATE X-E to evaluate it. We open source1 our contributions to encourage further research on gender debiasing.
Title: Understanding AI bias in clinical practice
Authors: Demilade Adedinsewo; Sana M. Al-Khatib
Affiliations: Universidade Federal de São Paulo; Instituto Federal de Goiás; Idiap Research Institute; Universidade Federal de São Paulo
Year: 2024
Domain: Health & Clinical AI

Abstract: A recent analysis highlighting the potential for algorithms to perpetuate existing racial biases in healthcare underscores the importance of thinking carefully about the labels used during algorithm development.
Title: Multidisciplinary considerations of fairness in medical AI: A scoping review
Authors: Yue Wang; Yaxin Song; Zhuo Ma; Xiaoxue Han
Affiliations: Tsinghua University; Shanghai Jiao Tong University; Tsinghua University; University of Waterloo; Tsinghua University; Tsinghua University; Tsinghua University; Tsinghua University; Tsinghua University; Tsinghua University; Tsinghua University
Year: 2023
Domain: Health & Clinical AI

Abstract: Data-driven innovation (DDI) gains its prominence due to its potential to transform innovation in the age of AI. Digital giants Amazon, Alibaba, Google, Apple, and Facebook, enjoy sustainable competitive advantages from DDI. However, little is known about algorithmic biases that may present in the DDI process, and result in unjust, unfair, or prejudicial data product developments. Thus, this guest editorial aims to explore the sources of algorithmic biases across the DDI process using a systematic literature review, thematic analysis and a case study on the Robo-Debt scheme in Australia. The findings show that there are three major sources of algorithmic bias: data bias, method bias and societal bias. Theoretically, the findings of our study illuminate the role of the dynamic managerial capability to address various biases. Practically, we provide guidelines on addressing algorithmic biases focusing on data, method and managerial capabilities.
Title: Rethinking fairness in unsupervised healthcare AI: A methodological scoping review
Authors: Malek Adouani; Djillali Annane; Zaineb Chelly Dagdia
Affiliations: University of Illinois Chicago; University of Illinois Urbana-Champaign
Year: 2026
Domain: Health & Clinical AI

Abstract: OBJECTIVE: Fairness in machine learning has been extensively studied in supervised settings, where labeled outcomes allow direct assessment of bias. In contrast, fairness in unsupervised learning-particularly in healthcare remains insufficiently examined. In the absence of labels, it is unclear how fairness should be defined or evaluated for discovered structures such as patient subgroups, disease subtypes, or trajectories, despite their growing influence on clinical understanding and decision-making. This review aims to systematically examine how fairness is conceptualized, operationalized, and evaluated in unsupervised healthcare AI. METHODS: We conducted a PRISMA-guided methodological scoping review of the literature on fairness in unsupervised learning applied to healthcare data. The review focused on identifying algorithmic mechanisms, evaluation strategies, and methodological assumptions rather than comparing predictive performance or quantitatively synthesizing results across the reviewed literature. Records were analyzed with respect to data modalities, unsupervised learning techniques, and the underlying definitions of fairness they employed. RESULTS: The review reveals rapid growth in interest in fairness-aware unsupervised healthcare AI, accompanied by substantial heterogeneity and conceptual inconsistency. Fairness is addressed across diverse data types and methodological approaches, often without explicit alignment to clinical or ethical objectives. To structure this fragmented landscape, we propose a taxonomy of fairness approaches organized into five families: Individual Fairness, Performance Dependence, Welfare-Anchored approaches, Statistical Inference-based approaches, and Representation Parity. Each family embodies a distinct conception of equity and entails specific ethical and methodological trade-offs. We further identify recurring challenges, including defining fairness without labeled outcomes, limited incorporation of clinical expertise, and weak alignment between fairness objectives and medical validity. CONCLUSION: Fairness in unsupervised healthcare AI is an emerging but conceptually unsettled field. Current approaches reflect diverse and sometimes incompatible notions of equity, underscoring the need for clearer theoretical grounding. Progress will require explicit articulation of fairness goals, stronger integration of domain expertise and participatory evaluation, and closer alignment between algorithmic fairness criteria and clinically meaningful structures. This review provides a conceptual and methodological foundation to support more rigorous and transparent development of fair unsupervised healthcare AI systems.
Title: Ontology-grounded knowledge graphs for mitigating hallucinations in large language models for clinical question answering
Authors: Mohamed Ali; Zaki Taha; Mohamed Mabrouk Morsey
Affiliations: University of Oxford; University of Oxford; University of Oxford; University of Oxford; University of Oxford
Year: 2026
Domain: Graph-Based Fairness & Bias Mitigation

Abstract: In this tutorial, we delve into recent advances in explainable recommendation using Knowledge Graphs (KGs). The session begins by introducing the fundamental principles behind the increasing adoption of KGs in modern recommender systems. Then, the tutorial explores recent techniques that leverage KGs as an input for language models tailored to explainable recommendation, describing also data types, methods, and evaluation protocols and metrics. Conceptual elements are complemented with hands-on sessions, providing practical implementations using open-source tools and public datasets. Concluding with a comprehensive case study in the education domain as a recap, the tutorial analyses emerging issues and outlines prospective trajectories in this field. The tutorial website is available at https://explainablerecsys.github.io/ecir2024/.
Title: Toward fair medical advice: Addressing and mitigating bias in large language model-based healthcare applications
Authors: Haohui Lu; Ye Lin; Zhidong Li; Man Lung Yiu; Yu Gao; Shahadat Uddin
Affiliations: Charles Darwin University; University of Technology Sydney; University of Technology Sydney; The Hong Kong Polytechnic University; Charles Darwin University; The University of Sydney
Year: 2025
Domain: Health & Clinical AI

Abstract: Large Language Models (LLMs) are increasingly deployed in web-based medical advice applications, offering scalable and accessible healthcare solutions. However, their outputs often reflect demographic biases, raising concerns about fairness and equity for vulnerable populations. In this work, we propose FairMed, a framework designed to mitigate biases in LLM-generated medical advice through fine-tuning and prompt engineering strategies. We evaluate FairMed using language-based and content-level metrics across demographic groups on publicly available (MedQA), synthetic (Synthea), and private (CBHS) datasets. Experimental results demonstrate consistent improvements over Llama3 - Med42, as well as over the zero-shot prompting baseline. For instance, in sentiment analysis for gender groups using MedQA, FairMed with Descriptive Prompting reduces the Statistical Parity Difference (SPD) from 0.0902 to 0.0658, improves the Disparate Impact Ratio from 1.1916 to 1.1566, and decreases the Kullback-Leibler Divergence from 0.0045 to 0.0024. Similarly, in directive language evaluation for gender groups using Synthea, SPD improves from 0.1056 to nearly zero, achieving near-perfect parity. On the CBHS dataset, FairMed with Descriptive Prompting increases Diagnostic Recommendation Divergence (DRD) for race groups from 0.9530 to 0.9848, indicating improved group-specific tailoring, while reducing the Action Disparity Index (ADI) from 0.0857 to 0.0469 and Referral Frequency Parity (RFP) from 0.0791 to 0.0511, reflecting enhanced fairness. These findings highlight FairMed's effectiveness in addressing demographic disparities and promoting equitable healthcare guidance through web technologies. This framework contributes to building trustworthy and inclusive systems for delivering medical advice by ensuring fairness in sensitive applications.
Title: Meta-Fair: AI-assisted fairness testing of large language models
Authors: Miguel Romero-Arjona; José A. Parejo; Juan C. Alonso; Ana B. Sánchez; Aitor Arrieta; Sergio Segura
Affiliations: Shanghai Normal University; Shanghai International Studies University; Shanghai Normal University
Year: 2026
Domain: LLMs & NLP

Abstract: Fairness—the absence of unjustified bias—is a core principle in the development of Artificial Intelligence (AI) systems, yet it remains difficult to assess and enforce. Current approaches to fairness testing in large language models (LLMs) often rely on manual evaluation, fixed templates, deterministic heuristics, and curated datasets, making them resource-intensive and difficult to scale. This work aims to lay the groundwork for a novel, automated method for testing fairness in LLMs, reducing the dependence on domain-specific resources and broadening the applicability of current approaches. Our approach, Meta-Fair, is based on two key ideas. First, we adopt metamorphic testing to uncover bias by examining how model outputs vary in response to controlled modifications of input prompts, defined by metamorphic relations (MRs). Second, we propose exploiting the potential of LLMs for both test case generation and output evaluation, leveraging their capability to generate diverse inputs and classify outputs effectively. The proposal is complemented by three open-source tools supporting LLM-driven generation, execution, and evaluation of test cases. We report the findings of several experiments involving 12 pre-trained LLMs, 14 MRs, 5 bias dimensions, and 7.9K automatically generated test cases. The results show that Meta-Fair is effective in uncovering bias in LLMs, achieving an average precision of 92% and revealing biased behaviour in 29% of executions. Additionally, LLMs prove to be reliable and consistent evaluators, with the best-performing models achieving F1-scores of up to 0.79. Although non-determinism affects consistency, these effects can be mitigated through careful MR design. This work highlights the feasibility and potential of integrating metamorphic testing with LLM-driven test generation and assessment. While challenges remain to ensure broader applicability, the results indicate a promising path towards an unprecedented level of automation in LLM testing.