cair-nepal/ai-bias-research-landscape huggingface dataset:
Title: AI Bias in Power Systems Domain—Exemplary Cases and Approaches
Authors: Chijioke Eze; Abraham Ezema; L. E. Roth; Zhiyu Pan; Ferdinanda Ponci; Antonello Monti
Affiliations: RWTH Aachen University; RWTH Aachen University; RWTH Aachen University; RWTH Aachen University; RWTH Aachen University; Fraunhofer Institute for Production Technology IPT
Year: 2025
Domain: General Fairness & Bias Mitigation

Abstract: This paper examines artificial intelligence (AI) bias in power systems applications through systematic analysis of three critical use cases: load forecasting, predictive maintenance, and ontology matching for system interoperability. While AI solutions show great potential for addressing complex power system challenges, they face adoption barriers due to biases that compromise fairness, reliability, and operational performance. Our investigation demonstrates how different bias types—including data representation, algorithmic, and sampling biases—manifest in power systems contexts, directly affecting grid efficiency, resource allocation, and socioeconomic equity across the electrical power and energy domain. For each use case, we provide quantitative evidence of bias impact and propose targeted mitigation strategies that emphasize data diversity, ensemble methods, explainable AI techniques, and fairness-aware algorithms. By establishing a comprehensive taxonomy of bias types relevant to power systems and developing practical mitigation frameworks, this work bridges the critical gap between abstract bias concepts and real-world power system applications. The resulting framework provides a structured approach for developing equitable, robust AI systems that align with power systems’ operational requirements while accelerating the responsible adoption of AI in safety-critical infrastructure.
Title: Bias in Conversational Search: The Double-Edged Sword of the Personalized Knowledge Graph
Authors: Emma J. Gerritse; Faegheh Hasibi; Arjen P. de Vries
Affiliations: Radboud University Nijmegen; Radboud University Nijmegen; Radboud University Nijmegen
Year: 2020
Domain: Graph-Based Fairness & Bias Mitigation

Abstract: Conversational AI systems are being used in personal devices, providing users with highly personalized content. Personalized knowledge graphs (PKGs) are one of the recently proposed methods to store users' information in a structured form and tailor answers to their liking. Personalization, however, is prone to amplifying bias and contributing to the echo-chamber phenomenon. In this paper, we discuss different types of biases in conversational search systems, with the emphasis on the biases that are related to PKGs. We review existing definitions of bias in the literature: people bias, algorithm bias, and a combination of the two, and further propose different strategies for tackling these biases for conversational search systems. Finally, we discuss methods for measuring bias and evaluating user satisfaction.
Title: Comprehensive Fair Meta-learned Recommender System
Authors: Tianxin Wei; Jingrui He
Affiliations: University of Illinois Urbana-Champaign; University of Illinois Urbana-Champaign
Year: 2022
Domain: Recommender Systems

Abstract: In recommender systems, one common challenge is the cold-start problem, where interactions are very limited for fresh users in the systems. To address this challenge, recently, many works introduce the meta-optimization idea into the recommendation scenarios, i.e. learning to learn the user preference by only a few past interaction items. The core idea is to learn global shared meta-initialization parameters for all users and rapidly adapt them into local parameters for each user respectively. They aim at deriving general knowledge across preference learning of various users, so as to rapidly adapt to the future new user with the learned prior and a small amount of training data. However, previous works have shown that recommender systems are generally vulnerable to bias and unfairness. Despite the success of meta-learning at improving the recommendation performance with cold-start, the fairness issues are largely overlooked.
Title: START Foundation: Coping with Bias and Fairness when Implementing and Using an AI System
Authors: Chiara Schwenke; Julia Brasse; Maximilian Förster; Mathias Klier
Affiliations: Ulm University; Ulm University; Ulm University; Ulm University
Year: 2024
Domain: General Fairness & Bias Mitigation

Abstract: START Foundation annually supports nearly 200 young people with a migratory background through an individualized scholarship program aimed at personal development and identity building. In 2022, START Foundation expanded its reach by launching a digital learning platform, which faced challenges such as information overload and navigation difficulties due to a growing number of courses and a lack of individual support. To improve user experience on the digital learning platform, START Foundation explored implementing a recommender system for personalized course navigation based on methods from the field of artificial intelligence (AI). However, this exploration raised ethical concerns, particularly regarding bias and fairness in AI systems. In light of these concerns, the following questions emerged: Should START Foundation integrate the AI-based recommender system into its digital learning platform? If so, what factors should START Foundation consider regarding bias and fairness in its AI-based recommender system in order to prevent negative consequences for its scholars? Readers are encouraged to explore the interactive learning module on bias and fairness in AI systems, available at https://bias-and-fairness-in-ai-systems.de/en/home/.
Title: Measuring Inter-Bias Effects and Fairness-Accuracy Trade-Offs in GNN-Based Recommender Systems
Authors: Nikzad Chizari; Keywan Tajfar; María N. Moreno García
Affiliations: Universidad de Salamanca; University of Tehran; Universidad de Salamanca
Year: 2025
Domain: Graph-Based Fairness & Bias Mitigation

Abstract: Bias in artificial intelligence is a critical issue because these technologies increasingly influence decision-making in a wide range of areas. The recommender system field is one of them, where biases can lead to unfair or skewed outcomes. The origin usually lies in data biases coming from historical inequalities or irregular sampling. Recommendation algorithms using such data contribute to a greater or lesser extent to amplify and perpetuate those imbalances. On the other hand, different types of biases can be found in the outputs of recommender systems, and they can be evaluated by a variety of metrics specific to each of them. However, biases should not be treated independently, as they are interrelated and can potentiate or mask each other. Properly assessing the biases is crucial for ensuring fair and equitable recommendations. This work focuses on analyzing the interrelationship between different types of biases and proposes metrics designed to jointly evaluate multiple interrelated biases, with particular emphasis on those biases that tend to mask or obscure discriminatory treatment against minority or protected demographic groups, evaluated in terms of disparities in recommendation quality outcomes. This approach enables a more comprehensive assessment of algorithmic performance in terms of both fairness and predictive accuracy. Special attention is given to Graph Neural Network-based recommender systems, due to their strong performance in this application domain.
Title: A framework for solving bias in graph-based recommender systems with a causal perspective
Authors: Ke Yang; Guogang Li; L. Wang; Jianrong Xie
Affiliations: Huaqiao University; Huaqiao University; Huaqiao University; Huaqiao University
Year: 2025
Domain: Graph-Based Fairness & Bias Mitigation

Abstract: Cross-Domain Recommendation (CDR) is an effective approach for mitigating the cold-start problem. However, previous research largely overlooks the issue of fairness: undesired bias associated with sensitive attributes in CDR scenarios. Specifically, through our data-driven experiments, we find that existing CDR-based methods are particularly vulnerable to the distribution bias in sparse cross-domain data, and such bias easily results in unfair treatment of cold-start users in the target domain, eventually exacerbating the recommendation feedback loop. Additionally, existing fairness-aware recommendation algorithms mainly focus on scenarios where unfairness arises during the “warm-start” phase. However, there is a lack of research dedicated to addressing user unfairness during the “cold-start” phase. To tackle the above problems, in this paper, we propose a novel fairness-aware algorithm for CDR, called FairCDR. Specifically, FairCDR granularly models the individual influence of each sample on both fairness and accuracy to facilitate data reweighting, effectively striking a balance between fairness and recommendation utility.Extensive experiments are conducted to demonstrate the effectiveness of our proposed method. To benefit the research community, we have released our project at https://faircdr.github.io/FairCDR/
Title: The Connection Between Popularity Bias, Calibration, and Fairness in Recommendation
Authors: Himan Abdollahpouri; Masoud Mansoury; Robin Burke; Bamshad Mobasher
Affiliations: University of Colorado Boulder; Eindhoven University of Technology; University of Colorado Boulder; DePaul University
Year: 2020
Domain: Recommender Systems

Abstract: Recently there has been a growing interest in fairness-aware recommender systems including fairness in providing consistent performance across different users or groups of users. A recommender system could be considered unfair if the recommendations do not fairly represent the tastes of a certain group of users while other groups receive recommendations that are consistent with their preferences. In this paper, we use a metric called miscalibration for measuring how a recommendation algorithm is responsive to users’ true preferences and we consider how various algorithms may result in different degrees of miscalibration for different users. In particular, we conjecture that popularity bias which is a well-known phenomenon in recommendation is one important factor leading to miscalibration in recommendation. Our experimental results using two real-world datasets show that there is a connection between how different user groups are affected by algorithmic popularity bias and their level of interest in popular items. Moreover, we show that the more a group is affected by the algorithmic popularity bias, the more their recommendations are miscalibrated.
Title: Fairness metrics and bias mitigation strategies for rating predictions
Authors: Ashwathy Ashokan; Christian Haas
Affiliations: University of Nebraska at Omaha; Vienna University of Economics and Business
Year: 2021
Domain: Recommender Systems

Abstract: As online music platforms continue to grow, music recommender systems play a vital role in helping users navigate and discover content within their vast musical databases. At odds with this larger goal, is the presence of popularity bias, which causes algorithmic systems to favor mainstream content over, potentially more relevant, but niche items. In this work we explore the intrinsic relationship between music discovery and popularity bias through the lens of individual fairness. We propose a domain-aware, individual fairness-based approach which addresses popularity bias in graph neural network based recommender systems. Our approach uses individual fairness to reflect a ground truth listening experience, i.e., if two songs sound similar, this similarity should be reflected in their representations. In doing so, we facilitate meaningful music discovery that is resistant to popularity bias and grounded in the music domain. We apply our BOOST methodology to two discovery based tasks, performing recommendations at both the playlist level and user level. Then, we ground our evaluation in the cold start setting, showing that our approach outperforms existing fairness benchmarks in both performance and recommendation of lesser-known content. Finally, our analysis makes the case for the importance of domain-awareness when mitigating popularity bias in music recommendation.
Title: Advancing Algorithmic Justice: A Systematic Review of Fair Decision-Making Protocols in Healthcare AI
Authors: Fredrick Kayusi; Srinivas Kasulla et al.
Affiliations: Pwani University; University of Gujrat; Kabale University; University of Gujrat; Kabale University
Year: 2024
Domain: Health & Clinical AI

Abstract: This paper provides a comprehensive analysis of artificial intelligence (AI) implementation in Ghana’s healthcare system, synthesizing findings from recent Ghanaian studies. It examines legal and regulatory frameworks, ethical considerations, bias and fairness, transparency and accountability, interdisciplinary approaches, patient rights and data protection, healthcare access and equity, and implementation and oversight. The analysis reveals both significant opportunities for AI to improve healthcare delivery in Ghana and substantial challenges in ensuring its ethical, equitable, and effective implementation. Key findings highlight the need for AI-specific healthcare regulations, investment in local capacity building, culturally sensitive AI design, robust fairness and accountability mechanisms, and ongoing stakeholder engagement. This paper contributes to the discourse on AI in healthcare in developing countries, offering insights into the unique challenges faced by lower-middle-income countries. It aims to inform evidence-based policymaking and enrich academic understanding of AI implementation in diverse cultural and economic contexts.
Title: MACAIF: Machine Learning Auditing for Clinical AI Fairness
Authors: Pepita Barnard; John Robert Bautista; Joshua Krook; Anqi Liu; Héctor D. Menéndez; Aurora Schmidt; Tamim Sookoor
Affiliations: University of Nottingham; The University of Texas at Austin; University of Southampton; Johns Hopkins University; King's College London; Johns Hopkins University; Johns Hopkins University
Year: 2023
Domain: Health & Clinical AI

Abstract: Artificial intelligence in the form of machine learning algorithms is driving the latest industrial revolution, leading to disruptive changes in the ways we communicate, interact, design, collect information, and express ourselves. While these changes offer new possibilities for our societies, they may also introduce biases that can lead to unfair decisions. This issue is particularly critical in the context of medical diagnosis, as bias can jeopardize patient treatment and health. To mitigate these biases, it is essential to such biases and involve all relevant stakeholders in the design of fair machine learning algorithms. In this context, the MACAIF project aims to develop user-centred interfaces that allow stakeholders, including doctors, to challenge the fairness of machine learning algorithms based on demographics, such as gender or race. Our project proposes a methodology to engage with stakeholders and incorporate their concerns during the design of a dashboard based on MLighter - an adversarial tool which is applied to identify fairness-related issues in machine learning models.
Title: Auditing fairness in clinical AI systems using provenance-based simulation
Authors: Fidelis Fidelis Alu; Sunkanmi Oluwadare; Nnennaya Ngwanma Halliday; Uchenna Emmanuel Agwunobi
Affiliations: Information Technology University; Information Technology University; Information Technology University; The University of Texas Southwestern Medical Center
Year: 2026
Domain: Health & Clinical AI

Abstract: Introduction The adoption of Artificial Intelligence (AI) in clinical decision support has encountered obstacles due to algorithmic bias and lack of transparency. To address this, we developed an auditing framework using detailed data provenance to audit fairness. Methods We simulated audits on a synthetic patient dataset ( N = 1,000), comparing logistic regression and random forest models to detect gender bias using 5-fold crossvalidation and permutation testing. Results Logistic regression achieved 75.2 ± 1.0% accuracy (AUC = 0.806 ± 0.030) and random forest achieved 70.1 ± 1.4% accuracy (AUC = 0.745 ± 0.020). Provenance logs successfully detected gender biases in both models. Logistic regression exhibited statistically significant bias (EOD = +0.256, p = 0.0080), while random forest’s smaller disparity (EOD = +0.055, p = 0.5664) was not statistically significant, demonstrating that our audit distinguishes systematic discrimination from random variation. Sensitivity analysis confirmed successful bias detection across magnitudes from β = −0.10 to β =−0.80. Discussion Despite lower accuracy, random forest showed 57% less bias than logistic regression, challenging assumptions that interpretability guarantees fairness. We introduce a standardized AI Fairness Provenance Record documenting data origin, model choices, and bias metrics, enabling auditors to trace decisions to their source. This framework maps to FDA transparency guidelines and ONC HTI-1 requirements, demonstrating how provenance-based auditing supports regulatory compliance and provides a pathway toward more responsible and equitable AI in clinical settings.
Title: A Review of Bias and Fairness in Artificial Intelligence
Authors: Rubén González-Sendino; Emilio Serrano; Javier Bajo; Paulo Nováis
Affiliations: Universidad Politécnica de Madrid; Universidad Politécnica de Madrid; Universidad Politécnica de Madrid; University of Minho
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: Automating decision systems has led to hidden biases in the use of artificial intelligence (AI). Consequently, explaining these decisions and identifying responsibilities has become a challenge. As a result, a new field of research on algorithmic fairness has emerged. In this area, detecting biases and mitigating them is essential to ensure fair and discrimination-free decisions. This paper contributes with: (1) a categorization of biases and how these are associated with different phases of an AI model’s development (including the data-generation phase); (2) a revision of fairness metrics to audit the data and AI models trained with them (considering agnostic models when focusing on fairness); and, (3) a novel taxonomy of the procedures to mitigate biases in the different phases of an AI model’s development (pre-processing, training, and post-processing) with the addition of transversal actions that help to produce fairer models.
Title: Mitigating Algorithmic Bias in AI-Driven Cardiovascular Imaging for Fairer Diagnostics
Authors: Md Abu Sufian; Lujain Alsadder; Wahiba Hamzi; Sadia Zaman; A. S. M. Sharifuzzaman Sagar; Boumediene Hamzi
Affiliations: University of Leicester; Queen Mary University of London; University of Blida; Queen Mary University of London; Sejong University; California Institute of Technology
Year: 2024
Domain: Health & Clinical AI

Abstract: Background/Objectives: The research addresses algorithmic bias in deep learning models for cardiovascular risk prediction, focusing on fairness across demographic and socioeconomic groups to mitigate health disparities. It integrates fairness-aware algorithms, susceptible carrier-infected-recovered (SCIR) models, and interpretability frameworks to combine fairness with actionable AI insights supported by robust segmentation and classification metrics. Methods: The research utilised quantitative 3D/4D heart magnetic resonance imaging and tabular datasets from the Cardiac Atlas Project’s (CAP) open challenges to explore AI-driven methodologies for mitigating algorithmic bias in cardiac imaging. The SCIR model, known for its robustness, was adapted with the Capuchin algorithm, adversarial debiasing, Fairlearn, and post-processing with equalised odds. The robustness of the SCIR model was further demonstrated in the fairness evaluation metrics, which included demographic parity, equal opportunity difference (0.037), equalised odds difference (0.026), disparate impact (1.081), and Theil Index (0.249). For interpretability, YOLOv5, Mask R-CNN, and ResNet18 were implemented with LIME and SHAP. Bias mitigation improved disparate impact (0.80 to 0.95), reduced equal opportunity difference (0.20 to 0.05), and decreased false favourable rates for males (0.0059 to 0.0033) and females (0.0096 to 0.0064) through balanced probability adjustment. Results: The SCIR model outperformed the SIR model (recovery rate: 1.38 vs 0.83) with a −10% transmission bias impact. Parameters (β=0.5, δ=0.2, γ=0.15) reduced susceptible counts to 2.53×10−12 and increased recovered counts to 9.98 by t=50. YOLOv5 achieved high Intersection over Union (IoU) scores (94.8%, 93.7%, 80.6% for normal, severe, and abnormal cases). Mask R-CNN showed 82.5% peak confidence, while ResNet demonstrated a 10.4% accuracy drop under noise. Performance metrics (IoU: 0.91–0.96, Dice: 0.941–0.980, Kappa: 0.95) highlighted strong predictive accuracy and reliability. Conclusions: The findings validate the effectiveness of fairness-aware algorithms in addressing cardiovascular predictive model biases. The integration of fairness and explainable AI not only promotes equitable diagnostic precision but also significantly reduces diagnostic disparities across vulnerable populations. This reduction in disparities is a key outcome of the research, enhancing clinical trust in AI-driven systems. The promising results of this study pave the way for future work that will explore scalability in real-world clinical settings and address limitations such as computational complexity in large-scale data processing.
Title: Data augmentation for fairness-aware machine learning
Authors: Ioannis Pastaltzidis; Nikolaos Dimitriou; Katherine Quezada Tavárez; Stergios Aidinlis; Thomas Marquenie; Agata Gurzawska; Dimitrios Tzovaras
Affiliations: Information Technologies Institute; Centre for Research and Technology Hellas; KU Leuven; Trilateral Research & Consulting; KU Leuven; Trilateral Research & Consulting; Information Technologies Institute
Year: 2022
Domain: General Fairness & Bias Mitigation

Abstract: Researchers and practitioners in the fairness community have highlighted the ethical and legal challenges of using biased datasets in data-driven systems, with algorithmic bias being a major concern. Despite the rapidly growing body of literature on fairness in algorithmic decision-making, there remains a paucity of fairness scholarship on machine learning algorithms for the real-time detection of crime. This contribution presents an approach for fairness-aware machine learning to mitigate the algorithmic bias / discrimination issues posed by the reliance on biased data when building law enforcement technology. Our analysis is based on RWF-2000, which has served as the basis for violent activity recognition tasks in data-driven law enforcement projects. We reveal issues of overrepresentation of minority subjects in violence situations that limit the external validity of the dataset for real-time crime detection systems and propose data augmentation techniques to rebalance the dataset. The experiments on real world data show the potential to create more balanced datasets by synthetically generated samples, thus mitigating bias and discrimination concerns in law enforcement applications.