cair-nepal/ai-bias-research-landscape huggingface dataset:
Title: HIFI: Explaining and Mitigating Algorithmic Bias Through the Lens of Game-Theoretic Interactions
Authors: Lingfeng Zhang; Zhaohui Wang; Yueling Zhang; Min Zhang; Jiangtao Wang
Affiliations: East China Normal University; East China Normal University; East China Normal University; East China Normal University; East China Normal University
Year: 2025
Domain: General Fairness & Bias Mitigation

Abstract: Machine Learning (ML) algorithms are increasingly used in decision-making process across various social-critical domains, but they often somewhat inherit and amplify bias from their training data, leading to unfair and unethical outcomes. This issue highlights the urgent need for effective methods to detect, explain, and mitigate bias to ensure the fairness of ML systems. Previous studies are prone to analyze the root causes of algorithmic bias from a statistical perspective. However, to the best of our knowledge, none of them has discussed how sensitive information inducing the final discriminatory decision is encoded by ML models. In this work, we attempt to explain and mitigate algorithmic bias from a game-theoretic view. We mathematically decode an essential and common component of sensitive information implicitly defined by various fairness metrics with Harsanyi interactions, and on this basis, we propose an in-processing method HIFI for bias mitigation. We conduct an extensive evaluation of HIFI with 11 state-of-the-art methods, 5 real-world datasets, 4 fairness criteria, and 5 ML performance metrics, while also considering intersectional fairness for multiple protected attributes. The results show that HIFI surpasses state-of-the-art in-processing methods in terms of fairness improvement and fairness-performance trade-off, and also achieves notable effectiveness in reducing violations of individual fairness simultaneously.
Title: Pre-processing Techniques to Mitigate Against Algorithmic Bias
Authors: Maliheh Heidarpour Shahrezaei; Róisín Loughran; Kevin Mc Daid
Affiliations: Dundalk Institute of Technology; Dundalk Institute of Technology; Dundalk Institute of Technology
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: A significant portion of current AI research is focused on ensuring that model decisions are fair and free of bias. Such research should consider not merely the algorithm but also the datasets, metrics and approaches used. In this paper, we work on several pre-processing techniques to achieve fair results for classification tasks by assigning weights, sampling and changing class labels. We used two well-known classifiers, Logistic Regression and Decision Tree, performing experiments on a popular data set in the fairness domain. This research aims to compare the effects of different pre-processing techniques on the resulting confusion matrix elements and the derived fairness metrics. We found that the Massaging technique with the Logistic regression classifier resulted in the Disparate Impact value that was closest to one. While, for the Decision Tree classifier, Reweighting and Uniform Sampling performed better than Massaging for all of our fairness metrics and both sensitive attributes.
Title: D-BIAS: A Causality-Based Human-in-the-Loop System for Tackling Algorithmic Bias
Authors: Bhavya Ghai; Klaus Mueller
Affiliations: Stony Brook University; Stony Brook University
Year: 2022
Domain: General Fairness & Bias Mitigation

Abstract: With the rise of AI, algorithms have become better at learning underlying patterns from the training data including ingrained social biases based on gender, race, etc. Deployment of such algorithms to domains such as hiring, healthcare, law enforcement, etc. has raised serious concerns about fairness, accountability, trust and interpretability in machine learning algorithms. To alleviate this problem, we propose D-BIAS, a visual interactive tool that embodies human-in-the-loop AI approach for auditing and mitigating social biases from tabular datasets. It uses a graphical causal model to represent causal relationships among different features in the dataset and as a medium to inject domain knowledge. A user can detect the presence of bias against a group, say females, or a subgroup, say black females, by identifying unfair causal relationships in the causal network and using an array of fairness metrics. Thereafter, the user can mitigate bias by refining the causal model and acting on the unfair causal edges. For each interaction, say weakening/deleting a biased causal edge, the system uses a novel method to simulate a new (debiased) dataset based on the current causal model while ensuring a minimal change from the original dataset. Users can visually assess the impact of their interactions on different fairness metrics, utility metrics, data distortion, and the underlying data distribution. Once satisfied, they can download the debiased dataset and use it for any downstream application for fairer predictions. We evaluate D-BIAS by conducting experiments on 3 datasets and also a formal user study. We found that D-BIAS helps reduce bias significantly compared to the baseline debiasing approach across different fairness metrics while incurring little data distortion and a small loss in utility. Moreover, our human-in-the-loop based approach significantly outperforms an automated approach on trust, interpretability and accountability.
Title: Detection and Mitigation of Algorithmic Bias via Predictive Parity
Authors: Cyrus DiCiccio; Brian Hsu; YinYin Yu; Preetam Nandy; Kinjal Basu
Affiliations: LinkedIn (United States); LinkedIn (United States); LinkedIn (United States); LinkedIn (United States); LinkedIn (United States)
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: Predictive parity (PP), also known as sufficiency, is a core definition of algorithmic fairness essentially stating that model outputs must have the same interpretation of expected outcomes regardless of group. Testing and satisfying PP is especially important in many settings where model scores are interpreted by humans or directly provide access to opportunity, such as healthcare or banking. Solutions for PP violations have primarily been studied through the lens of model calibration. However, we find that existing calibration-based tests and mitigation methods are designed for independent data, which is often not assumable in large-scale applications such as social media or medical testing. In this work, we address this issue by developing a statistically rigorous non-parametric regression based test for PP with dependent observations. We then apply our test to illustrate that PP testing can significantly vary under the two assumptions. Lastly, we provide a mitigation solution to provide a minimally-biased post-processing transformation function to achieve PP.
Title: Mitigating Algorithmic Bias with Limited Annotations
Authors: Guanchu Wang; Mengnan Du; Ninghao Liu; Na Zou; Xia Hu
Affiliations: Rice University; New Jersey Institute of Technology; University of Georgia; Texas A&M University; Rice University
Year: 2022
Domain: General Fairness & Bias Mitigation

Abstract: Existing work on fairness modeling commonly assumes that sensitive attributes for all instances are fully available, which may not be true in many real-world applications due to the high cost of acquiring sensitive information. When sensitive attributes are not disclosed or available, it is needed to manually annotate a small part of the training data to mitigate bias. However, the skewed distribution across different sensitive groups preserves the skewness of the original dataset in the annotated subset, which leads to non-optimal bias mitigation. To tackle this challenge, we propose Active Penalization Of Discrimination (APOD), an interactive framework to guide the limited annotations towards maximally eliminating the effect of algorithmic bias. The proposed APOD integrates discrimination penalization with active instance selection to efficiently utilize the limited annotation budget, and it is theoretically proved to be capable of bounding the algorithmic bias. According to the evaluation on five benchmark datasets, APOD outperforms the state-of-the-arts baseline methods under the limited annotation budget, and shows comparable performance to fully annotated bias mitigation, which demonstrates that APOD could benefit real-world applications when sensitive information is limited.
Title: A step toward building a unified framework for managing AI bias
Authors: Saadia Afzal Rana; Zati Hakim Azizul; Ali Afzal Awan
Affiliations: University of Malaya; University of Malaya; University of Malaya
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: Integrating artificial intelligence (AI) has transformed living standards. However, AI's efforts are being thwarted by concerns about the rise of biases and unfairness. The problem advocates strongly for a strategy for tackling potential biases. This article thoroughly evaluates existing knowledge to enhance fairness management, which will serve as a foundation for creating a unified framework to address any bias and its subsequent mitigation method throughout the AI development pipeline. We map the software development life cycle (SDLC), machine learning life cycle (MLLC) and cross industry standard process for data mining (CRISP-DM) together to have a general understanding of how phases in these development processes are related to each other. The map should benefit researchers from multiple technical backgrounds. Biases are categorised into three distinct classes; pre-existing, technical and emergent bias, and subsequently, three mitigation strategies; conceptual, empirical and technical, along with fairness management approaches; fairness sampling, learning and certification. The recommended practices for debias and overcoming challenges encountered further set directions for successfully establishing a unified framework.
Title: AI and Algorithmic Bias: Source, Detection, Mitigation and Implications
Authors: Runshan Fu; Yan Huang; Param Vir Singh
Affiliations: Carnegie Mellon University; Carnegie Mellon University; Carnegie Mellon University
Year: 2020
Domain: General Fairness & Bias Mitigation

Abstract: In recent years, popularity bias has consistently accompanied the development of recommender systems, significantly impacting their fairness. Currently, numerous studies have been conducted to explore this topic, such as Causal Inference and Contrastive Learning. However, while these methods have alleviated popularity bias to some extent, most of them have not fully exploited the rich information embedded in item popularity within user sequences. To better utilize this type of information, we propose a novel method—the Item Popularity Attention Recommender System (IPARec). By introducing item popularity, IPARec can further mitigate popularity bias. This method incorporates item popularity embeddings and an attention mechanism, allowing the model to effectively capture user interests while considering the impact of item popularity on recommendations, thus addressing the popularity bias issue more comprehensively. The proposed IPARec effectively models the interplay of item popularity within user sequences and achieves outstanding performance across various datasets. IPARec significantly improves the recommendation of unpopular items, showcasing its unique advantage in alleviating popularity bias. With these advantages, IPARec not only improves the diversity and fairness of recommendations but also offers an innovative solution to address popularity bias in recommender systems.
Title: Algorithmic Fairness in Clinical Predictive Models: A Review of Bias Audits and Mitigation Strategies in Epidemiological Research
Authors: Redha Jaffar Albaqshi; Hassan Ali Ahmad Najaei; Hani Ghazi Abdulmalik; Salman Saeed Saad Alzamaa; Saeed Oudah Hammad Alshahrani; Mohammad Mosad M. Alshaarari; Abdullah Saleh Essa Aldurayhim; Ahmad Mohammad AlGhazal; Khaled Ibrahim Sohail; Ibtihal Abdullah Alotibi; Ali Essa Dallak; Ahmad Ali Ahmad Sahli; Mohammed Saleh Najdi
Affiliations: King Fahad Hospital Hufūf; Ministry of Health; Maternity and Children's Hospital; Ministry of Health; Ministry of Health; Ministry of Health; Health First; Ministero della Salute; Ministerio de Salud; Imam Abdulrahman Al Faisal Hospital; Babylon Health; Prince Mohammed bin Abdulaziz Hospital; Ministry of Health
Year: 2025
Domain: Health & Clinical AI

Abstract: Background: Digital Epidemiology has emerged as a transformative approach to infectious disease surveillance, leveraging digital data streams such as social media, search queries, and mobility patterns. While these methods offer speed and scale, they introduce significant statistical and ethical challenges, particularly bias and fairness concerns in predictive modeling. Aim: This review aims to examine algorithmic fairness in clinical predictive models within epidemiological research, focusing on bias audits and mitigation strategies in the context of Digital Epidemiology. Methods: A comprehensive literature review was conducted, analyzing methodological differences between classical and digital approaches, sources of bias, and corrective strategies. Key themes include representativeness, measurement error, and algorithmic bias in machine learning models trained on digital data. Results: Findings reveal that Digital Epidemiology offers real-time, large-scale data collection but suffers from structural biases due to self-selection, platform design, and digital divides. Bias mitigation is often retrospective, relying on weighting, normalization, and cross-validation. Ethical concerns such as privacy and informed consent intersect with fairness, as predictive models risk amplifying inequities. Integration of classical rigor with digital flexibility and continuous bias audits is essential for equitable outcomes. Conclusion: Digital Epidemiology complements classical methods but requires robust frameworks for bias detection, ethical governance, and algorithmic transparency. Sustained collaboration, standardization, and inclusive data practices are critical to ensure predictive models support fair and actionable public health decisions.
Title: Add-Remove-or-Relabel: Practitioner-Friendly Bias Mitigation via Influential Fairness
Authors: Brianna Richardson; Prasanna Sattigeri; Dennis Wei; Karthikeyan Natesan Ramamurthy; Kush R. Varshney; Amit Dhurandhar; Juan E. Gilbert
Affiliations: University of Florida; IBM (United States); IBM (United States); IBM (United States); IBM (United States); IBM (United States); University of Florida
Year: 2023
Domain: General Fairness & Bias Mitigation

Abstract: Commensurate with the rise in algorithmic bias research, myriad algorithmic bias mitigation strategies have been proposed in the literature. Nonetheless, many voice concerns about the lack of transparency that accompanies mitigation methods and the paucity of mitigation methods that satisfy protocol and data limitations of practitioners. Influence functions from robust statistics provide a novel opportunity to overcome both issues. Previous work demonstrates the power of influence functions to improve fairness outcomes. This work proposes a novel family of fairness solutions, coined influential fairness (IF), that is human-understandable and also agnostic to the underlying machine learning model and choice of fairness metric. We conduct an investigation of practitioner profiles and design mitigation methods for practitioners whose limitations discourage them from utilizing existing bias mitigation methods.
Title: Fairness And Bias Mitigation in AI Models for Diabetes Diagnosis: A Comparative Evaluation of Algorithmic Approaches
Authors: Muhammad K.A. Hamdan; Mohamad Faizal Ab Jabal; Shuzlina Abdul-Rahman; Azyan Yusra Kapi
Affiliations: University of Technology Malaysia; University of Technology Malaysia; Universiti Teknologi MARA System; University of Technology Malaysia
Year: 2025
Domain: Health & Clinical AI

Abstract: Bias in AI-driven diagnostic models has raised serious concerns regarding fairness in healthcare delivery, particularly for chronic diseases like diabetes. This study investigates algorithmic bias in diabetes prediction models and evaluates the effectiveness of three fairness-aware approaches: Fairness-Aware Interpretable Modelling (FAIM), Fairness-Aware Machine Learning (FAML), and Fairness-Aware Oversampling (FAWOS). The same dataset and experimental setup were used to ensure a fair comparison across models. FAIM employs interpretable decision trees to enhance transparency but lacks explicit fairness mechanisms. FAML incorporates adversarial fairness constraints, achieving perfect fairness metrics while maintaining acceptable accuracy. FAWOS addresses class imbalance using SMOTE, improving overall classification accuracy without enforcing fairness. Results show that while each method has strengths, none independently achieves an optimal balance of accuracy, fairness, and interpretability. Therefore, this paper proposes a hybrid approach that integrates multiple bias mitigation strategies to support fairer and more reliable AI applications in clinical settings. This study contributes a structured comparative evaluation framework and offers actionable insights for the development of ethical AI models in healthcare diagnostics.
Title: BIAS AUDIT FRAMEWORKS: Developing Tools for Early Detection of Algorithmic Bias in AI Development
Authors: Richard O. Oveh; N. D. Isitor
Affiliations: University of Delta Agbor; Delta State University
Year: 2025
Domain: General Fairness & Bias Mitigation

Abstract: Algorithmic bias in artificial intelligence (AI) systems continues to pose significant ethical and societal challenges, especially in critical domains such as healthcare, education, and finance. Current approaches to bias mitigation often fail to provide a holistic, proactive solution that integrates fairness, accountability, and transparency into the AI development lifecycle. This study introduces a Bias Audit Framework designed to detect and mitigate algorithmic bias during the early stages of AI development. The framework comprises four core components: Data Bias Assessment, Model Bias Evaluation, Developer Awareness and Training, and Continuous Monitoring and Feedback. A healthcare dataset was used as a case study to evaluate the framework's efficacy. Initially, the logistic regression model trained on the imbalanced dataset achieved high overall performance with Accuracy: 85%, Precision: 0.89, and Recall: 0.83, but exhibited fairness issues. Disparate Impact Ratio (DIR) was 0.67, and Equal Opportunity Difference (EOD) was 0.13, reflecting gender bias. After applying the Bias Audit Framework,—including oversampling, data augmentation, and threshold optimization—the model was retrained. Its performance remained robust (Accuracy: ~84–85%, Precision: ~0.88, Recall: ~0.88), while fairness significantly improved: Female recall increased to 0.88, reducing EOD to ~0, and DIR improved to 0.85–0.95, indicating a more balanced and equitable model. By equipping developers with practical tools and emphasizing interdisciplinary collaboration, the framework ensures a systematic and ethical approach to addressing algorithmic bias. These findings underscore the importance of embedding bias mitigation practices into all stages of AI development to foster equitable and trustworthy AI systems.
Title: Justice in Judgment: Unveiling (Hidden) Bias in LLM-assisted Peer Reviews
Authors: Sai Suresh Macharla Vasu; Ivaxi Sheth; Hui‐Po Wang; Ruta Binkyte; Mario Fritz
Affiliations: Saarland University
Year: 2025
Domain: LLMs & NLP

Abstract: The adoption of large language models (LLMs) is transforming the peer review process, from assisting reviewers in writing detailed evaluations to generating entire reviews automatically. While these capabilities offer new opportunities, they also raise concerns about fairness and reliability. In this paper, we investigate bias in LLM-generated peer reviews through controlled interventions on author metadata, including affiliation, gender, seniority, and publication history. Our analysis consistently shows a strong affiliation bias favoring authors from highly ranked institutions. We also identify directional preferences associated with seniority and prior publication record, which can influence acceptance decisions for borderline papers. Gender effects are smaller but present in several models. Notably, implicit biases become more pronounced when examining token-level soft ratings, suggesting that alignment may mask but not fully eliminate underlying preferences
Title: Does RAG Introduce Unfairness in LLMs? Evaluating Fairness in Retrieval-Augmented Generation Systems
Authors: Xueyan Wu; S.C. Li; Huaxing Wu; Zhiqiang Tao; Yi Fang
Affiliations: Santa Clara University; 
Santa Clara University;
DOCOMO Innovations, Inc;
Rochester Institute of Technology;
Santa Clara University

Year: 2024
Domain: LLMs & NLP

Abstract: Retrieval-Augmented Generation (RAG) has recently gained significant attention for its enhanced ability to integrate external knowledge sources into open-domain question answering (QA) tasks. However, it remains unclear how these models address fairness concerns, particularly with respect to sensitive attributes such as gender, geographic location, and other demographic factors. First, as language models evolve to prioritize utility, like improving exact match accuracy, fairness considerations may have been largely overlooked. Second, the complex, multi-component architecture of RAG methods poses challenges in identifying and mitigating biases, as each component is optimized for distinct objectives. In this paper, we aim to empirically evaluate fairness in several RAG methods. We propose a fairness evaluation framework tailored to RAG, using scenario-based questions and analyzing disparities across demographic attributes. Our experimental results indicate that, despite recent advances in utility-driven optimization, fairness issues persist in both the retrieval and generation stages. These findings underscore the need for targeted interventions to address fairness concerns throughout the RAG pipeline. The dataset and code used in this study are publicly available at this GitHub Repository https://github.com/elviswxy/RAG_fairness .
Title: Ethical Imperatives for Retrieval-Augmented Generation in Clinical Nursing: Viewpoint on Responsible AI Use
Authors: Xinyi Tu; Cynthia X. Shi; Peilin Qian; Lizhu Wang
Affiliations: Second Affiliated Hospital of Zhejiang University; Second Affiliated Hospital of Zhejiang University; Chinese Academy of Medical Sciences & Peking Union Medical College; Second Affiliated Hospital of Zhejiang University
Year: 2025
Domain: Health & Clinical AI

Abstract: Unlabelled: Retrieval-augmented generation (RAG) systems have emerged as a powerful technique to enhance the capabilities of large language models by enabling them to access external, up-to-date knowledge in real time, and RAG systems are being increasingly adopted by researchers in the medical field. In this viewpoint article, we explore the ethical imperatives for implementing RAG systems in clinical nursing environments, with particular attention to how these technologies affect patient care quality and safety. The purpose of this paper is to examine the ethical risks introduced by RAG-enhanced large language models in clinical nursing and to propose strategic guidelines for their responsible implementation. Key considerations include ensuring accuracy, fairness, transparency, and accountability, as well as maintaining essential human oversight, as discussed through a structured analysis. We argue that robust data governance, explainable artificial intelligence (AI) techniques, and continuous monitoring are critical components of a responsible RAG implementation strategy. Ultimately, realizing the benefits of RAG while mitigating ethical concerns requires sustained collaboration among health care professionals, AI developers, and policymakers, fostering a future where AI supports patient safety, reduces disparities, and improves the quality of nursing care.