cair-nepal/ai-bias-research-landscape huggingface dataset:
Title: Personalized Recommender System for Improving Gender-fairness in Teaching
Authors: Andreas Arens-Volland; Patrick Gratz; Alexandre Baudet; Louis Deladiennée; Marie Gallais; Yannick Naudet
Affiliations: Luxembourg Institute of Science and Technology; Luxembourg Institute of Science and Technology; Luxembourg Institute of Science and Technology; Luxembourg Institute of Science and Technology; Luxembourg Institute of Science and Technology; Luxembourg Institute of Science and Technology
Year: 2019
Domain: Recommender Systems

Abstract: This paper presents the concept of a personalized recommender system (RS) supporting the goal of improving gender-fairness in teaching. The RS is part of the Gender4STEM Teaching Assistant (GTA) that has been launched on 11 February 2019 as an essential part of the Gender4STEM project. A foreseen 12 months testing phase with more than 400 participants taking place in several European countries will provide valuable evaluation measures.
Title: Fairness and Bias in E-commerce Recommendation Systems: A Literature Review
Authors: Amelia Morencia Irena; Aretha Natalova Wahyudi; Irma Kartika Wairooy; Brilly Andro Makalew
Affiliations: Binus University; Binus University; Binus University; Binus University
Year: 2024
Domain: Recommender Systems

Abstract: The paper presents one of the most important themes in the fair treatment and biases of e-commerce recommendation algorithms. Algorithms recommending products to users have a dramatic impact on their shopping experience; that is, online shopping has rapidly grown in popularity. These systems, however, have a popularity bias, which skews user exposure to popular items, thus pulling down, in some manners, the diversity of recommendations. This article follows a systematic approach and uses the PICOC framework to organize the study while highlighting major categories of bias such as popularity bias, position bias, and exposure bias. The paper also analyzes the current approaches in the literature concerning bias mitigation. The attempt is to add to the existing literature around a greater understanding of bias identification approaches and mitigation in the recommendation system. The paper underlined the urgency with which real-world e-commerce platforms have to begin providing more equitable user recommendations. Some future work must, therefore, be conducted on developing strategies that impact fairness without hurting the effectiveness of recommendation systems, leading toward a digitally inclusive marketplace.
Title: De-Biasing user conformity bias and item popularity bias in Group recommendation
Authors: Junjie Jia; Tianyue Shang; LiFang Li; Si Chen
Affiliations: Northwest Normal University; Northwest Normal University; Northwest Normal University; Northwest Normal University
Year: 2023
Domain: Recommender Systems

Abstract: The recommendation system is usually faced with the problem of popularity bias, and the collected interactive data often presents quite unbalanced or even long-tail distribution.However, most of the existing studies have considered how to eliminate the negative effects brought by the popularity bias, thus ignoring the two sides of the popularity bias.At the item level, people tend to comment on popular items. So is it all positive? Most research on popularity bias has focused on the conformity behavior and has increased the diversity of personal recommendation lists. At the user level, does everyone want to pursue the preferences of the mainstream group? The reported user responses are skewed toward popular items rather than users’ true interest. We believe that not all popularity bias are harmful. In addition to the consistency effect, the uneven distribution of items can be attributed to the diversity of items’ quality. To address this problem, we propose a new de-biasing mechanism, which uses user conformity bias and item popularity bias to improve the accuracy of recommendation.
Title: DF-CDR: A Unified Framework for Dual Fairness in Cross-Domain Recommendation Systems
Authors: Ziyu Li; Jinglong Duan; Shiqing Wu; Weihua Li; Quan Bai; Jianhua Jiang
Affiliations: Auckland University of Technology; Auckland University of Technology; City University of Macau; Auckland University of Technology; University of Tasmania; Jilin University of Finance and Economics
Year: 2025
Domain: Recommender Systems

Abstract: Cross-domain recommendation (CDR) is a common approach for solving cold-start problems in target domains. However, the fairness issue remains underexplored in CDR. In sparse and distribution-shifted settings, prediction errors and exposure often show systematic disparities across sensitive user groups and between short-head and long-tail items, disadvantaging new users and reinforcing feedback loops. This paper introduces DF-CDR, a two-stage framework that jointly optimizes accuracy and dual fairness on both the user and the item sides. Specifically, a residual GNN enriches non-overlapping users by diffusing knowledge from overlapping counterparts, while a dual autoencoder aligns user representations across domains. To enhance both accuracy and fairness, an adaptive weighting scheme based on the influence function is deployed during training. Additionally, a re-ranking module is employed to balance accuracy with fairness objectives at both the user level and the item level. Experiments and ablations demonstrate that representation enhancement, adaptive reweighting, and fairness-aware re-ranking are complementary, together producing recommendations that alleviate both user-and item-side inequities without compromising accuracy.
Title: Enhancing Netflix's Recommendation System: A Hybrid AI Approach for Personalization, Bias Reduction, and Engagement Optimization
Authors: Aswathy K Cherian; Harish S; Soram Patel; S. Patil
Affiliations: SRM Institute of Science and Technology; SRM Institute of Science and Technology; SRM Institute of Science and Technology; SRM Institute of Science and Technology
Year: 2025
Domain: Recommender Systems

Abstract: Recommender systems play a very important role in improving user engagement on video streaming platforms like Netflix. Collaborative filtering-based classic models and deep learning encounter issues like cold start issues, sparsity of data, and algorithmic bias. This paper suggests a hybrid recommendation system that integrates collaborative filtering, content-based filtering, and deep learning to enhance recommendation accuracy. To overcome cold start problems, platform-external metadata from external sources such as IMDb and Rotten Tomatoes is combined using transfer learning. A multiuser identification scheme supports personalized recommendations for shared accounts. Sentiment analysis of social media posts and user reviews dynamically updates recommendations. In addition, fairness-aware algorithms are added to minimize bias and provide diverse and inclusive content recommendations. The effectiveness of the system is measured through A/B testing and revenue modeling, demonstrating substantial improvements in recommendation accuracy, user satisfaction, and long-term engagement.
Title: Properly Scoring Users' Mainstreamness to Evaluate Recommendation Bias
Authors: Y. Hara; Masahiro Sato; Hajime Nobuhara
Affiliations: University of Tsukuba; University of Tsukuba
Year: 2023
Domain: Recommender Systems

Abstract: In recommender systems, a mainstream bias (MS bias) exists, which degrades the fairness of recommender systems and may decrease user satisfaction. In other words, the recommendation quality for users who prefer mainstream items is higher than that for users who prefer niche items. The MS bias degrades fairness in the recommender system and might decrease user satisfaction. To address the MS bias, we first need to quantify the degree of bias. However, the mainstreamness of users (the MS score) is difficult to measure properly, and we reveal that previous MS scores are greatly confounded by the number of user interactions that are irrelevant to mainstreamness. To address this issue, we propose a novel MS score that is unaffected by the number of interactions. Specifically, we introduce a new similarity metric that differs from the conventional Jaccard similarity measure to eliminate the effect of interaction numbers. We confirm the validity of our proposed MS score through a simulation study that discriminates mainstream users from niche users under varying numbers of user interactions. The proposed unconfounded MS score enables the proper evaluation of MS biases and the selection of a recommendation model with less MS bias. We evaluate the MS biases of two popular recommendation models using seven real-world datasets. We demonstrate that MS biases exist in all datasets and that a better model differs depending on the datasets. Furthermore, we confirm that the previous MS scores fail to select recommendation models with small biases. The results show the effectiveness of our new MS score in improving recommendations for fairness among users.
Title: A Bias Aware News Recommendation System
Authors: Anish Anil Patankar; Joy Bose; Harshit Khanna
Affiliations: Samsung (India); Samsung (India); Samsung (India)
Year: 2019
Domain: Recommender Systems

Abstract: In this era of fake news and political polarization, it is desirable to have a system to enable users to access balanced news content. Current solutions focus on top down, server based approaches to decide whether a news article is fake or biased, and display only trusted news to the end users. In this paper, we follow a different approach to help the users make informed choices about which news they want to read, making users aware in real time of the bias in news articles they were browsing and recommending news articles from other sources on the same topic with different levels of bias. We use a recent Pew research report to collect news sources that readers with varying political inclinations prefer to read. We then scrape news articles on a variety of topics from these varied news sources. After this, we perform clustering to find similar topics of the articles, as well as calculate a bias score for each article. For a news article the user is currently reading, we display the bias score and also display other articles on the same topic, out of the previously collected articles, from different news sources. This we present to the user. This approach, we hope, would make it possible for users to access more balanced articles on given news topics. We present the implementation details of the system along with some preliminary results on news articles.
Title: Cross-Modality Bias Auditing and Fairness Evaluation in Generalist Clinical AI Models
Authors: Christopher G. Harris
Affiliations: Department of Mathematical Sciences
Year: 2025
Domain: Health & Clinical AI

Abstract: We introduce a cross-modality fairness audit for generalist clinical AI systems that integrate structured and unstructured EHR data. The framework combines SHAP, LIME, and counterfactual simulation to expose modality-specific bias and evaluate mitigation strategies. Experiments using MIMIC-IV show that a multimodal transformer achieves AUC 0.88 and F1 0.81, while fairness interventions (adversarial debiasing and subgroup reweighting) reduce Demographic Parity Difference (DPD) from 0.13 to 0.04 and Equalized Odds Difference (EOD) from 0.18 to 0.06 with minimal performance loss. We contribute color, single-column TikZ figures that make the audit interpretable for clinical collaborators, and we discuss deployment implications.
Title: A Survey of Bias and Fairness in Healthcare AI
Authors: Ibomoiye Domor Mienye; George Obaido; Ikiomoye Douglas Emmanuel; Ayodeji Akeem Ajani
Affiliations: Walter Sisulu University; University of California, Berkeley; University of Salford
Year: 2024
Domain: Health & Clinical AI

Abstract: The integration of artificial intelligence (AI) tech-nologies in healthcare has shown great promise in improving patient outcomes. However, there have been concerns about bias and ethical implications. This review explores the key issues of bias, fairness, and ethical considerations in health care AI. The review provides a foundation for future research and policy development in the field of health care AI. It will be beneficial to researchers, policymakers, healthcare providers, and technology developers who are involved in the development of AI systems in healthcare.
Title: The Problem Of Data Bias In Healthcare AI
Authors: Ashutosh S Vajpayee; Deepak S. Khobragade
Affiliations: Datta Meghe Institute of Medical Sciences; Datta Meghe Institute of Medical Sciences
Year: 2024
Domain: Health & Clinical AI

Abstract: Bias in data collection for machine learning entails complexity in the fairness of the forecast and the efficiency of the model. This occurs especially when data is biased, gathered from unrepresentative samples, or experiences systematic errors in data gathering; the consequences of such data include unfair policies and practices that perpetuate inequity in society. This abstract focuses on how biases affect machine learning algorithms and how the biases work to arise and spread. It describes methods of how to avoid the bias problem in data, how to build sets that are as diverse as possible and properly preprocess the data, and such things as algorithmic fairness metrics. Tackling biases in data is a close-to- impossible task that can help ensure fairness in equal and right machine learning.
Title: Gradient-Based Reconciliation of Fairness and Performance in Healthcare AI
Authors: Xiaoyan Wang
Affiliations: Drexel University
Year: 2025
Domain: Health & Clinical AI

Abstract: The enrichment of healthcare data coupled with advancements in computational capabilities has driven the adoption of artificial intelligence in healthcare. However, these approaches, when implemented without fairness considerations, may exacerbate existing disparities, leading to inequitable resource allocation and diagnostic inaccuracies across different demographic groups. This study introduces a novel approach via gradient projection to multi-attribute fairness optimization in healthcare AI, optimizing fairness across multiple demographic attributes and predictive performance concurrently. The approach aligns conflicting optimization objectives by projecting each gradient onto the normal plane of the other. Our method also enhances interpretability by elucidating the adjustments made during optimization, providing insights into trade-offs between fairness and accuracy.
Title: Integrated Framework for Equitable Healthcare AI: Bias Mitigation, Community Participation, and Regulatory Governance
Authors: Joydeep Chandra; Ramanjot Kaur; Rashi Sahay
Affiliations: Chandigarh University; Chandigarh University; Manav Rachna International Institute of Research and Studies
Year: 2025
Domain: Health & Clinical AI

Abstract: Artificial intelligence (AI) has revolutionized healthcare by enhancing diagnostic precision, personalizing treatments, and optimizing clinical workflows. However, algorithmic biases arising from imbalanced training data dis-proportionately affect marginalized populations, perpetuating health disparities. For instance, a widely-used predictive model underprioritized Black patients by equating healthcare costs with illness severity, restricting access to essential care. To address these challenges, this paper introduces an integrated framework combining adversarial debiasing, reinforcement learning (RL)-driven threshold optimization, synthetic data augmentation, and community engagement strategies. Empirical evaluations using the MIMIC-III dataset demonstrate near-perfect Area Under the ROC Curve (AUC = 0.994) alongside significant reductions in demographic parity (DP = 0.000) and error rate differences across subgroups. While the RL-optimized threshold ensures no missed diagnoses, it increases false positives, highlighting the trade-offs between fairness and precision. This work underscores the importance of synthesizing technical, regulatory, and community-driven approaches to promote equitable AI deployments. By embedding participatory design principles and aligning with emerging governance frameworks like the European Union’s AI Act, the proposed methodology offers a scalable pathway toward transparent, accountable, and high-impact healthcare AI solutions.
Title: Racial Bias in Multimodal Classification of Healthcare Data - Evidence from MIMIC-IV
Authors: Suchit Patel; Pranjali Attarde; Manik Gupta
Affiliations: Birla Institute of Technology and Science - Hyderabad Campus; Birla Institute of Technology and Science - Hyderabad Campus; Birla Institute of Technology and Science - Hyderabad Campus
Year: 2025
Domain: Health & Clinical AI

Abstract: The increase in the number of patients per doctor has created a significant need for digital assistance in healthcare. Managing patient data has become more challenging as it continues to grow, making timely treatment more difficult. AI/ML techniques can assist clinicians in providing personalized and timely health care services. Leveraging multimodal data from various sources can further improve pattern recognition and predictions. This study introduces a multimodal deep learning approach for predicting chronic health issues, utilizing image, text, and tabular data from the MIMIC-IV dataset. The model integrates three neural networks, each processing a different data type, and combines their outputs for predictions. However, multimodal data makes bias analysis challenging, as each data modality may introduce its own biases. The paper’s main contribution is to conduct a detailed bias analysis to evaluate the multimodal model performance across different racial groups. The results demonstrate strong overall performance but reveal significant discrepancies leading to unequal treatment and poorer diagnosis, particularly between WHITE and BLACK groups. These findings highlight the necessity of bias mitigation in healthcare AI models to ensure equitable outcomes for all patients.
Title: Fairness Metrics in AI Healthcare Applications: A Review
Authors: Ibomoiye Domor Mienye; Theo G. Swart; George Obaido
Affiliations: University of Johannesburg; University of Johannesburg
Year: 2024
Domain: Health & Clinical AI

Abstract: As artificial intelligence (AI) systems increasingly become popular in the healthcare sector, it is important to ensure the output of these technologies is fair and bias-free. This paper provides a concise survey of fairness metrics applied in healthcare AI, including their mathematical representations, suitable use cases, and limitations, which are lacking in the existing literature. The study also highlights the significance of implementing fairness metrics to ensure equitable outcomes across diverse patient populations and discusses the challenges and future directions in this rapidly evolving field.
Title: Enhancing Multi-Attribute Fairness in Healthcare Predictive Modeling
Authors: Xiaoyan Wang; Christopher C. Yang
Affiliations: Drexel University; Drexel University
Year: 2025
Domain: Health & Clinical AI

Abstract: Artificial intelligence (AI) systems in healthcare have demonstrated remarkable potential to improve patient outcomes. However, if not designed with fairness in mind, they also carry the risks of perpetuating or exacerbating existing health disparities. Although numerous fairness-enhancing techniques have been proposed, most focus on a single sensitive attribute and neglect the broader impact that optimizing fairness for one attribute may have on the fairness of other sensitive attributes. In this work, we introduce a novel approach to multi-attribute fairness optimization in healthcare AI, tackling fairness concerns across multiple demographic attributes concurrently. Our method follows a two-phase approach: initially optimizing for predictive performance, followed by fine-tuning to achieve fairness across multiple sensitive attributes. We develop our proposed method using two strategies, sequential and simultaneous. Our results show a significant reduction in Equalized Odds Disparity (EOD) for multiple attributes, while maintaining high predictive accuracy. Notably, we demonstrate that single-attribute fairness methods can inadvertently increase disparities in non-targeted attributes whereas simultaneous multi-attribute optimization achieves more balanced fairness improvements across all attributes. These findings highlight the importance of comprehensive fairness strategies in healthcare AI and offer promising directions for future research in this critical area.
Title: De-Biased Disentanglement Learning for Pulmonary Embolism Survival Prediction on Multimodal Data
Authors: Zhusi Zhong; Jie Li; Shreyas Kulkarni; Helen Zhang; Fayez H. Fayad; Li Yang; Scott Collins; Harrison X. Bai; Sun Ho Ahn; Michael K. Atalay; Xinbo Gao; Zhicheng Jiao
Affiliations: Xidian University; Xidian University; Providence College; Providence College; Providence College; Central South University; Providence College; Johns Hopkins University; Providence College; Providence College; Xidian University; Providence College
Year: 2024
Domain: Health & Clinical AI

Abstract: Health disparities among marginalized populations with lower socioeconomic status significantly impact the fairness and effectiveness of healthcare delivery. The increasing integration of artificial intelligence (AI) into healthcare presents an opportunity to address these inequalities, provided that AI models are free from bias. This paper aims to address the bias challenges by population disparities within healthcare systems, existing in the presentation of and development of algorithms, leading to inequitable medical implementation for conditions such as pulmonary embolism (PE) prognosis. In this study, we explore the diverse bias in healthcare systems, which highlights the demand for a holistic framework to reducing bias by complementary aggregation. By leveraging de-biasing deep survival prediction models, we propose a framework that disentangles identifiable information from images, text reports, and clinical variables to mitigate potential biases within multimodal datasets. Our study offers several advantages over traditional clinical-based survival prediction methods, including richer survival-related characteristics and bias-complementary predicted results. By improving the robustness of survival analysis through this framework, we aim to benefit patients, clinicians, and researchers by enhancing fairness and accuracy in healthcare AI systems.
Title: Mitigating Racial Bias in Skin Lesion Classification with a Novel Deep Learning-Driven Dataset
Authors: Priyanka Joshi; Chanet Ruangrit; Poom Konghuayrob
Affiliations: King Mongkut's Institute of Technology Ladkrabang; King Mongkut's Institute of Technology Ladkrabang; King Mongkut's Institute of Technology Ladkrabang
Year: 2025
Domain: Health & Clinical AI

Abstract: (1) Background: A critical issue in the application of Machine Learning (ML) in dermatology is the presence of racial bias in training datasets, which can lead to disparities in diagnostic performance across different skin tones. This paper proposes a mitigation strategy for the BCN20000 dataset, addressing its limitations regarding darker skin lesions by augmenting it with a custom dataset. (2) Method: This was achieved by augmenting the BCN20000 dataset with a custom-developed dataset of dark skin lesion images, generated by combining serial style transfer and latent diffusion-based upscaling. The impact of this augmented dataset was assessed on the accuracy of several established Convolutional Neural Network (CNN) architectures, including DenseNet, ConvNeXt, EfficientNet, RegNet and ResNet. (3) Results: Our findings underscore the necessity of diverse datasets for AI-driven dermatology tools with the best performing model achieving 92% accuracy when trained on the augmented dataset containing additional dark-skin images compared to baseline models trained solely on BCN20000. These results align with prior studies emphasizing the critical role of representative training data in mitigating racial bias in medical AI systems.