cair-nepal/ai-bias-research-landscape huggingface dataset:
Title: Underrepresented Tones: Addressing Skin Bias in Medical Imaging for Eczema, Psoriasis, and Melanoma Detection Using CNNs
Authors: Elizabeth Ango Fomuso Ekellem; Lutz Köhler
Affiliations: University of Cologne; University of Cologne
Year: 2023
Domain: Health & Clinical AI

Abstract: Medical imaging with convolutional neural networks (CNNs) has the potential to improve early disease detection. However, the success of CNNs relies heavily on the quality and diversity of the training data. Our study focused on training CNNs using datasets for conditions like eczema, psoriasis, and melanoma across various skin tones. We found that Black skin data was underrepresented. This lack of data led to lower accuracy when analyzing African skin samples, highlighting a clear bias in medical imaging. Many believe that just using more data can solve these problems. But our results, along with studies by Romero Lopez et al. (2017) [11] and Yu et al. (2017) [13], show that having a diverse and high-quality dataset is more important. In summary, there’s a need to include data from all skin tones to ensure fairness in medical imaging. This isn’t just a technical problem, but a matter of equal healthcare for all. We hope for a future where medical AI is inclusive and represents everyone fairly.
Title: Assessing Bias Within Diabetes Risk Prediction in Machine Learning Techniques
Authors: Ayesha Faruki
Affiliations: Mentor
Year: 2025
Domain: Health & Clinical AI

Abstract: As machine learning (ML) models increasingly influence clinical decision-making, concerns over fairness and bias have emerged, particularly in predictive healthcare tools. This study investigates racial disparities in diabetes risk prediction using Random Forest, XGBoost, and Linear Regression models trained on the 2021-2023 National Health and Nutrition Examination Survey (NHANES) dataset. Model performance was evaluated across key metrics—accuracy, sensitivity, specificity, F1 score, and ROC AUC—both overall and stratified by race and ethnicity. Initial results revealed strong aggregate performance (ROC AUC greater than 0.91) but significant disparities in recall and precision for underrepresented racial groups. After incorporating demographic stratification during preprocessing, prediction improved across minority groups, though overall model performance slightly declined. These results emphasize the trade-off between model performance and equity, and underscore the need for fairness-aware approaches in healthcare AI.
Title: fair-LDP: Uncertainty-Guided Fairness and Privacy for Federated Healthcare Learning
Authors: Dian Chen; Qi Zhang; Lance Kaplan; Audun Jøsang; Donghyun Jeong; Feng Chen; Jin-Hee Cho
Affiliations: Virginia Tech; Virginia Tech; DEVCOM Army Research Laboratory; University of Oslo; University of the District of Columbia; The University of Texas at Dallas; Virginia Tech
Year: 2025
Domain: Health & Clinical AI

Abstract: Federated Learning (FL) offers a promising approach for collaborative model training in healthcare while preserving data privacy. However, existing FL methods often fall short in addressing two critical challenges: client-level fairness and compounded uncertainty from data heterogeneity and privacy-preserving mechanisms. We propose fair-LDP, a fairness-aware Local Differential Privacy framework that promotes fairness and privacy via uncertainty-guided aggregation in federated healthcare AI. fair-LDP leverages evidential neural networks (ENNs) to quantify predictive uncertainty and introduces a novel strategy that uses uncertainty-driven local differential privacy to guide fairness-aware updates while preserving data privacy. This ensures equitable performance across clients with varying data quality while mitigating the influence of unreliable or outlier updates. fair-LDP incorporates an adaptive mechanism that adjusts each client's privacy budget based on model performance, balancing fairness, privacy, and accuracy. We evaluate fair-LDP on real-world healthcare datasets under both IID and non-IID settings. Our experimental results show that it consistently outperforms state-of-the-art fairness-aware and privacy-preserving FL baselines, with no added computational overhead, while maintaining privacy guarantees comparable to homomorphic encryption and secure multiparty computation. By integrating uncertainty modeling, fairness-aware aggregation, and adaptive local differential privacy, fair-LDP provides a practical and principled solution for responsible, equitable, and privacy-preserving federated learning in healthcare.
Title: Addressing Algorithmic Bias in Diabetes Care: A Multi-Modal Machine Learning Framework for Equitable Readmission Prediction
Authors: Md Mehedi Hasan; Md Mujibur Rahman; Quazi Mamun; Jiao Wu; MA Rana
Affiliations: Charles Sturt University; Charles Sturt University; Charles Sturt University; Waseda University; Charles Sturt University
Year: 2025
Domain: Health & Clinical AI

Abstract: Diabetes readmissions affect 14.4-22.7% of patients, disproportionately impacting minority populations and costing $3.8 billion annually. Existing prediction models suffer from algorithmic bias and limited clinical interpretability. Develop a fairness-optimized multi-modal machine learning framework for equitable diabetes readmission prediction with actionable clinical insights. We designed a novel architecture integrating four specialized neural networks: clinical data processing through deep neural networks, medication interactions via graph neural networks, demographic factors through fairness-aware models, and temporal patterns using LSTM networks. Adversarial debiasing and multi-objective optimization ensured equitable performance across demographic groups. The framework was validated using 101,766 diabetes encounters from 130 US hospitals and pilottested in three healthcare systems. Results: The multi-modal framework achieved AUC-ROC of 0.923, representing 18.5% improvement over Random Forest baselines. Bias mitigation reduced demographic disparities by 85%, with all fairness metrics below 5% thresholds. SHAP analysis identified 15 key risk factors, 60% being clinically modifiable. Six-month pilot implementation demonstrated 23.7% readmission reduction and $2.3 million healthcare savings. This framework successfully integrates predictive accuracy, algorithmic fairness, and clinical interpretability, establishing a new paradigm for equitable healthcare AI that transforms diabetes care from reactive treatment to proactive prevention.
Title: FairCLIP: Harnessing Fairness in Vision-Language Learning
Authors: Yan Luo et al.
Affiliations: Harvard University; Harvard University; New York University; New York University; New York University; New York University; Harvard University; Harvard University; Harvard University; Harvard University; New York University; Harvard University
Year: 2024
Domain: Health & Clinical AI

Abstract: Fairness is a critical concern in deep learning, especially in healthcare, where these models influence diagnoses and treatment decisions. Although fairness has been investigated in the vision-only domain, the fairness of medical vision-language (VL) models remains unexplored due to the scarcity of medical VL datasets for studying fairness. To bridge this research gap, we introduce the first fair vision-language medical dataset (Harvard-FairVLMed) that provides detailed demographic attributes, ground-truth labels, and clinical notes to facilitate an in-depth examination of fairness within VL foundation models. Using Harvard-FairVLMed, we conduct a comprehensive fairness analysis of two widely-used VL models (CLIP and BLIP2), pre-trained on both natural and medical domains, across four different protected attributes. Our results highlight significant biases in all VL models, with Asian, Male, Non-Hispanic, and Spanish being the preferred subgroups across the protected attributes of race, gender, ethnicity, and language, respectively. In order to alleviate these biases, we propose FairCLIP an optimal-transport-based approach that achieves a favorable trade-off between performance and fairness by reducing the Sinkhorn distance between the overall sample distribution and the distributions corresponding to each demographic group. As the first VL dataset of its kind, Harvard-FairVLMed holds the potential to catalyze advancements in the development of machine learning models that are both ethically aware and clinically effective. Our dataset and code are available at https://ophai.hms.harvard.edu/datasets/harvard-fairvlmed10k.
Title: AGR: Age Group fairness Reward for Bias Mitigation in LLMs
Authors: Shuirong Cao; Roger Cheng; Zhiqiang Wang
Affiliations: Nanjing University; Beijing Electronic Science and Technology Institute; Beijing Electronic Science and Technology Institute
Year: 2025
Domain: LLMs & NLP

Abstract: LLMs can exhibit age biases, resulting in unequal treatment of individuals across age groups. While much research has addressed racial and gender biases, age bias remains little explored. The scarcity of instruction-tuning and preference datasets for age bias hampers its detection and measurement, and existing fine-tuning methods seldom address age-related fairness. In this paper, we construct age bias preference datasets and instruction-tuning datasets for RLHF. We introduce AGR, an age fairness reward to reduce differences in the response quality of LLMs across different age groups. Extensive experiments demonstrate that this reward significantly improves response accuracy and reduces performance disparities across age groups. Our source code and datasets are available at the anonymous link.
Title: Causal Mitigation of Pink Tax: Intersectional Bias Reduction in LLMs
Authors: Yaxuan Huang; Peter Y. T. Sun
Affiliations: Beijing International Studies University; Science Oxford
Year: 2025
Domain: LLMs & NLP

Abstract: While many stereotypes target intersectional demographic groups, prior studies on bias within Large Language Models (LLMs) primarily focus on single, broad categories. This research bridges this gap by introducing a novel, manually-validated intersectional stereotype dataset. Motivated by the real-world "Pink Tax" phenomenon—where products marketed to women are often priced higher than similar ones for men—we investigate its algorithmic counterpart. We propose and quantify the "Algorithmic Pink Tax," a systematic economic bias in LLMs where female-associated profiles lead to higher recommended prices for comparable products. We first quantify this LLM's discriminatory bias at the functional decision level using Ordinary Least Squares (OLS) regression and Hypothesis Testing, moving beyond purely lexical bias, explicitly controlling for product comparability in our prompts. Moving beyond purely lexical bias, we rigorously discuss the causal assumptions underpinning our intervention. Building on this quantitative analysis, we propose two technical mitigation strategies: Intervention Point Analysis (targeting the cost-effective instruction-tuning phase) and Counterfactual Data Augmentation (CDA), which applies a causal fairness framework for targeted knowledge injection via Parameter-Efficient Fine-Tuning (LoRA). Our experiments demonstrate that by focusing intervention on the instruction-tuning phase and leveraging the efficiency of LoRA and CDA, the model's primary discriminatory price coefficient, a key measure of the Algorithmic Pink Tax, can be significantly reduced by over 40%, providing a highly efficient and practical technical path for building fairer LLMs.
Title: Evaluating Bias Detection and Mitigation Approaches Across Classical and Large Language Models
Authors: S. Anand; Aditi Gopinath; Karthikeyan Periyasami
Affiliations: School of Computer Science and Engineering, RV University, Bengaluru, Karnataka 560059, India
Year: 2026
Domain: LLMs & NLP

Abstract: In recent years, the increasing reliance on Artificial Intelligence (AI) and Large Language Model (LLM) systems have drawn significant attention to issues of fairness and bias in Machine Learning (ML) models that often inherit disparities present in training data or algorithmic design. IBM’s AI Fairness 360 (AIF360) toolkit provides tools for detecting these biases with fairness metrics and correcting them with mitigation techniques. The mitigation techniques are implemented at three stages: Pre-Processing, In-Processing, and Post-Processing. As no single approach is effective across all scenarios, careful consideration of dataset properties and model behavior is required to mitigate bias for optimizing the model’s performance. This study has taken a closer look at these methods and has compared them in terms of what kind of bias they respond to, amount of bias they can reduce, what degree of accuracy they maintain, and how computationally efficient they are. Through comparing the techniques, the study proposes the best techniques based on the situation, which provides the study with a recommendation on how the techniques can be applied in real-life situations. The research analyses bias reduction in LLMs, including DistilBERT (Bidirectional Encoder Representations from Transformers), GPT-2 (Generative Pre-trained Transformer-2), and BLOOM (BigScience Large Open-science Open-access Multilingual Language Model), assessing fairness measures and mitigation performance on transformer-based models. The applicability of AIF360 is expanded by this integration, which proves the success of the model when it comes to bias reduction among ML models and Natural Language Processing (NLP) systems.
Title: Leveraging Large Language Models for Bias Detection and Mitigation in Data Analytics Models
Authors: Manideep Marripudugala
Affiliations: University of New Haven
Year: 2025
Domain: LLMs & NLP

Abstract: Bias in data analytics models presents significant challenges, leading to unfair outcomes and undermining trust in artificial intelligence systems. Traditional methods for bias detection often rely on statistical techniques that may not capture complex patterns embedded within data. This study proposes a novel approach utilizing Large Language Models (LLMs) to detect and mitigate bias in data analytics models. By harnessing the deep contextual understanding of LLMs, we aim to identify subtle biases that standard methods might overlook. We developed a framework where LLMs are employed to analyze both the input data and the outputs of predictive models to detect bias related to sensitive attributes such as race, gender, and age. The LLMs are fine-tuned on datasets annotated for bias-related features, enabling them to recognize and interpret nuanced patterns of discrimination. To validate our approach, we conducted experiments on real-world datasets, including the COMPAS dataset for criminal recidivism risk assessment and the Adult Income dataset from the UCI Machine Learning Repository. Our LLM-based method successfully identified biased predictions and provided insights into their origins. For instance, in the COMPAS dataset, the LLM detected disproportionate risk assessments against certain demographic groups. Upon applying our mitigation strategies, there was a notable improvement in the fairness metrics without compromising the overall model performance. The results demonstrate that LLMs can be powerful tools for enhancing the fairness of data analytics models. They offer a more comprehensive understanding of bias by considering contextual information that traditional statistical methods may miss. This research contributes to the field by providing an effective methodology for bias detection and mitigation, promoting the development of more equitable AI systems.
Title: ThaiSBi: A Thai Social Bias Dataset for Large Language Model
Authors: Waranrach Viriyavit; Pasinee Rukpromrach; Titapon Keawbua; Napas Jenpromrash; Thanadol Saelem; Sitthaa Phaholphinyo; Sarawoot Kongyoung
Affiliations: Burapha University; Burapha University; Burapha University; Kasetsart University; Kasetsart University; National Science and Technology Development Agency; National Science and Technology Development Agency
Year: 2025
Domain: LLMs & NLP

Abstract: Large Language Models (LLMs) are revolutionizing various fields such education, business, and communication, with LLMs' ability to process data and provide complex answers. However, these technologies still face significant challenges, e.g., hallucinations, biases, and difficulties in understanding cultural and social contexts. Privacy and ethical concerns also require careful consideration. This study aims to address these issues by developing a set of comparative prompts designed to capture diverse perspectives across 67 demographic groups within Thai society, categorized into 15 groups. This approach highlights the importance of adopting culturally sensitive methods in AI development to mitigate risks associated with social biases. Therefore, we introduce ThaiSBi a curated dataset designed to assess and reduce social biases in LLMs within the Thai context. ThaiSBi provides a structured framework for evaluating LLM performance across various social dimensions, offering a deeper understanding of how these models interact with different groups. By incorporating these perspectives, ThaiSBi promotes the development of ethical and fair AI systems aligned with local cultural values. This research contributes to the creation of tools that prioritize fairness, safety, and equity in AI applications, supporting policy-making and social analysis in Thailand. The dataset is publicly available at https://github.com/9meo/thaisbi
Title: Fairness in AI: Evaluating and Mitigating Bias in Decision-Making Models
Authors: Tejas Kambli; S. Divakara Shetty
Affiliations: Birla Institute of Technology and Science, Pilani - Dubai Campus; Birla Institute of Technology and Science, Pilani - Dubai Campus
Year: 2025
Domain: LLMs & NLP

Abstract: Artificial Intelligence (AI) has been at the forefront of technological progress in recent years, driving innovation across multiple domains, and has become an integral part of modern decision-making systems, influencing industries such as healthcare, finance, and criminal justice. In the banking sector, AI-powered models are widely used for credit risk assessment, fraud detection, and loan approvals streamlining financial decision-making and improving efficiency. However, concerns regarding fairness, transparency, and bias have emerged, as AI systems often inherit and amplify societal inequalities present in their training data. This can lead to discriminatory lending practices, disproportionately affecting marginalized communities by restricting access to financial services. Addressing bias in AI-driven financial models is critical to ensuring honest and fair access to credit, preventing systemic discrimination, and fostering trust in automated banking solutions. Integrating XAI frameworks, fairness metrics (e.g., Equalized Odds, Demographic Parity, Disparate Impact), and continuous model auditing will be essential in building responsible AI-driven financial ecosystems that align with ethical AI principles and regulatory compliance standards such as GDPR, the AI Act, and the Algorithmic Accountability Act. This study explores the sources of bias in AI, evaluates key fairness metrics, and examines various mitigation techniques such as reweighting, adversarial debiasing, and fairness-aware algorithms to develop a comprehensive framework for creating fair, accountable, and transparent AI systems in banking.
Title: Quantifying and Mitigating Occupational Bias in Open-Source Large Language Models
Authors: Animesh Raj; Zayed Haque; K Shantha Kumari; P. Kanmani; Raghava Rao Mukkamala
Affiliations: Chennai Mathematical Institute; Chennai Mathematical Institute; Chennai Mathematical Institute; Chennai Mathematical Institute; Copenhagen Business School
Year: 2025
Domain: LLMs & NLP

Abstract: Generative artificial intelligence has the potential to both exacerbate and ameliorate existing socioeconomic inequalities. In this article, we provide a state-of-the-art interdisciplinary overview of the potential impacts of generative AI on (mis)information and three information-intensive domains: work, education, and healthcare. Our goal is to highlight how generative AI could worsen existing inequalities while illuminating how AI may help mitigate pervasive social problems. In the information domain, generative AI can democratize content creation and access, but may dramatically expand the production and proliferation of misinformation. In the workplace, it can boost productivity and create new jobs, but the benefits will likely be distributed unevenly. In education, it offers personalized learning, but may widen the digital divide. In healthcare, it might improve diagnostics and accessibility, but could deepen pre-existing inequalities. In each section we cover a specific topic, evaluate existing research, identify critical gaps, and recommend research directions, including explicit trade-offs that complicate the derivation of a priori hypotheses. We conclude with a section highlighting the role of policymaking to maximize generative AI’s potential to reduce inequalities while mitigating its harmful effects. We discuss strengths and weaknesses of existing policy frameworks in the European Union, the United States, and the United Kingdom, observing that each fails to fully confront the socioeconomic challenges we have identified. We propose several concrete policies that could promote shared prosperity through the advancement of generative AI. This article emphasizes the need for interdisciplinary collaborations to understand and address the complex challenges of generative AI.
Title: Exploring Bias Evaluation Techniques for Quantifying Large Language Model Biases
Authors: Junheng He; Nankai Lin; Menglan Shen; Dong Zhou; Aimin Yang
Affiliations: Guangdong University of Technology; Guangdong University of Technology; Peking University; Guangdong University of Foreign Studies; Lingnan Normal University
Year: 2023
Domain: LLMs & NLP

Abstract: In recent years, there has been a surge in the adoption of large language models (LLMs) such as “ChatG PT” trained by OpenAI. These models have gained popularity due to their impressive performance in various real-world applications. However, research has shown that small pre-trained language models (PLMs) such as BERT exhibit biases, particularly gender bias, that mirror societal stereotypes. Given the shared architecture between LLMs and small PLMs like Transformer, there is concern that these biases may also exist in LLMs. Although some studies suggest the presence of biases in LLMs, there is no consensus on how these biases should be measured. This paper employs three internal bias metrics, namely SEAT, StereoSet, and CrowS Pairs, to evaluate nine bias involving gender, age, race, occupation, nationality, religion, sexual orientation, physical appearance and disability in five open source LLMs (Llama, Llama2, Alpaca, Vicuna, and MPT), thereby determining their specific bias level. The experimental results demonstrate varying degrees of bias within these LLMs, with some models displaying high levels of bias that could potentially lead to harm in specific domains. Interestingly, we also discover that despite their larger architectures and greater number of parameters compared to small PLMs like BERT, these LLMs exhibit a lower level of bias. We posit that the inclusion of fairness considerations during the pre-training phase of these Language Model-based Learners (LLMs) is the primary contributing factor. This involves prioritizing the use of “fair” corpora while constructing the training data, and our experimental findings confirm the effectiveness of such an approach. Finally, by identifying the presence and measuring the specific level of bias, we contribute to the ongoing discourse on the mitigation of bias and the responsible usage of LLMs in various applications.