cair-nepal/ai-bias-research-landscape huggingface dataset:
Title: Enhancing Data Diversity and Traceability to Mitigate Bias in Healthcare AI: A Blockchain-Driven Approach
Authors: Lu Xiong; Yaping Zhu; Sam Zaza
Affiliations: Middle Tennessee State University; Middle Tennessee State University; Middle Tennessee State University
Year: 2024
Domain: Health & Clinical AI

Abstract: In the field of healthcare AI, the prevalence of algorithmic bias poses a significant challenge, often stemming from data biases. This research presents a novel approach to mitigate data biases by harnessing the potential of blockchain technology to enhance data diversity and traceability within federated learning systems. Our focus is to establish a framework where blockchain bolsters data integrity and broadens data inclusivity. The core of our methodology lies in leveraging blockchain's distributed architecture, which inherently facilitates the integration of a varied and voluminous data pool. This contrasts sharply with limitations found in conventional health data standards like HL7 and FHIR, which often lack in providing a comprehensive data spectrum. Our study contributes to the ongoing discourse on fair and unbiased AI development in healthcare.
Title: Addressing Bias and Ensuring Fairness in AI Systems for Healthcare
Authors: Armita Zarnegar
Affiliations: Swinburne University of Technology
Year: 2025
Domain: Health & Clinical AI

Abstract: Artificial Intelligence (AI) is revolutionising healthcare by enhancing diagnostics, enabling personalised treatment, and improving patient outcome predictions. However, these systems also pose significant risks of perpetuating and amplifying biases, which could lead to inequitable healthcare delivery. This paper presents a synthesis of 25 peer-reviewed studies, narrowed down from an initial pool of 498 using eight screening criteria, focused on identifying sources of bias and evaluating mitigation strategies in medical AI. Sources of bias include imbalanced training data, algorithmic design flaws, deployment bias, and clinical implementation inequities, which often disproportionately impact marginalised populations. Notably, 22 out of the 25 studies emphasised data bias as the primary source, underscoring a strong research focus on dataset representativeness and quality. Common bias types identified include racial, gender, age-related, and socio-economic disparities. Mitigation approaches encompass pre-processing and data-centric techniques (e.g., resampling, disaggregation, data augmentation using GAN), model-centric strategies (e.g., multi-calibration and smoothing), fairness evaluation and transparency monitoring (e.g., SHAP, LIME, fairness metrics), and participatory approaches that promote transparency and stakeholder engagement. Notable gaps in the literature include limited attention to temporal fairness, mental health AI, and paediatric or geriatric specific biases. Among the selected studies, medical imaging emerged as the most frequently addressed application area. Some of these techniques, reported in single studies, indicate a need for further validation and integration into standard practice. These findings underscore the need for more comprehensive, equitable, and domain-specific approaches to responsible AI implementation in healthcare.
Title: An Agentic Framework for Compliant, Ethical and Trustworthy GenAI Applications in Healthcare
Authors: Veena Priscilla Menezes; Mohammad Jabed Morshed Chowdhury; Abdun Naser Mahmood
Affiliations: La Trobe University; La Trobe University; La Trobe University
Year: 2025
Domain: Health & Clinical AI

Abstract: Recent progress in generative artificial intelligence (GenAI) has yielded significant advancements in healthcare, affecting radiology, medical imaging, drug development, patient diagnostics, and supply chain optimisation.These innovations promise more improved diagnoses and time-saving cost-effectiveness.However, GenAI's rapid implementation poses significant challenges for meeting regulatory, ethical, and trustworthiness standards.These challenges include data privacy issues, reproducibility concerns, algorithmic bias in training data causing disparities in outcomes, and a lack of transparency and explainability.Unresolved, these issues could negatively affect the public's confidence in and perception of GenAI systems.Addressing these challenges, international AI governance frameworks, including the EU AI Act and WHO guidelines, prioritize regulatory adherence, trustworthiness, and the explainability of healthcare AI systems.While such frameworks have expanded, a deficiency remains in translating policy into effective compliance mechanisms.We propose a Compliance Agentic Model (CAM) framework to help organizations comply with GenAI and machine learning (ML)-based solutions.The CAM framework establishes trustworthiness in GenAI applications used in healthcare, ensuring alignment with organizational values and ethical standards to enhance accountability and regulatory adherence.
Title: Fair Foundation Models for Medical Image Analysis: Challenges and Perspectives
Authors: Dilermando Queiroz; Anderson Carlos; André Anjos; Lilian Berton
Affiliations: Universidade Federal de São Paulo; Instituto Federal de Goiás; Idiap Research Institute; Universidade Federal de São Paulo
Year: 2026
Domain: Health & Clinical AI

Abstract: Ensuring equitable Artificial Intelligence (AI) in healthcare demands systems that make unbiased decisions across all demographic groups, bridging technical innovation with ethical principles. Foundation Models (FMs), trained on vast datasets through self-supervised learning, enable efficient adaptation across medical imaging tasks while reducing dependency on labeled data. These models demonstrate potential for enhancing fairness, though significant challenges remain in achieving consistent performance across demographic groups. Our review indicates that effective bias mitigation in FMs requires systematic interventions throughout all stages of development. While previous approaches focused primarily on model-level bias mitigation, our analysis reveals that fairness in FMs requires integrated interventions throughout the development pipeline, from data documentation to deployment protocols. This comprehensive framework advances current knowledge by demonstrating how systematic bias mitigation, combined with policy engagement, can effectively address both technical and institutional barriers to equitable AI in healthcare. The development of equitable FMs represents a critical step toward democratizing advanced healthcare technologies, particularly for underserved populations and regions with limited medical infrastructure and computational resources.
Title: Towards Bias-Free Teledermatology: A Mobile-Optimized Framework Fusing Causal AI and Federated Learning
Authors: Niveditha N. Reddy; Pooja Agarwal
Affiliations: PES University; PES University
Year: 2025
Domain: Health & Clinical AI

Abstract: The integration of Large Language Models (LLMs) into legal practice raises pressing concerns about judicial fairness, particularly due to the nature of their ''black-box'' processes. This study introduces JustEva, a comprehensive, open-source evaluation toolkit designed to measure LLM fairness in legal tasks. JustEva features several advantages: (1) a structured label system covering 65 extra-legal factors; (2) three core fairness metrics -- inconsistency, bias, and imbalanced inaccuracy; (3) robust statistical inference methods; and (4) informative visualizations. The toolkit supports two types of experiments, enabling a complete evaluation workflow: (1) generating structured outputs from LLMs using a provided dataset, and (2) conducting statistical analysis and inference on LLMs' outputs through regression and other statistical methods. Empirical application of JustEva reveals significant fairness deficiencies in current LLMs, highlighting the lack of fair and trustworthy LLM legal tools. JustEva offers a convenient tool and methodological foundation for evaluating and improving algorithmic fairness in the legal domain. The toolkit is available for deployment at
Title: JustEva: A Toolkit to Evaluate LLM Fairness in Legal Knowledge Inference
Authors: Zongyue Xue; Siyuan Zheng et al.
Affiliations: Tsinghua University; Shanghai Jiao Tong University; Tsinghua University; University of Waterloo; Tsinghua University; Tsinghua University; Tsinghua University; Tsinghua University; Tsinghua University; Tsinghua University; Tsinghua University
Year: 2025
Domain: LLMs & NLP

Abstract: Recent advancements in Large Language Models (LLMs) have made them a popular information-seeking tool among end users. However, the statistical training methods for LLMs have raised concerns about their representation of under-represented topics, potentially leading to biases that could influence real-world decisions and opportunities. These biases could have significant economic, social, and cultural impacts as LLMs become more prevalent, whether through direct interactions—such as when users engage with chatbots or automated assistants—or through their integration into third-party applications (as agents), where the models influence decision-making processes and functionalities behind the scenes. Our study examines the biases present in LLMs recommendations of U.S. cities and towns across three domains: relocation, tourism, and starting a business. We explore two key research questions: (i) How similar LLM responses are, and (ii) How this similarity might favor areas with certain characteristics over others, introducing biases. We focus on the consistency of LLM responses and their tendency to over-represent or under-represent specific locations. Our findings point to consistent demographic biases in these recommendations, which could perpetuate a “rich-get-richer” effect that widens existing economic disparities.
Title: Investigating Perception of Gender Stereotypes in Large Language Models: A Computational Grounded Theory Approach
Authors: Rohan Charudatt Salvi; Nigel Bosch
Affiliations: University of Illinois Chicago; University of Illinois Urbana-Champaign
Year: 2025
Domain: LLMs & NLP

Abstract: Artificial Intelligence has expanded its influence far beyond traditional boundaries in our society. One prominent application of artificial intelligence is the use of large language models, which have transcended their initial roles in high-tech industries and academic research and are now actively utilized by individual users. These models have continually improved over the years in their generative capabilities and performance across numerous tasks. However, they still pose a persistent risk of reproducing biases and stereotypes. Previous research has predominantly focused on quantitatively measuring biases in these large language models. In this study, we seek to assess not just the presence of bias itself, but the perception of stereotypes by these models via in-depth exploration of their responses. We demonstrate how the computational grounded theory framework, which integrates qualitative and quantitative approaches, can be applied in this context to assess the conceptualization of stereotypes. Furthermore, we contrast language model results with a survey of 400 human participants who also completed similar prompts as the model in order to understand people’s perception of gender stereotypes. The results indicate substantial similarities between language model and human perceptions of stereotypes, highlighting that a model’s perception stems from societal perception of stereotypes.
Title: Indian-BhED: A Dataset for Measuring India-Centric Biases in Large Language Models
Authors: Khyati Khandelwal; Manuel Tonneau; Andrew M. Bean; Hannah Rose Kirk; Scott A. Hale
Affiliations: University of Oxford; University of Oxford; University of Oxford; University of Oxford; University of Oxford
Year: 2024
Domain: LLMs & NLP

Abstract: Large Language Models (LLMs), now used daily by millions, can encode societal biases, exposing their users to representational harms. A large body of scholarship on LLM bias exists but it predominantly adopts a Western-centric frame and attends comparatively less to bias levels and potential harms in the Global South. In this paper, we quantify stereotypical bias in popular LLMs according to an Indian-centric frame through Indian-BhED, a first of its kind dataset, containing stereotypical and anti-stereotypical examples in the context of caste and religious stereotypes in India. We find that the majority of LLMs tested have a strong propensity to output stereotypes in the Indian context, especially when compared to axes of bias traditionally studied in the Western context, such as gender and race. Notably, we find that GPT-2, GPT-2 Large, and GPT 3.5 have a particularly high propensity for preferring stereotypical outputs as a percent of all sentences for the axes of caste (63–79%) and religion (69–72%). We finally investigate potential causes for such harmful behaviour in LLMs, and posit intervention techniques to reduce both stereotypical and anti-stereotypical biases. The findings of this work highlight the need for including more diverse voices when researching fairness in AI and evaluating LLMs.
Title: Unmasking Bias in Financial AI: A Robust Framework for Evaluating and Mitigating Hidden Biases in LLMs
Authors: Shreshth Mehrotra; P Raghavendra; Balraj Prajesh; Hrishikesh Kambale; Puspita Majumdar
Affiliations: Mastercard;Mastercard;Mastercard;Mastercard;Mastercard
Year: 2025
Domain: LLMs & NLP

Abstract: Large Language Models (LLMs) are increasingly used in finance for tasks like market analysis, customer support, sentiment analysis, and automated reporting. However, LLMs often inherit and perpetuate biases from their training data, raising concerns about fairness and accuracy in high-stakes financial applications. While other domains such as medicine, law, and education have advanced in identifying, measuring, and reducing bias, finance lacks domain-specific datasets and robust fairness metrics. To address this, we introduce the FinBias dataset which includes bias-eliciting prompts related to the finance domain, and a comprehensive evaluation framework for publicly available LLMs, including robustness tests against jailbreaking. We also propose a new metric, SAFE (Safety-Adjusted Fairness Evaluation), which penalizes stereotypical and refusal responses while rewarding debiased outputs. Additionally, we present a prompt engineering-based mitigation strategy that effectively reduces bias. Experiments conducted on three publicly available LLMs - Mixtral, Gemma, and LLaMA demonstrate that these models exhibit significant bias, but the proposed prompt engineering-based mitigation strategy effectively reduces this bias. This research provides a practical foundation for the detection, evaluation and mitigation of bias in financial LLM applications.
Title: Fairness of Large Language Models in Education
Authors: Rong Gao; Qin Ni; Bingying Hu
Affiliations: Shanghai Normal University; Shanghai International Studies University; Shanghai Normal University
Year: 2024
Domain: LLMs & NLP

Abstract: The paper investigates the fairness of Large Language Models (LLMs) in education. It discusses the transformative impact of LLMs on teaching and learning practices, highlighting their potential biases and emphasizing the necessity for fairness. It categorizes the applications of AI in education and outlines strategies for improving fairness in LLMs through pre-processing, in-processing, and post-processing techniques. The conclusion advocates for a multidimensional strategy in leveraging AI's potential in education, ensuring a balance between technological advancements and a steadfast commitment to inclusivity and fairness.
Title: EVENS: Equality versus Equity Notion Spectrum of LLMs
Authors: Qingjing Chen; Rongxin Cheng; Yan Liu et al.
Affiliations: University of Bologna; Tsinghua University; Tsinghua University; Tsinghua University; Tsinghua University; Tsinghua University; University of Bologna; Tsinghua University; Tsinghua University
Year: 2025
Domain: LLMs & NLP

Abstract: The controversy surrounding COMPAS exposes a significant gap between computer science and social science in understanding bias, highlighting the need to align computational fairness metrics with humanistic interpretations. In response, we introduce the EVENS benchmark to align the Equality versus Equity Notion Spectrum in LLMs. Our contributions include constructing an equality–equity notion spectrum and generating a corresponding dataset of key fairness scenarios, evaluating models’ initial stances and test stance adjustments under external legal regulations and internal organizational regulations using Retrieval-Augmented Generation (RAG), introducing Chain-of-Thought (CoT) prompting to guide fairness reasoning, and adding an uncertain choice to assess its impact. Our findings indicate that LLMs initially favor equality over equity. Incorporating legal and organizational regulations of equity through RAG can reduce proportional equality in most models and enhance equity recognition in GPT4o significantly. CoT improves the equity reasoning of Chinese models but may also rationalize existing biases, and the uncertain option promotes more cautious responses. The links to the code and datasets: https://github.com/CrexCheng/EVEN.
Title: The Root Shapes the Fruit: On the Persistence of Gender-Exclusive Harms in Aligned Language Models
Authors: Anaelia Ovalle; Krunoslav Lehman Pavasovic; Louis Martin et al.
Affiliations: Seattle University; UCLA Health
Year: 2025
Domain: LLMs & NLP

Abstract: Natural-language assistants are designed to provide users with helpful responses while avoiding harmful outputs, largely achieved through alignment to human preferences.Yet there is limited understanding of whether alignment techniques may inadvertently perpetuate or even amplify harmful biases inherited from their prealigned base models.This issue is compounded by the choice of bias evaluation benchmarks in popular preference-finetuned models, which predominantly focus on dominant social categories, such as binary gender, thereby limiting insights into biases affecting underrepresented groups.To address this gap, we center transgender, nonbinary, and other gender-diverse (TGNB) identities to investigate how alignment procedures encode and interact with harmful TGNB biases in LLMs.Our key contributions include: 1) a comprehensive survey of bias evaluation modalities across leading preference-finetuned LLMs, highlighting critical gaps in genderdiverse representation, 2) systematic evaluation of gender-diverse biases across 16 publicly-available models across Direct Preference Optimization (DPO) stages, uncovering harms popular bias benchmarks fail to detect, and 3) a flexible framework for measuring harmful gender-diverse biases in implicit reward signals applicable to other social contexts.Our findings suggest that DPOaligned models are particularly sensitive to supervised finetuning (SFT), and may amplify two forms of real-world gender-diverse harms reflected in their base models: stigmatization and gender non-affirmative language.We conclude with recommendations tailored to DPO and broader alignment practices, advocating for the adoption of community-informed bias evaluation frameworks to more effectively identify and address underrepresented harms in LLMs.
Title: Addressing algorithmic bias and the perpetuation of health inequities: An AI bias aware framework
Authors: Ritu Agarwal; Margrét V. Bjarnadóttir; Lauren A. Rhue; Michelle Dugas; K. Crowley; Justin Clark; Guodong Gao
Affiliations: Johns Hopkins University; University of Maryland, College Park; University of Maryland, College Park; World Bank; Accenture (United States); University of Maryland, College Park; Johns Hopkins University
Year: 2022
Domain: Health & Clinical AI

Abstract: Fairness in recommender systems is a complex concept, involving multiple definitions, different parties for whom fairness is sought, and various scopes over which fairness might be measured. Researchers seeking fairness-aware systems have derived a variety of solutions, usually highly tailored to specific choices along each of these dimensions, and typically aimed at tackling a single fairness concern, i.e., a single definition for a specific stakeholder group and measurement scope. However, in practical contexts, there are a multiplicity of fairness concerns within a given recommendation application and solutions limited to a single dimension are therefore less useful. We explore a general solution to recommender system fairness using social choice methods to integrate multiple heterogeneous definitions. In this paper, we extend group-fairness results from prior research to provider-side individual fairness, demonstrating in multiple datasets that both individual and group fairness objectives can be integrated and optimized jointly. We identify both synergies and tensions among different objectives with individual fairness correlated with group fairness for some groups and anti-correlated with others.
Title: Mitigating algorithmic bias in credit scoring support systems through adversarial learning
Authors: Junye Lin; Liping Yang
Affiliations: Renmin University of China; Tencent (China); Tencent (China); Tencent (China); Renmin University of China; Renmin University of China
Year: 2026
Domain: General Fairness & Bias Mitigation

Abstract: The use of recommender systems in the recruitment domain has been labeled as ‘high-risk’ in recent legislation. As a result, strict requirements regarding explainability and fairness have been put in place to ensure proper treatment of all involved stakeholders. To allow for stakeholder-specific explainability, while also handling highly heterogeneous recruitment data, we propose a novel explainable multi-stakeholder job recommender system using graph neural networks: the Occupational Knowledge-based Recommender using Attention (OKRA). The proposed method is capable of providing both candidate- and company-side recommendations and explanations. We find that OKRA performs substantially better than six baselines in terms of nDCG for two datasets. Furthermore, we find that the tested models show a bias toward candidates and vacancies located in urban areas. Overall, our findings suggest that OKRA provides a balance between accuracy, explainability, and fairness.
Title: Algorithmic bias and transparency in artificial intelligence tools for nursing education: A scoping review
Authors: Mohammad Ishtiaque Rahman
Affiliations: Queen Mary University of London; Queen Mary University of London
Year: 2026
Domain: Health & Clinical AI

Abstract: Machine learning based systems are reaching society at large and in many aspects of everyday life. This phenomenon has been accompanied by concerns about the ethical issues that may arise from the adoption of these technologies. ML fairness is a recently established area of machine learning that studies how to ensure that biases in the data and model inaccuracies do not lead to models that treat individuals unfavorably on the basis of characteristics such as e.g. race, gender, disabilities, and sexual or political orientation. In this manuscript, we discuss some of the limitations present in the current reasoning about fairness and in methods that deal with it, and describe some work done by the authors to address them. More specifically, we show how causal Bayesian networks can play an important role to reason about and deal with fairness, especially in complex unfairness scenarios. We describe how optimal transport theory can be used to develop methods that impose constraints on the full shapes of distributions corresponding to different sensitive attributes, overcoming the limitation of most approaches that approximate fairness desiderata by imposing constraints on the lower order moments or other functions of those distributions. We present a unified framework that encompasses methods that can deal with different settings and fairness criteria, and that enjoys strong theoretical guarantees. We introduce an approach to learn fair representations that can generalize to unseen tasks. Finally, we describe a technique that accounts for legal restrictions about the use of sensitive attributes.