cair-nepal/ai-bias-research-landscape huggingface dataset:
Title: A resource-efficient framework for cultural alignment in large language models (LLMs): The Chinese context
Authors: Jingyu Wu; Shi Chen; Pengchen Chen; Wantong Du; Hong Gao; Lu Wang; Preben Hansen
Affiliations: University of Bologna; Tsinghua University; Tsinghua University; Tsinghua University; Tsinghua University; Tsinghua University; University of Bologna; Tsinghua University; Tsinghua University
Year: 2026
Domain: LLMs & NLP

Abstract: • We propose a cost-effective Human-AI cultural alignment design framework to enhance LLMs’ cultural understanding and conversational capabilities in Chinese contexts, focusing on cultural values, culturally specific content, and conversational logic design. • We introduced a targeted fine-tuning framework that combines LoRA fine-tuning, Retrieval-Augmented Generation (RAG), and Chain-of- Thought (CoT) prompting to address cultural challenges without extensive resource requirements, emphasizing resource-efficient design. • We conduct extensive evaluations across six datasets using several stateof-the-art LLMs, demonstrating consistent and significant improvements in cultural sensitivity, comprehension accuracy, and user satisfaction,validating the effectiveness of our design approach. The rapid development of Artificial Intelligence (AI) has underscored the urgent need to align technological innovation with ethical standards and Environmental, Social, and Governance (ESG) principles. A critical challenge lies in the design and deployment of Large Language Models (LLMs), predominantly trained on English-centric data, that can understand and respond to diverse cultural contexts without perpetuating bias, discrimination, or societal inequalities, posing a risk to responsible AI system design. In this study, we propose a low-cost and scalable Human-AI cultural alignment design framework to enhance LLMs cultural sensitivity and ethical alignment, with a focus on the Chinese context as a representative case. Our approach uses LoRA fine-tuning, Retrieval-Augmented Generation (RAG), and Chain-of-Thought (CoT) prompting to address three design dimensions: (1) embedding cultural values into model reasoning, (2) improving comprehension of culturally specific content, and (3) fostering inclusive and ethically consistent conversational logic design. Experimental results across six downstream tasks demonstrate significant improvements in cultural awareness, fairness, and user trust, achieving performance comparable to GPT-4 and Gemini Pro while requiring fewer computational resources. Beyond cultural alignment, our method advances ESG goals: environmentally, by reducing the carbon footprint through lightweight fine-tuning; socially, by promoting inclusivity and reducing cultural bias through better UX design; and governance-wise, by offering a transparent, adaptable, and resource-efficient alignment strategy. This work highlights how culturally adaptive AI systems can contribute to a more equitable, sustainable, and ethically responsible technological future.
Title: Diagnosing the bias iceberg in large language models: A three-level framework of explicit, evaluative, and implicit gender bias
Authors: Anling Xiang
Affiliations: Seattle University; UCLA Health
Year: 2025
Domain: LLMs & NLP

Abstract: Large language models (LLMs) have achieved remarkable performance gains, yet concerns about their embedded biases remain pressing. Existing research often targets a single dimension, lacking systematic comparisons across different types of bias. This study introduces a three-level diagnostic framework—explicit, evaluative, and implicit—to characterize the “bias iceberg” in LLMs. We construct a bilingual dataset (Chinese–English) spanning seven socio-psychological dimensions (appearance, competence, dominance, emotion, leadership, morality, and physicality), comprising approximately 420 minimal-pair sentences and 400 word-association sets, and conduct unified evaluations across eight mainstream models (GPT-4o, Claude-3.7, Gemini-2.5, Grok-3, Qwen-Plus, DeepSeek-v3, Doubao, and Kimi-2). Results reveal that explicit bias remains generally low ( mean ≈ 0.0141), though residual disadvantages for women persist in appearance and emotion. Evaluative bias intensifies to a moderate level ( mean ≈ 0.0259), with directional divergence in morality and dominance. Implicit bias emerges as the most pronounced ( mean ≈ 0.1738, peaking at 0.45), manifesting stable male-anchoring effects in dominance and physicality, with a magnitude 12.3 times greater than explicit bias. Network analysis uncovers three structural archetypes—hyper-centralized, balanced clustering, and de-centralized. Cross-cultural comparisons further show that U.S. models more strongly reproduce “male–power/physicality” associations at evaluative and implicit levels, whereas Chinese models exhibit greater convergence. The proposed framework and dataset are reproducible and cross-culturally adaptable, offering new empirical evidence and structured insights for uncovering and mitigating deep-seated biases in LLMs.
Title: Fine-tuning large language models in federated learning with fairness-aware prompt selection
Authors: Yalan Jiang; Zhongliang Li; Bin Song
Affiliations: Johns Hopkins University; University of Maryland, College Park; University of Maryland, College Park; World Bank; Accenture (United States); University of Maryland, College Park; Johns Hopkins University
Year: 2025
Domain: LLMs & NLP

Abstract: Federated Learning (FL) is an increasingly popular form of distributed machine learning that addresses privacy concerns by allowing participants to collaboratively train machine learning models without exchanging their private data. Although FL emerged as a privacy-preserving alternative to centralized machine learning approaches, it faces significant challenges in preserving the privacy of its clients and mitigating potential bias against clients or disadvantaged groups. Most existing research in FL has addressed these two ethical notions separately, whereas ensuring privacy and fairness simultaneously in FL systems is of paramount importance. Moreover, current research efforts fail to balance privacy, fairness, and model performance, leaving systems vulnerable to various problems. To provide a comprehensive overview of these critical challenges, this work presents an integrated study of privacy and fairness concerns in the context of FL. In addition to providing an extensive review of the current literature on privacy and fairness issues, we also examine the existing approaches for achieving a balance between these two ethical notions to develop robust FL systems. Finally, we highlight potential research directions related to the challenges of implementing privacy-preserving and fairness-aware FL systems.
Title: Fairness and social bias quantification in Large Language Models for sentiment analysis
Authors: Majdi I. Radaideh et al.
Affiliations: Renmin University of China; Tencent (China); Tencent (China); Tencent (China); Renmin University of China; Renmin University of China
Year: 2025
Domain: LLMs & NLP

Abstract: Large Language Models (LLMs) have enhanced various Natural Language Processing (NLP) tasks, including text generation and classification. However, studies reveal that LLMs exhibit social biases, such as associating certain occupations with specific genders. While previous research has focused on bias in text generation, limited attention has been given to text classification, such as sentiment analysis. This study quantifies social bias in sentiment analysis using five open-source LLMs: BERT, GPT-2, LLaMA-2, Falcon, and MistralAI, fine-tuned on two large social media datasets, the first one is related to nuclear energy, while the second dataset is general and contains tweets related to various subjects. We conducted approximately 1,500 prompt experiments with variations in words reflecting energy source, gender, politics, age, and ethnicity. A fair language model should provide the same sentiment for both prompts; differing sentiments would indicate bias. Explainable methods were employed to analyze how words related to the five subjects (energy, gender, politics, age, ethnicity) contributed to the sentiment. Findings show that social bias persists in LLMs for sentiment analysis, and while fine-tuning can enhance fairness, it does not always eliminate bias, particularly regarding age groups.
Title: Implicit Gender, Racial, and Ethnic Biases in Large Language Models: An Audit Study of Automated Psychiatric Diagnoses
Authors: Sachin R. Pendse; Mini Jain; Neha Kumar; Munmun De Choudhury
Affiliations: Queen Mary University of London; Queen Mary University of London
Year: 2026
Domain: LLMs & NLP

Abstract: <h2>Abstract</h2><h3>Objective</h3> To evaluate whether large language models (LLMs) exhibit implicit gender, racial, and ethnic biases when used to provide psychiatric diagnoses, and to understand how such biases may impact the accuracy and fairness of AI-assisted clinical decision support. <h3>Patients and Methods</h3> We conducted a large-scale audit of six LLMs, including general-purpose and medical-specific models, using 97 DSM-5 psychiatric training cases, conducted between October 1<sup>st</sup> 2023, and June 23<sup>rd</sup> 2025. Cases were systematically altered to suggest different gender and racial/ethnic identities—across 39 demographic groups—by changing names, pronouns, and descriptors while keeping clinical symptoms constant. We assessed diagnostic accuracy, additional or missed diagnoses, and language of diagnostic reasoning. GPT-4o emerged as the most accurate model and was selected for deeper analysis. <h3>Results</h3> GPT-4o accurately identified at least one correct diagnosis in 82.8% of cases but often added non-ground-truth diagnoses (70.3% of cases), suggesting a tendency to overdiagnose. Accuracy varied by gender, with higher performance for female patients and lower for non-binary individuals. Although overall accuracy did not differ significantly by race/ethnicity, biased diagnostic patterns emerged—for example, cultural bereavement and antisocial behavior were diagnosed exclusively in patients of color, and terms like "disruptive" were used more frequently for Black men. <h3>Conclusion</h3> Our findings demonstrate that LLMs reproduce and reinforce clinical biases even when symptoms are constant. AI-based tools must be audited not only for accuracy but also for bias in both diagnoses and explanatory language, especially when used in high-stakes mental health contexts.
Title: Consistent focus: Mitigating permutation bias in large language models through attention weight averaging
Authors: Kunxi Li; Zesheng Shi; Miao Hu et al.
Affiliations: Delft University of Technology; Delft University of Technology
Year: 2025
Domain: LLMs & NLP

Abstract: Algorithmic bias indicates the discrimination caused by algorithms, which occurs with protected features such as gender and race. Many researchers have tried to define the fairness and devise methods to mitigate bias, but it is still premature to reach the unanimous definition and evaluation metrics of fairness according to society, times and cultures. In this paper, we introduce three evaluation metrics such as parity difference, equalized opportunity and equalized odds that can deal with various definitions of algorithmic bias, and concretely divide the three general approaches further into seven methods with some challenges, resulting in relabeling, generation, fair representation (for pre-processing), constraint optimization, regularization (for in-processing), calibration and thresholding (for post-processing). Among them, the pre-processing method is widely used due to its versatility, but it has limitation to deal with the information on data and features related with bias appropriately. In order to preserve the characteristics of the original data while excluding the information about the features causing bias, we propose a preprocessing approach based on information theory that avoids collision in the dual optimization, where the latent space is divided into two subspaces. Experiments are conducted with the well-known benchmark datasets of Census and COMPAS, and two real-world tasks: facial emotion recognition and text sentiment analysis. The information theoretic approach is promising to achieve fair machine learning by reducing the bias caused by several features such as age, race and gender.
Title: LIBRA: Measuring Bias of Large Language Model from a Local Context
Authors: B. Y. Pang; Tingrui Qiao; Caroline Walker; Chris Cunningham; Yun Sing Koh
Affiliations: University College Dublin; University College Dublin; University College Dublin; University College Dublin; University College Dublin
Year: 2025
Domain: LLMs & NLP

Abstract: Over the past decade, machine learning (ML) and artificial intelligence (AI) have become increasingly prevalent in the medical field. In the United States, the Food and Drug Administration (FDA) is responsible for regulating AI algorithms as “medical devices” to ensure patient safety. However, recent work has shown that the FDA approval process may be deficient. In this study, we evaluate the evidence supporting FDA-approved neuroalgorithms, the subset of machine learning algorithms with applications in the central nervous system (CNS), through a systematic review of the primary literature. Articles covering the 53 FDA-approved algorithms with applications in the CNS published in PubMed, EMBASE, Google Scholar and Scopus between database inception and January 25, 2022 were queried. Initial searches identified 1505 studies, of which 92 articles met the criteria for extraction and inclusion. Studies were identified for 26 of the 53 neuroalgorithms, of which 10 algorithms had only a single peer-reviewed publication. Performance metrics were available for 15 algorithms, external validation studies were available for 24 algorithms, and studies exploring the use of algorithms in clinical practice were available for 7 algorithms. Papers studying the clinical utility of these algorithms focused on three domains: workflow efficiency, cost savings, and clinical outcomes. Our analysis suggests that there is a meaningful gap between the FDA approval of machine learning algorithms and their clinical utilization. There appears to be room for process improvement by implementation of the following recommendations: the provision of compelling evidence that algorithms perform as intended, mandating minimum sample sizes, reporting of a predefined set of performance metrics for all algorithms and clinical application of algorithms prior to widespread use. This work will serve as a baseline for future research into the ideal regulatory framework for AI applications worldwide.
Title: PureBiasoMeter: Decoupling Popularity Bias from User Fairness in LLM-Based Recommender Systems
Authors: Shirin Tahmasebi; Muhammad Hamad; Amir H. Payberah; Mihhail Matskin
Affiliations: Hong Kong Baptist University; Hong Kong Baptist University; Hong Kong Baptist University
Year: 2026
Domain: Recommender Systems

Abstract: Large language models (LLMs) have demonstrated potential to transform many industries, but their reliance on vast internet data raises concerns about inherent biases. This study investigates occupational bias in leading open-source LLMs, highlighting differences between technical and creative careers. Using occupation-specific prompts from the BOLD dataset and evaluating outputs with the regard metric, chosen for its ability to capture nuanced social perceptions such as respect and stereotyping, we quantify sentiment patterns underlying occupational bias. Our analysis reveals a consistent tendency for LLMs to express more positive sentiment toward technical occupations, with this bias intensifying as model size increases. We evaluate multiple mitigation strategies, including fairness-aware fine-tuning and fairness-oriented prompt engineering, finding that their combination most effectively reduces bias without compromising performance. These findings underscore the need for proactive bias assessment and integrated mitigation approaches to promote fairness and equity in AI-driven language generation systems.
Title: Bias Mitigation and Shifting Ideologies of LLM by Prompting
Authors: Yuji Ueda; Hirohiko Suwa
Affiliations: Carnegie Mellon University; LinkedIn (United States); LinkedIn (United States); LinkedIn (United States)
Year: 2025
Domain: LLMs & NLP

Abstract: This paper critically reviews emergent biases in Large Language Models (LLMs), introducing a five-mode taxonomy: Scale-induced, Instruction-induced, Interaction-induced, Temporally-evolving, and Culturally-reinforced. We define emergent bias as biases whose effects become visible only when models are deployed or scaled, distinguishing them from pre-existing and technical biases. By synthesizing recent literature, we highlight how conventional static bias evaluations fail to capture dynamic, context-sensitive behaviors, and survey emerging methodologies for lifecycle-specific assessments. The review underscores the need for adaptive evaluation pipelines, such as dynamic benchmarks, sandbox environments, and continuous monitoring systems, to detect and mitigate biases as they evolve. Finally, we identify theoretical, methodological, and practical directions for advancing a fair and context-aware LLM deployment. We frame emergence broadly to include biases that become visible only with scaling or alignment as well as those that surface post-deployment. Beyond conceptual clarity, the proposed taxonomy is also intended to be actionable, guiding practitioners and researchers in structuring bias audits, evaluations, and monitoring across the LLM lifecycle.
Title: Detection, Classification, and Mitigation of Gender Bias in Large Language Models
Authors: Xiaoqing Cheng; Hongying Zan; Lulu Kong; Jinwang Song; Min Peng
Affiliations: Middle Tennessee State University; Middle Tennessee State University; Middle Tennessee State University
Year: 2025
Domain: LLMs & NLP

Abstract: Skin cancer diagnosis models often suffer from demographic bias and spurious correlations such as reliance on skin tone, background artifacts, or imaging tools due to imbalanced training datasets. These shortcuts lead to reduced performance on underrepresented populations, particularly individuals with darker skin tones (Fitzpatrick IV–VI). This paper proposes a practical and privacy-preserving federated learning framework that synergistically combines causal debiasing techniques to mitigate spurious correlations in skin cancer diagnosis to address these issues without compromising patient privacy. We integrate Invariant Risk Minimization (IRM) with DoWhy-based causal inference within the federated optimization loop, and introduce a causal-aware aggregation strategy to enforce fairness across clients, enabling clients with non-IID skin lesion data to collaboratively train fair and robust models. This method is evaluated on partitioned versions of the ISIC 2020 and Fitzpatrick17k datasets, simulating geographically diverse hospitals. Compared to centralized and vanilla federated baselines, our approach achieves a macro-Area Under the Curve(AUC) of 0.914 and reduces Equal Opportunity Difference and Disparate Impact Ratio disparities by over 50%. This work contributes a privacy-preserving, equity-centered AI framework for dermatology that is both technically rigorous and deployable in low-resource settings. By simultaneously addressing fairness, privacy, and deployability, this work provides a scalable blueprint for equitable AI in global health, achieving sub-600ms inference on low-cost mobile devices, making it viable for deployment in underserved communities.
Title: ReFaRAG: Re-ranking for Bias Mitigation in Retrieval-Augmented Generation
Authors: Yingqi Zhao; Vasilis Efthymiou; Jyrki Nummenmaa; Kostas Stefanidis
Affiliations: Tampere University;Harokopio University of Athens;Tampere University;Tampere University
Year: 2025
Domain: LLMs & NLP

Abstract: The emergence and increasing use of artificial intelligence and machine learning (AI/ML) in healthcare practice and delivery is being greeted with both optimism and caution. We focus on the nexus of AI/ML and racial disparities in healthcare: an issue that must be addressed if the promise of AI to improve patient care and health outcomes is to be realized in an equitable manner for all populations. We unpack the challenge of algorithmic bias that may perpetuate health disparities. Synthesizing research from multiple disciplines, we describe a four-step analytical process used to build and deploy AI/ML algorithms and solutions, highlighting both the sources of bias as well as methods for bias mitigation. Finally, we offer recommendations for moving the pursuit of fairness further.
Title: Anchorlogy: An Ontology for Anchoring Bias Detection in Forecasting
Authors: Mateus Peixoto; Fernanda Baião; Renata Guizzardi; Giancarlo Guizzardi
Affiliations: Ben-Gurion University of the Negev; University of Sussex; Ben-Gurion University of the Negev; Beth Israel Deaconess Medical Center; Harvard–MIT Division of Health Sciences and Technology; Ben-Gurion University of the Negev; BI Norwegian Business School
Year: 2025
Domain: Graph-Based Fairness & Bias Mitigation

Abstract: Integrating machine learning (ML) into decision support systems (DSS) has improved the efficiency and quality of decision making. Yet historical data can embed hidden biases that may lead to unfair outcomes, making algorithmic bias an important societal concern. To address this issue, we develop a fairness-aware DSS based on adversarial learning (AL). The system uses a dual-network architecture: a prediction network that learns task-relevant representations and predicts target outcomes, and an adversarial network that infers protected attributes and thereby helps mitigate bias. As a computationally oriented design science artifact, our DSS embeds fairness norms as executable mechanisms within the model training phase, supporting fairer decisions in practice. We evaluate the model on a Chinese P2P lending dataset and the German Credit dataset. The results show that the proposed approach performs well with respect to four standard fairness criteria, namely unawareness, independence, sufficiency, and separation, while achieving higher predictive accuracy than widely used ML and AL baselines. Overall, this study advances the design of fairness-aware DSS and provides evidence that adversarial learning can reduce algorithmic bias and support fairer decision making.